Mellanox Technologies MNPH29C-XSR User manual

www.mellanox.com
ConnectX®-2 EN Dual Port SFP+ Ethernet
Adapter Card User Manual
P/N:
MNPH29C-XSR, MNPH29C-XTR, MNPH29D-XSR, MNPH29D-XTR
Rev 1.6

Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
PO Box 586 Hermon Building
Yokneam 20692
Israel
Tel: [phone]
Fax: [phone]
© Copyright 2011. Mellanox Technologies. All rights reserved.
Mellanox®, BridgeX®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniRISC®, InfiniScale®, InfiniPCI®, PhyX® and
Virtual Protocol Interconnect® are registered trademarks of Mellanox Technologies, Ltd. CORE-Direct and FabricIT are trademarks of
Mellanox Technologies, Ltd. All other marks and names mentioned herein may be trademarks of their respective companies.
ConnectX-2 Ethernet Card User Manual
Document Number: 3160
Rev 1.6
Mellanox Technologies
2
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE
PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE
OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE
CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX
TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX
TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE
DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,
PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND
RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ConnectX-2 Ethernet Card User Manual Rev 1.6
Mellanox Technologies 3
Table of Contents
Table of Contents 3
List of Figures 5
List of Tables 6
Revision History 7
About this Manual 8
Intended Audience 8
Related Documentation 8
Online Resources 8
Document Conventions 8
Technical Support 8
Firmware and Software Updates 9
Chapter 1 Overview 10
1.1 Adapter Cards Covered in this Manual 11
1.2 Mellanox Part Numbering Legend 13
1.3 Finding the MAC and Serial Number on the Adapter Cards 14
1.4 Safety Warnings 14
Chapter 2 Adapter Card Interfaces 16
2.1 I/O Interfaces 16
2.2 Power 18
2.3 Memory 18
2.4 VPD Layout 19
Chapter 3 Driver Software and Firmware 22
3.1 Driver Software 22
3.2 RDMA over Converged Ethernet (RoCE) 22
3.3 Updating Card Firmware 23
3.4 ConnectX EN PXE 24
Chapter 4 Ethernet Adapter Card Installation 25
4.1 Hardware and Software Requirements 25
4.2 Installation Instructions 25
4.3 Set Up 26
4.4 Cables and Modules 31
Appendix A Specifications 34
A.1 MNPH29D-X[ST]R Specifications 34
A.2 MNPH29C-X[ST]R Specifications 35
A.3 Board Mechanical Drawing and Dimensions 35
A.4 EMC Certification Statements 37
Appendix B Interface Connectors Pinout 40
B.1 I2C-Compatible Connector Pinout 40
B.2 PCI Express x8Connector Pinout 40
B.3 PCI Express Connector Pinout 41
B.4 SFP+ Connector Pinout 41
Appendix C Replacing a Tall Bracket With a Short Bracket 43
C.1 Remove the Existing Bracket from the Adapter Card 43
C.2 Installing the New Bracket 43
Appendix D Avertissements de sécurité d’installation (Warnings in French) 45

Rev 1.6
Mellanox Technologies
4
Appendix E Sicherheitshinweise (Warnings in German) 46
Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 47

ConnectX-2 Ethernet Card User Manual Rev 1.6
Mellanox Technologies 5
List of Figures
Figure 1: SFP+ NIC Component Side and Bracket Views 12
Figure 2: Card Product Label 14
Figure 3: Port Numbering 16
Figure 4: LED - Port Association 17
Figure 5: Physical and Logical Link Indications 17
Figure 6: I2C Connector 18
Figure 7: Flash Jumper 19
Figure 8: Support Download Assistant 24
Figure 9: Hardware Devices 27
Figure 10: PCI Device 28
Figure 11: Module With Locking Mechanism Closed 32
Figure 12: Module With Locking Mechanism Open 32
Figure 13: Schematic of the ConnectX-2 Small Form Factor MNPH29D Adapter Card 36
Figure 14: Schematic of the ConnectX-2 Card MNPH29C Adapter Card 37
Figure 15: Compatible Connector Plug and Pinout 40
Figure 16: Rear View of Module With Pin Placement 41
Figure 17: Remove the Bracket 43
Figure 18: Putting on the Bracket 44

Rev 1.6
Mellanox Technologies
6
List of Tables
Table 1: Revision History Table 7
Table 2: Documents List 8
Table 3: Network Interface Cards 11
Table 4: Mellanox Cards Part Numbering Key 13
Table 5: LEDs 17
Table 6: Jumper Configuration 19
Table 7: VPD Layout for MNPH29D-X[ST]R 19
Table 8: VPD Layout for MNPH29C-X[ST]R 20
Table 9: Hardware and Software Requirements 25
Table 10: Specifications for MNPH29D-X[ST]R 34
Table 11: Specifications for MNPH29C-X[ST]R 35
Table 12: Adapter Cards Certification Status 37
Table 13: I2C-Compatible Connector Pinout 40
Table 14: SFP+ Connector Pinout 42

ConnectX-2 Ethernet Card User Manual Rev 1.6
Mellanox Technologies 7
Revision History
This document was printed on 2/15/11.
Table 1 - Revision History Table
Date Rev Comments/Changes
Jan. 2011 1.6 Changed Air flow requirement
Added to the Set up section
Dec. 19th,
2010
1.5 Minor formatting changes
Nov. 22nd,
2010
1.4 Added Generation D card Mini Form Factor
Oct. 28th,
2010
1.3 Added Technical Support and Firmware and Software Updates sections to the Pref-
ace section
Added RoCE
Added ConnectX EN PXE
Added Set up section, This is a sanity check after card installation.
Added KCC statement to Appendix A
Added QSFP to SFP+ QSA adapter
Removed Gen 1 cards
Formatted EMC table
Changed EN 60950-1: 2001 to EN 60950-1: 2006
Changed RDMA over Ethernet
Added RDMA over Ethernet
New template
April 2010 1.2 New power numbers
Feb. 2010 1.1 Added Section on PXE boot
Feb. 2010 1.0 Initial Release

Rev 1.6
Mellanox Technologies
8
About this Manual
This User Manual describes Mellanox Technologies ConnectX®-2 Dual Port 10Gigabit Ethernet
PCI Express x8 adapter cards. It provides details as to the interfaces of the board, specifications,
required software and firmware for operating the board, and relevant documentation.
Intended Audience
This manual is intended for the installer and user of these cards.
The manual assumes basic familiarity with Ethernet networks and architecture specifications.
Related Documentation
Table 2 - Documents List
Online Resources
• Mellanox Technologies Web pages: http://www.mellanox.com
• Mellanox Technologies Firmware download Web page:
http://www.mellanox.com =>
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
bytes. The use of Mb or Mbits (small b) indicates size in mega bits.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact us
through the following methods.
•URL:http://www.mellanox.com => Support
• E-mail: [e-mail]
Mellanox Firmware Tools (MFT) User Manual
Document no. 2204UG
User Manual describing the set of MFT firmware management tools for a
single node.
See http://www.mellanox.com => Firmware & Downloads => Download
Firmware Tools
Or
http://www.mellanox.com => Support => Download Firmware Tools
IEEE Std 802.3 Specification This is the IEEE Ethernet specification
http://standards.ieee.org/getieee802
PCI Express 2.0 Specifications Industry Standard PCI Express 2.0 Card Electromechanical Specification,
Rev 1.3.

ConnectX-2 Ethernet Card User Manual Rev 1.6
Mellanox Technologies 9
• Tel: [phone]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract for
details regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seek assistance through their reseller.
Firmware and Software Updates
The Mellanox support downloader contains software, firmware and knowledge database informa-
tion for Mellanox products. Access the data base from the Mellanox Support Web page,
http://www.mellanox.com=>Support
or use the following link to go directly to the Mellanox Support Download Assistant page,
http://www.mellanox.com/supportdownloader/.

Overview
Rev 1.6
Mellanox Technologies
10
1 Overview
This document is a User Manual for Mellanox Technologies cards based on the ConnectX®-2
ENintegrated circuit device. The cards described in this manual have the following main features:
• IEEE Std 802.3 compliant
• 10 Gb/s Ethernet
• Dual SFP+ ports for connection Ethernet traffic
• PCI Express 2.0 (1.1 compatible) through an x8 edge connector up to 5GT/s
• Fiber Channel encapsulation (FCoE)
• RDMA over Converged Ethernet (RoCE)
•RoHS-R6
• Two bracket heights: short and tall

ConnectX-2 Ethernet Card User Manual Rev 1.6
Mellanox Technologies 11
1.1 Adapter Cards Covered in this Manual
Table 3 lists the Ethernet adapter cards described in this manual.
Table 3 - Network Interface Cards
Ordering Part
Number (OPN)
PCI Express
SERDES
Speed
Data Transmission
Rate/
# of ports
Short / Tall
Bracket RoHS Adapter IC Part Number
MNPH29D-XTR PCIe Gen2
5.0 GT/s
10 +10 Gb/s
Ethernet
2 ports
Tall R-6 MT25408B0-FCCR-TE
MNPH29D-XSR PCIe Gen2
5.0 GT/s
10 +10 Gb/s
Ethernet
2 ports
Short R-6 MT25408B0-FCCR-TE
MNPH29C-XTR PCIe Gen2
5.0 GT/s
10 +10 Gb/s
Ethernet
2 ports
Tall R-6 MT25408B0-FCCR-TE
MNPH29C-XSR PCIe Gen2
5.0 GT/s
10 +10 Gb/s
Ethernet
2 ports
Short R-6 MT25408B0-FCCR-TE

Overview
Rev 1.6
Mellanox Technologies
12
Figure 1: SFP+ NIC Component Side and Bracket Views
J1
U1
MNPH29C
J1
U1
MNPH29D

ConnectX-2 Ethernet Card User Manual Rev 1.6
Mellanox Technologies 13
1.2 Mellanox Part Numbering Legend
Table 4 describes the Mellanox Technologies adapter cards part numbering legend.
For example, the part number MNPH29C-XTR describes Mellanox Technologies’ ConnectX EN
adapter card with two SFP+ ports, a PCIe2.0 x8 5.0GT/s interface, no on-board memory (mem-
free), a tall bracket, and RoHS R6 compliance. Using the legend,
• field M = M to indicate a Mellanox Technologies product,
• field H = N to indicate a Network Interface Card,
• field T = P to indicate 10GBASE-SR/LR,
• field S = H to indicate ConnectX
• field # = 2 to indicate two ports,
• field I = 9 to indicate PCI Express 2.0 x8 running at 5.0GT/s,
• field G = C to indicate Generation C ConnectX-2
• field X = X to indicate no on-board memory,
• field B = T to indicate a tall bracket, and
• field R = R to indicate RoHS R6 compliance
Table 4 - Mellanox Cards Part Numbering Key
Adapter Card
OPN
MHTS#I-XBR Field Decoder
M Mellanox
Technologies
H Adapter Type H = InfiniBand Host Channel Adapter, N = Ethernet Network Interface Card
T Media P = SFP+,
S Adapter
Architecture
H = ConnectX®or ConnectX-2
# # ports 1 = 1, 2 = 2
I Host Interface X = PCI-X, 4 = PCIe x4, 8 = PCIe Gen1 x8, 9 = PCIe (Gen2 x8),
G Generation <blank> = Initial product generation, B= generation B, C= generation C
- Separator
X Memory Size X = MemFree, 1=128MB, 2=256MB, 3=512MB
B Bracket S = Short, T = Tall, N = None
R RoHS <blank> = non RoHS, C = RoHS R-5 w/ Exemption, R = RoHS R-6 Lead-Free

Overview
Rev 1.6
Mellanox Technologies
14
1.3 Finding the MAC and Serial Number on the Adapter Cards
All Mellanox adapter cards have a label on the printed side of the adapter card that has the card
serial number, and the card MAC.
Figure 2: Card Product Label
1.4 Safety Warnings
For safety warnings in French see “Avertissements de sécurité d’installation (Warnings in French)”
on page 45. For safety warnings in German see “Sicherheitshinweise (Warnings in German)” on
page 46. For safety warnings in Spanish see “Advertencias de seguridad para la instalación (Warn-
ings in Spanish)” on page 47.
1. Installation Instructions
2. Over-temperature
Port 1 uses the GUID or MAC ID described on the label, for port 2 GUID or MAC add
1 to port 1's description.
Read all installation instructions before connecting the equipment to the power source.
This equipment should not be operated in an area with an ambient temperature exceed-
ing the maximum recommended: 55°C (131°F).
To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ven-
tilation openings.
S/N:MT0744X00012
P/N:MNPH29C-XTR REV: X1
Made in IL
MAC: 0002C9020040DDB0

ConnectX-2 Ethernet Card User Manual Rev 1.6
Mellanox Technologies 15
3. During Lightning - Electrical Hazard
4. Copper Cable Connecting/Disconnecting
5. Equipment Installation
6. Equipment Disposal
7. Local and National Electrical Codes
During periods of lightning activity, do not work on the equipment or connect or dis-
connect cables.
Some copper cables are heavy and not flexible, as such they should be carefully
attached to or detached from the connectors. Refer to the cable manufacturer for spe-
cial warnings and instructions.
This equipment should be installed, replaced, or serviced only by trained and qualified
personnel.
Disposal of this equipment should be in accordance to all national laws and regula-
tions.
This equipment should be installed in compliance with local and national electrical
codes.

Adapter Card Interfaces
Rev 1.6
Mellanox Technologies
16
2Adapter Card Interfaces
2.1 I/O Interfaces
Each card includes the following interfaces:
•SFP+ports
• I/O panel LEDs
•I
2C compatible connector (for debug)
For dual port cards, port 1 connects to connector 1 of the device, while port 2 connects to connec-
tor 2 of the device.
ConnectX-2 Ethernet Adapter Cards are compliant with the IEEE Std 802.3 Specification.
Figure 3: Port Numbering
2.1.1 Ethernet SFP+ Interface
The ConnectX®-2 device is compliant with the IEEE Std 802.3. The SFP+ port has one Tx/Rx pair
of SerDes.Ethernet NIC cards listed in Table 3 on page 11 based on this device provide access to
the Ethernet ports by means of SFP+ connectors.
2.1.2 PCI Express Interface
The ConnectX-2 adapter cards support PCI Express 2.0 (1.1 compatible) through an x8 edge con-
nector. The device can be either a master initiating the PCI Express bus operations or a slave
responding to PCI bus operations.
Port 2
Port 1

ConnectX-2 Ethernet Card User Manual Rev 1.6
Mellanox Technologies 17
2.1.3 LED Assignment
The board has I/O LEDs located on the I/O panel. The green LED, when lit, indicates that the
driver is running and a valid physical connection between nodes exists. If the green LED is blink-
ing, it indicates a problem with the physical link. The yellow LED when lit, indicates a valid data
activity link, this is the logical link. The yellow LED lights up when the network is discovered
over the physical link. A valid data activity link without data transfer is designated by a constant
yellow LED indication. A valid data activity link with data transfer is designated by a blinking yel-
low LED indication. If the LEDs are not active, either the physical link or the logical link (or both)
connections have not been established.
Figure 4: LED - Port Association
Figure 5: Physical and Logical Link Indications
Port 1
Port 2
Table 5 - LEDs
Port Number LED Name
Port 1 Physical Link - Green
Constant on indicates a good physical link
Blinking indicates a problem with the Physical
link
Data Activity - Yellow
Blinking indicates Data Transfer
Constant on indicates no Data Transfer
Port 2 Physical Link - Green
Constant on indicates a good physical link
Blinking indicates a problem with the Physical
link
Data Activity - Yellow
Blinking indicates Data Transfer
Constant on indicates no Data Transfer
Port 1
Port 2

Adapter Card Interfaces
Rev 1.6
Mellanox Technologies
18
2.1.4 I2C Compatible Interface
A three-pin header on the card is provided as the I2C compatible interface. See
Appendix A,“Specifications,” on page 34 for the location on the board.
Figure 6: I2C Connector
2.2 Power
All adapter cards receive 12V and 3.3V power from the PCI Express Edge connector. All other
required power voltages are generated by on-board switch mode regulators. See “Specifications”
on page 34.
2.3 Memory
The adapter cards support multiple memory devices through the PCI Flash, and I2C.
2.3.1 System Memory
The card utilizes the PCI Express interface to store and access Ethernet fabric connection informa-
tion and packet data on the system memory.
2.3.2 Flash
Each of the cards include one SPI Flash device accessible via the Flash interface of
theMT25408B0 ConnectX-2 EN device.
There is a jumper on each adapter card that indicates to the device whether an on-board Flash
device is to be used. Table 6 provides information on this jumper. See the schematic in
Appendix A,“Specifications,” on page 34 for the jumper location.
The short bracket has the same port and LED footprint as the tall bracket.

ConnectX-2 Ethernet Card User Manual Rev 1.6
Mellanox Technologies 19
2.3.3 EEPROM
Each board incorporates an EEPROM that is accessible through the I2C. The EEPROM is used for
storing the Vital Product Data (VPD). The EEPROM capacity is 4Kb.
2.4 VPD Layout
The PCI VPD (Vital Product Data) layout, for each of the described Mellanox Technologies Con-
nectX®-2 adapter cards complies with the format defined in the PCI 2.3 Specification, Appendix I.
All ConnectX-2®adapter cards have the same PCI VPD layout.
Table 6 - Jumper Configuration
Description Option Card Default
Configuration Comments
Flash present/
not present
connection open – Flash present
connection shorted – Flash not present
Figure 7: Flash Jumper
connection open –
Flash present
Header 1x2
Table 7 - VPD Layout for MNPH29D-X[ST]R
Offset (Decimal) Item Value Format Description
0 Large Resource Type ID String Tag
(0x02)
0x82
1 Length [7:0] LSB 0xE
2 Length [15:8] MSB 0x0
3 Data Hawk Dual Port STR
17 Large Resource Type VPD-R Tag
(0x10)
0x90
18 Length [7:0] LSB 0x4F
19 Length [15:8] MSB 0x00
20 VPD Keyword PN STR Add in Card Part Number
22 Length 0x15
23 PN PN %STR_SPC
44 VPD Keyword EC STR Engineering Change Level of the card
(rev)
46 Length 0x2
47 Revision RV %STR PCB revision
49 VPD Keyword SN STR Serial Number
51 Length 0x18
52 SerialNumber %STR_SPC “00..00XXXX..XX”

Adapter Card Interfaces
Rev 1.6
Mellanox Technologies
20
76 VPD Keyword V0 STR Misc Information
78 Length 0x10
79 Data PCIe Gen2 x8 STR_SPC
95 VPD Keyword RV STR
97 Length 0x1
98 Data 0,97 %CS0
99 Large Resource Type VPD-W Tag
(0x11)
0x91
100 Length [7:0] LSB 0x99
101 Length [15:8] MSB 0x00
102 VPD Keyword V1 STR EFI Driver version
104 Length 0x6
105 Data N/A STR_SPC
111 VPD Keyword YA STR Asset Tag
113 Length 0x20
114 Data N/A STR_SPC “N/A”
146 VPD Keyword RW STR Remaining read/write area
148 Length 0x6A
149 Data STR_ZERO Reserved (0x00)
255 Small Resource Type END Tag (0x11) 0x78
Table 8 - VPD Layout for MNPH29C-X[ST]R
Offset
(Decimal) Item Value Format Description
0 Large Resource Type ID String Tag (0x02) 0x82
1 Length [7:0] LSB 0xE
2 Length [15:8] MSB 0x0
3 Data Hawk Dual Port STR
17 Large Resource Type VPD-R Tag (0x10) 0x90
18 Length [7:0] LSB 0x4F
19 Length [15:8] MSB 0x00
20 VPD Keyword PN STR Add in Card Part Number
22 Length 0x15
23 PN PN %STR_SPC
44 VPD Keyword EC STR Engineering Change Level of
the card (rev)
46 Length 0x2
47 Revision RV %STR PCB revision
49 VPD Keyword SN STR Serial Number
51 Length 0x18
52 SerialNumber %STR_SPC “00..00XXXX..XX”
Table 7 - VPD Layout for MNPH29D-X[ST]R
Offset (Decimal) Item Value Format Description