Mellanox Technologies ConnectX MHRH19-XTC User manual

www.mellanox.com
ConnectX®IB HCA Adapter Card with QSFP
Connectors User Manual
P/N: MHRH19-XTC, MHRH19-XSC, MHRH29-XSC, MHRH29-XTC,
MHQH19-XTC, MHQH19-XSC, MHQH29-XSC, MHQH29-XTC
Rev 1.5

Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
PO Box 586 Hermon Building
Yokneam 20692
Israel
Tel: [phone]
Fax: [phone]
© Copyright 2011. Mellanox Technologies. All rights reserved.
Mellanox®, BridgeX®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniRISC®, InfiniScale®, InfiniPCI®, PhyX® and
Virtual Protocol Interconnect® are registered trademarks of Mellanox Technologies, Ltd. CORE-Direct and FabricIT are trademarks of
Mellanox Technologies, Ltd. All other marks and names mentioned herein may be trademarks of their respective companies.
Document Number: 2909
Rev 1.5
Mellanox Technologies
2
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE
PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE
OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE
CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX
TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX
TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE
DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,
PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND
RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Rev 1.5
Mellanox Technologies 3
Table of Contents
Table of Contents 3
List of Figures 5
List of Tables 6
Revision History 7
About this Manual 8
Intended Audience 8
Related Documentation 8
Online Resources 8
Document Conventions 8
Technical Support 8
Firmware and Software Updates 9
Chapter 1 Overview 10
1.1 Adapter Cards Covered in this Manual 11
1.2 Mellanox Part Numbering Legend 13
1.3 Finding the and Serial Number on the Adapter Cards 14
1.4 Safety Warnings 14
Chapter 2 Adapter Card Interfaces 16
2.1 I/O Interfaces 16
2.2 Power 18
2.3 Memory 18
2.4 VPD Layout 18
Chapter 3 Driver Software and Firmware 23
3.1 Driver Software 23
3.2 FlexBoot 23
3.3 NVIDIA GPUDirect Support 23
3.4 Updating HCA Adapter Card Firmware 24
Chapter 4 HCA InfiniBand Card Installation 26
4.1 Hardware and Software Requirements 26
4.2 Installation Instructions 26
4.3 Set Up 26
4.4 Cables and Modules 32
Appendix A Specifications 34
A.1 MHRH19-X[ST]C Specifications 34
A.2 MHQH19-X[ST]C Specifications 35
A.3 MHRH29-X[ST]C Specifications 35
A.4 MHQH29-X[ST]C Specifications 37
A.5 Board Mechanical Drawing and Dimensions 37
A.6 EMC Certification Statements 38
Appendix B Interface Connectors Pinout 42
B.1 I2C-Compatible Connector Pinout 42
B.2 PCI Express x8 Connector Pinout 42
B.3 PCI Express Connector Pinout 43
B.4 QSFP Connector Pinout 43
Appendix C Replacing a Tall Bracket With a Short Bracket 45
C.1 Replacing a Bracket 45
C.2 Remove the Existing Bracket from the Adapter Card 45

Rev 1.5
Mellanox Technologies
4
C.3 Installing the New Bracket 45
Appendix D Avertissements de sécurité d’installation (Warnings in French) 47
Appendix E Sicherheitshinweise (Warnings in German) 48
Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 49

Rev 1.5
Mellanox Technologies 5
List of Figures
Figure 1: MH[RQ]H29-XTC Card 12
Figure 2: Card Product Label 14
Figure 3: Port Numbering 16
Figure 4: Physical and Logical Link Indications 17
Figure 5: I2C Connector 17
Figure 6: Flash Jumper 18
Figure 7: Support Download Assistant 25
Figure 8: Hardware Devices 28
Figure 9: PCI Device 29
Figure 10: Schematic of the Dual Port Adapter Card MH[RQ]H29B-X[ST]R 38
Figure 11: Compatible Connector Plug and Pinout 42
Figure 12: Connector and Cage Views 43
Figure 13: Remove the Bracket 45
Figure 14: Gasket Installation 46
Figure 15: Placing the Bracket on the Card 46

Rev 1.5
Mellanox Technologies
6
List of Tables
Table 1: Revision History Table 7
Table 2: Documents List 8
Table 3: HCA Cards List 11
Table 4: Mellanox Cards Part Numbering Key 13
Table 5: LEDs 17
Table 6: Jumper Configuration 18
Table 7: VPD Layout for MHRH[12]9-X[ST]C 19
Table 8: VPD Layout for MHQH[12]9-X[ST]C 21
Table 9: Hardware and Software Requirements 26
Table 10: Specifications for MHRH19-X[ST]C 34
Table 11: Specifications for MHQH19-X[ST]C 35
Table 12: Specifications for MHRH29-XSC/-XTC 35
Table 13: Specifications for MHQH29-XSC/-XTC 37
Table 14: HCA Cards Certification Status 38
Table 15: I2C-Compatible Connector Pinout 42
Table 16: Connector Pin Name and Number to Signal Name Correspondence 43

Rev 1.5
Mellanox Technologies 7
Revision History
This document was printed on 2/14/11.
Table 1 - Revision History Table
Date Rev Comments/Changes
February
2011 1.5 Added Set up section.
August
2009 1.4 Added MH[QR]H 19 Short cards Added gaskets to bracket installation
moved Bracket installation from Appendix to Chapter 5
December
2008 1.3 Fixed links to website Added single port cards
August
2008 1.05 Added French Translation of Safety Warnings Removed Watermark
April 2008 0.20 Added MHRH
Changed Link from OFED to Mellanox ODFED
Removed Confidential
March 2008 0.10 Initial Release

Rev 1.5
Mellanox Technologies
8
About this Manual
This User Manual describes Mellanox Technologies MH[RQ]H[12]9-X[ST]C ConnectX IB PCI
Express x8 HCA Adapter cards. HCAIt provides details as to the interfaces of the board, specifica-
tions, required software and firmware for operating the board, and relevant documentation.
Intended Audience
This manual is intended for the installer and user of these cards.
The manual assumes basic familiarity with InfiniBand® and networks and architecture specifica-
tions.
Related Documentation
Table 2 - Documents List
Online Resources
• Mellanox Technologies Web pages: http://www.mellanox.com
• Mellanox Technologies Firmware download Web page:
http://www.mellanox.com =>
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
bytes. The use of Mb or Mbits (small b) indicates size in mega bits.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact us
through the following methods.
•URL:http://www.mellanox.com => Support
• E-mail: [e-mail]
Mellanox Firmware Tools (MFT) User Manual
Document no. 2204UG User Manual describing the set of MFT firmware management tools for a single node.
See http://www.mellanox.com => Firmware & Downloads => Download
Firmware Tools
Or
http://www.mellanox.com => Support => Download Firmware Tools
IBTA Specification Release 1.2.1 InfiniBand Architecture Specification
PCI Express 2.0 Specifications Industry Standard PCI Express 2.0 Card Electromechanical Specification,
Rev 1.3.

Rev 1.5
Mellanox Technologies 9
• Tel: [phone]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract for
details regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seek assistance through their reseller.
Firmware and Software Updates
The Mellanox support downloader contains software, firmware and knowledge database informa-
tion for Mellanox products. Access the data base from the Mellanox Support Web page,
http://www.mellanox.com=>Support
or use the following link to go directly to the Mellanox Support Download Assistant page,
http://www.mellanox.com/supportdownloader/.

Overview
Rev 1.5
Mellanox Technologies
10
1 Overview
This document is a User Manual for Mellanox Technologies host channel adapter (HCA) cards
based on MT25408 , ConnectX®IBthe ConnectXintegrated circuit device. The cards described in
this manual have the following main features:
• QSFP ports for connecting InfiniBand traffic at 10Gb/s (SDR), 20Gb/s (DDR), 40Gb/s (QDR)
• InfiniBand speeds: SDR/DDR/QDR: IB10Gb/s (SDR) or 20Gb/s (DDR) or 40Gb/s (QDR)
(Fiber solutions run at SDR only)
• Two bracket heights: short and tall

Rev 1.5
Mellanox Technologies 11
1.1 Adapter Cards Covered in this Manual
Table 3 lists the InfiniBandHCA cards described in this manual.
Table 3 - HCA Cards List
Ordering Part
Number (OPN)
IB
Port
Speed
On-
board
Memory
Size
Short
/ Tall
Brack
et
HCA Card with Brackets
MHRH19-XTC 20Gb/s
(DDR)
Single
port
No Mem Tall
MHRH19-XSC 20Gb/s
(DDR)
Single
port
No Mem Short
MHQH19-XTC 40Gb/s
(QDR)
Single
port
No Mem Tall
MHQH19-XSC 40Gb/s
(QDR)
Single
port
No Mem Short

Overview
Rev 1.5
Mellanox Technologies
12
These cards are RoHS-R5 Compliant.
MHRH29-XTC 20Gb/s
(DDR)
Dual
port
No Mem Tall
MHRH29-XSC Short
MHQH29-XTC 40Gb/s
(QDR)
Dual
port
No Mem Tall
MHQH29-XSC Short
Table 3 - HCA Cards List
Ordering Part
Number (OPN)
IB
Port
Speed
On-
board
Memory
Size
Short
/ Tall
Brack
et
HCA Card with Brackets
Short Bracket Tall Bracket
Figure 1: MH[RQ]H29-XTC Card

Rev 1.5
Mellanox Technologies 13
1.2 Mellanox Part Numbering Legend
Table 4 describes the Mellanox Technologies adapter cards part numbering legend.
For example, the part number MHQH29-XSC describes Mellanox Technologies’ ConnectX®IB
HCA card with dual QSFP ports, a PCIe2.0 x8 5.0GT/s interface, no on-board memory (mem-
free), a short PCI bracket, and RoHS R5 compliance. Using the legend,
• field M = M to indicate a Mellanox Technologies product,
• field H = H to indicate an InfiniBand Adapter Card,
• field T = Q to indicate QSFP QDR,
• field S = H to indicate the ConnectX family,
• field # = 2 to indicate two ports,
• field I = 9 to indicate PCI Express 2.0 x8 running at 5.0GT/s,
• field X = X to indicate no on-board memory,
• field B = S to indicate a short bracket, and
• field R = C to indicate RoHS R5 (w/ Exemptions) compliance
Table 4 - Mellanox Cards Part Numbering Key
Adapter Card
OPN
MHTS#I-XBR Field Decoder
M Mellanox
Technologies
H Adapter Type H = InfiniBand Host Channel Adapter, N = Ethernet Network Interface Card
TMedia
S Adapter
Architecture H = ConnectX®or ConnectX-2
# # ports 1 = 1, 2 = 2
I Host Interface X = PCI-X, 4 = PCIe x4, 8 = PCIe Gen1 x8, 9 = PCIe (Gen2 x8),
G Generation <blank> = Initial product generation, B= generation B, C= generation C
- Separator
X Memory Size X = MemFree, 1=128MB, 2=256MB, 3=512MB
B Bracket S = Short, T = Tall, N = None
R RoHS <blank> = non RoHS, C = RoHS R-5 w/ Exemption, R = RoHS R-6 Lead-Free

Overview
Rev 1.5
Mellanox Technologies
14
1.3 Finding the and Serial Number on the Adapter Cards
All Mellanox adapter cards have a label on the printed side of the adapter card that has the card
serial number and the card GUID for InfiniBand protocol .
Figure 2: Card Product Label
1.4 Safety Warnings
For safety warnings in French see “Avertissements de sécurité d’installation (Warnings in French)”
on page 47. For safety warnings in German see “Sicherheitshinweise (Warnings in German)” on
page 48. For safety warnings in Spanish see “Advertencias de seguridad para la instalación (Warn-
ings in Spanish)” on page 49.
1. Installation Instructions
2. Over-temperature
3. During Lightning - Electrical Hazard
4. Copper Cable Connecting/Disconnecting
Read all installation instructions before connecting the equipment to the power source.
This equipment should not be operated in an area with an ambient temperature exceed-
ing the maximum recommended: 55°C (131°F).
To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ven-
tilation openings.
During periods of lightning activity, do not work on the equipment or connect or dis-
connect cables.
Some copper cables are heavy and not flexible, as such they should be carefully
attached to or detached from the connectors. Refer to the cable manufacturer for spe-
cial warnings and instructions.
S/N:MT0744X00012
P/N:MHQH29-XTC
GUID: 0002C90300025 B06
Made in IL
REV: X1

Rev 1.5
Mellanox Technologies 15
5. Equipment Installation
6. Equipment Disposal
7. Local and National Electrical Codes
This equipment should be installed, replaced, or serviced only by trained and qualified
personnel.
Disposal of this equipment should be in accordance to all national laws and regula-
tions.
This equipment should be installed in compliance with local and national electrical
codes.

Adapter Card Interfaces
Rev 1.5
Mellanox Technologies
16
2Adapter Card Interfaces
2.1 I/O Interfaces
Each HCA Adapter card includes the following interfaces:
• QSFP ports
• PCI Express x8 edge connector
• I/O panel LEDs
•I
2C compatible connector (for debug)
For dual port cards, port 1 connects to connector 1 of the device, while port 2 connects to connec-
tor 2 of the device.
Figure 3: Port Numbering
2.1.1 InfiniBand Interface
The ConnectX device is compliant with the InfiniBand Architecture Specification, Release 1.2.1. It has
compliant 4X InfiniBandports, with four Tx/Rx pairs of SerDes. HCA adapter cards (listed in
Table 3) based on this device provide access to its ports by means of QSFP connectorsconnectors.
2.1.2 PCI Express Interface
The ConnectX-adapter cards support PCI Express 2.0 (1.1 compatible) through an x8 edge con-
nector. The device can be either a master initiating the PCI Express bus operations or a slave
responding to PCI bus operations.
Port 2
Port 1

Rev 1.5
Mellanox Technologies 17
2.1.3 LED Assignment
The board has I/O LEDs located on the I/O panel- 2 LEDs per port. The green LED, when lit, indi-
cates that the driver is running and a valid physical connection between nodes exists. If the green
LED is blinking, it indicates a problem with the physical link. The yellow LED when lit, indicates
a valid data activity link, this is the logical link. The yellow LED lights up when the network is
discovered over the physical link. A valid data activity link without data transfer is designated by a
constant yellow LED indication. A valid data activity link with data transfer is designated by a
blinking yellow LED indication. If the LEDs are not active, either the physical link or the logical
link (or both) connections have not been established.
Figure 4: Physical and Logical Link Indications
2.1.4 I2C Compatible Interface
A three-pin header on the HCA Adapter card is provided as the I2C compatible interface. See
Appendix A,“Specifications,” on page 34 for the location on the board.
Figure 5: I2C Connector
The short bracket has the same port and LED footprint as the tall bracket.
Table 5 - LEDs
Port Number LED Name
Port 1 Physical Link - Green
Constant on indicates a good physical link
Blinking indicates a problem with the Physical
link
Data Activity - Yellow
Blinking indicates Data Transfer
Constant on indicates no Data Transfer
Port 2 Physical Link - Green
Constant on indicates a good physical link
Blinking indicates a problem with the Physical
link
Data Activity - Yellow
Blinking indicates Data Transfer
Constant on indicates no Data Transfer
Port 2
Port 1

Adapter Card Interfaces
Rev 1.5
Mellanox Technologies
18
2.2 Power
2.3 Memory
The adapter cards support multiple memory devices through the PCI Flash, and I2C.
2.3.1 System Memory
The HCA Adapter card utilizes the PCI Express interface to store and access connection informa-
tion and packet data on the system memory.
2.3.2 Flash
Each of the HCA Adapter cards include one 2MB SPI Flash device (M25P80 device by ST Micro-
electronics) accessible via the Flash interface of the ConnectX-IB device.
There is a jumper on each adapter card that indicates to the device whether an on-board Flash
device is to be used. Table 6 provides information on this jumper. See the schematic in
Appendix A,“Specifications,” on page 34 for the jumper location.
2.3.3 EEPROM
Each board incorporates an EEPROM that is accessible through the I2C. The EEPROM is used for
storing the Vital Product Data (VPD). The EEPROM capacity is 512 bytes.
2.4 VPD Layout
The PCI VPD (Vital Product Data) layout, for each of the described Mellanox Technologies Con-
nectX®-HCA Adapter adapter cards complies with the format defined in the PCI 2.3 Specifica-
tion, Appendix I. All ConnectX-adapter cards have the same PCI VPD layout.“A1” was used as the
HCA card (PCB) revision. Later revisions of the HCA card will have the same format.
Table 6 - Jumper Configuration
Description Option Card Default
Configuration Comments
Flash present/
not present connection open – Flash present connection shorted – Flash not present
Figure 6: Flash Jumper
connection open –
Flash present Header 1x2

Rev 1.5
Mellanox Technologies 19
2.4.1 PCI VPD Layout
Table 7 - VPD Layout for MHRH[12]9-X[ST]C
Offset
(Decimal) Item Value Format Description
0Large Resource Type ID String
Tag (0x02) 0x82
1Length [7:0] LSB 0xD
2Length [15:8] MSB 0x0
3Data FALCON IB
QDR STR
16 Large Resource Type VPD-R Tag
(0x10) 0x90
17 Length [7:0] LSB 0x4F
18 Length [15:8] MSB 0x00
19 VPD Keyword PN STR Add in Card Part Number
21 Length 0x15
22 PN PN %STR_SPC
43 VPD Keyword EC STR Engineering Change Level of
the card (rev)
45 Length 0x2
46 Revision A1 %STR PCB revision
48 VPD Keyword SN STR Serial Number
50 Length 0x18
51 SerialNumber %STR_SPC “00..00XXXX..XX”
75 VPD Keyword V0 STR Misc Information
77 Length 0x10
78 Data PCIe Gen2 x8 STR_SPC
94 VPD Keyword RV STR
96 Length 0x1
97 Data 0,96 %CS0
98 Large Resource Type VPD-W
Tag (0x11) 0x91
99 Length [7:0] LSB 0x9A
100 Length [15:8] MSB 0x00
101 VPD Keyword V1 STR EFI Driver version
103 Length 0x6
104 Data N/A STR_SPC
110 VPD Keyword YA STR Asset Tag
112 Length 0x20
113 Data N/A STR_SPC “N/A”
145 VPD Keyword RW STR Remaining read/write area
147 Length 0x6b
148 Data STR_ZERO Reserved (0x00)

Adapter Card Interfaces
Rev 1.5
Mellanox Technologies
20
255 Small Resource Type END Tag
(0x11) 0x78
Table 7 - VPD Layout for MHRH[12]9-X[ST]C
Offset
(Decimal) Item Value Format Description