Mellanox Technologies ConnectX-2 Series User manual

www.mellanox.com
ConnectX®-2 LX VPI Single Port QSFP
InfiniBand and Ethernet Adapter Card User
Manual
P/N: MHQA19-XSR, MHQA19-XTR, MHQA19-XNR
Rev 1.0

Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
PO Box 586 Hermon Building
Yokneam 20692
Israel
Tel: [phone]
Fax: [phone]
© Copyright 2010. Mellanox Technologies. All rights reserved.
Mellanox, BridgeX, ConnectX, Virtual Protocol Interconnect, InfiniBlast, InfiniBridge, InfiniHost, InfiniRISC, InfiniScale, and InfiniPCI
are registered trademarks of Mellanox Technologies, Ltd. CORE-Direct, FabricIT, and PhyX are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
All other marks and names mentioned herein may be trademarks of their respective companies.
ConnectX-2 VPI Card User Manual
Document Number: 3375
Rev 1.0
Mellanox Technologies
2
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE
PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE
OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE
CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX
TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX
TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE
DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,
PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND
RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ConnectX-2 Card User Manual Rev 1.0
Mellanox Technologies 3
Table of Contents
Table of Contents 3
List of Figures 5
List of Tables 6
Revision History 7
About this Manual 8
Intended Audience 8
Related Documentation 8
Online Resources 8
Document Conventions 8
Technical Support 8
Firmware and Software Updates 9
Chapter 1 Overview 10
1.1 Adapter Cards Covered in this Manual 11
1.2 Mellanox Part Numbering Legend 12
1.3 Finding the and Serial Number on the Adapter Cards 13
1.4 Safety Warnings 13
Chapter 2 Adapter Card Interfaces 15
2.1 I/O Interfaces 15
2.2 Power 17
2.3 QSFP Power Level 17
2.4 Memory 17
2.5 VPD Layout 18
Chapter 3 Driver Software and Firmware 20
3.1 Driver Software 20
3.2 FlexBoot 20
3.3 Updating Adapter Card Firmware 20
Chapter 4 VPI Adapter Card Installation 22
4.1 Hardware and Software Requirements 22
4.2 Installation Instructions 22
4.3 Set Up 22
4.4 Cables and Modules 29
Appendix A Specifications 31
A.1 MHQA19-X[STN]R Specifications 31
A.2 Board Mechanical Drawing and Dimensions 31
A.3 EMC Certification Statements 32
Appendix B Interface Connectors Pinout 35
B.1 I2C-Compatible Connector Pinout 35
B.2 PCI Express x8Connector Pinout 35
B.3 PCI Express Connector Pinout 36
B.4 QSFP Connector Pinout 36
Appendix C Replacing a Tall Bracket With a Short Bracket 38
C.1 Remove the Existing Bracket from the Adapter Card 38
C.2 Installing the New Bracket 38
Appendix D Avertissements de sécurité d’installation (Warnings in French) 41
Appendix E Sicherheitshinweise (Warnings in German) 42

Rev 1.0
Mellanox Technologies
4
Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 43

ConnectX-2 Card User Manual Rev 1.0
Mellanox Technologies 5
List of Figures
Figure 1: Single Port Card Component Side 11
Figure 2: Card Product Label 13
Figure 3: Physical and Logical Link Indications 16
Figure 4: I2C Connector 17
Figure 5: Flash Jumper 18
Figure 6: Support Download Assistant 21
Figure 7: Hardware Devices 23
Figure 8: PCI Device 24
Figure 9: MHQA-19 Mechanical Drawing 32
Figure 10: Compatible Connector Plug and Pinout 35
Figure 11: Connector and Cage Views 36
Figure 12: Remove the Bracket 38
Figure 13: Gasket Installation 39
Figure 14: Putting on the Bracket 39

Rev 1.0
Mellanox Technologies
6
List of Tables
Table 1: Revision History Table 7
Table 2: Documents List 8
Table 3: Single Port Cards List 11
Table 4: Mellanox Cards Part Numbering Key 12
Table 5: LEDs 16
Table 6: Jumper Configuration 18
Table 7: MHQA19X[STN]R 18
Table 8: Hardware and Software Requirements 22
Table 9: Specifications for MHQA19-X[ST]C 31
Table 10: Adapter Cards Certification Status 32
Table 11: I2C-Compatible Connector Pinout 35
Table 12: Connector Pin Name and Number to Signal Name Correspondence 36

ConnectX-2 Card User Manual Rev 1.0
Mellanox Technologies 7
Revision History
This document was printed on 11/17/10 .
Table 1 - Revision History Table
Date Rev Comments/Changes
Nov. 2010 1.0 Initial draft

Rev 1.0
Mellanox Technologies
8
About this Manual
This User Manual describes Mellanox Technologies ConnectX®-2 Single Port InfiniBand and
Ethernet PCI Express x8 adapter cards. It provides details as to the interfaces of the board, specifi-
cations, required software and firmware for operating the board, and relevant documentation.
Intended Audience
This manual is intended for the installer and user of these cards.
The manual assumes basic familiarity with InfiniBand® and Ethernet networks and architecture
specifications.
Related Documentation
Table 2 - Documents List
Online Resources
• Mellanox Technologies Web pages: http://www.mellanox.com
• Mellanox Technologies Firmware download Web page:
http://www.mellanox.com => Downloads => Firmware
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
bytes. The use of Mb or Mbits (small b) indicates size in mega bits.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact us
through the following methods.
•URL:http://www.mellanox.com => Support
• E-mail: [e-mail]
Mellanox Firmware Tools (MFT) User Manual
Document no. 2204UG User Manual describing the set of MFT firmware management tools for a single node.
See http://www.mellanox.com => Downloads => Firmware Tools
IBTA Specification Release 1.2.1 InfiniBand Architecture Specification
IEEE Std 802.3 Specification This is the IEEE Ethernet specification
http://standards.ieee.org/getieee802
PCI Express 2.0 Specifications Industry Standard PCI Express 2.0 Card Electromechanical Specification,
Rev 1.3.

ConnectX-2 Card User Manual Rev 1.0
Mellanox Technologies 9
• Tel: [phone]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract for
details regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seek assistance through their reseller.
Firmware and Software Updates
The Mellanox support downloader contains software, firmware and knowledge database informa-
tion for Mellanox products. Access the data base from the Mellanox Support Web page,
http://www.mellanox.com=>Support
or use the following link to go directly to the Mellanox Support Download Assistant page,
http://www.mellanox.com/supportdownloader/.

Overview
Rev 1.0
Mellanox Technologies
10
1 Overview
This document is a User Manual for Mellanox Technologies cards based on the integrated circuit
device. The cards described in this manual have the following main features:
• Virtual Protocol Interconnect IBTA v1.2.1 compliant
• Compliant with QSFP MSA spec Rev. 1.0
• Compatible with copper cables and optical cables with the use of QSFP connectors. Support for
SFP+ cables is available through QSA (Quad to Serial) adapter.
• CPU offload of transport operations
• End-to-end QoS and congestion control
• Hardware-based I/O virtualization
• TCP/UDP/IP stateless offload
•RoHS-R6
• PCI Express 2.0 (1.1 compatible) through an x8 edge connector up to 5GT/s
• Two bracket heights: short and tall

ConnectX-2 Card User Manual Rev 1.0
Mellanox Technologies 11
1.1 Adapter Cards Covered in this Manual
Table 3 lists
Figure 1: Single Port Card Component Side
Table 3 - Single Port Cards List
Ordering Part Number
(OPN)
PCI
Express
SERDES
Speed
Data Transmis-
sion Rate/
# of ports
Short /
Tall
Bracket RoHS Adapter IC Part Number
MHQA19-XTR PCIe Gen2
5.0 GT/s 40 Gb/s QDR 1 port Tall R-6 MT25414B0-FCCR-QI
MHQA19-XSR PCIe Gen2
5.0 GT/s 40 Gb/s QDR 1 port Short R-6 MT25414B0-FCCR-QI
MHQA19-XNR PCIe Gen2
5.0 GT/s 40 Gb/s QDR 1 port No bracket R-6 MT25414B0-FCCR-QI
J1
U1

Overview
Rev 1.0
Mellanox Technologies
12
1.2 Mellanox Part Numbering Legend
Table 4 describes the Mellanox Technologies adapter cards part numbering legend.
For example, the part number MHQA19-XTR describes Mellanox Technologies’ ConnectX®-2
LX VPI card with 1 QSFP port, a PCIe2.0 x8 5.0GT/s interface, no on-board memory (mem-free),
a tall PCI bracket, and RoHS R6 compliance. Using the legend,
• field M = M to indicate a Mellanox Technologies product,
• field H = H to indicate an InfiniBand Adapter Card,
• field T = Q to indicate QSFP QDR,
• field S = A to indicate the ConnectX®-2 LX family,
• field # = 1 to indicate one port,
• field I = 9 to indicate PCI Express 2.0 x8 running at 5.0GT/s,
• field G = blank to indicate first generation card
• field X = X to indicate no on-board memory,
• field B = T to indicate a tall bracket, and
• field R = R to indicate RoHS R6 (with no exemptions) compliance
Table 4 - Mellanox Cards Part Numbering Key
Adapter Card
OPN
MHTS#I-XBR Field Decoder
M Mellanox
Technologies
H Adapter Type H = InfiniBand Host Channel Adapter, N = Ethernet Network Interface Card
T Media P = SFP+, T = UTP, Z = one SFP+ connector and one QSFP connector
S Adapter
Architecture T= InfiniHost®, A = InfiniHost® III Ex, S = InfiniHost®III Lx,
H = ConnectX®,
# # ports 1 = 1, 2 = 2
I Host Interface X = PCI-X, 4 = PCIe x4, 8 = PCIe x8, 9 = PCIe (SerDes @ 5.0 GT/s),
G Generation <blank> = Initial product generation, B= generation B, C= generation C
- Separator
X Memory Size X = MemFree, 1=128MB, 2=256MB, 3=512MB
B Bracket S = Short, T = Tall, N = None
R RoHS <blank> = non RoHS, C = RoHS R-5 w/ Exemption, R = RoHS R-6 Lead-Free

ConnectX-2 Card User Manual Rev 1.0
Mellanox Technologies 13
1.3 Finding the and Serial Number on the Adapter Cards
All Mellanox adapter cards have a label on the printed side of the adapter card that has the card
serial number.
Figure 2: Card Product Label
1.4 Safety Warnings
For safety warnings in French see “Avertissements de sécurité d’installation (Warnings in French)”
on page 41. For safety warnings in German see “Sicherheitshinweise (Warnings in German)” on
page 42. For safety warnings in Spanish see “Advertencias de seguridad para la instalación (Warn-
ings in Spanish)” on page 43.
1. Installation Instructions
2. Over-temperature
3. During Lightning - Electrical Hazard
Read all installation instructions before connecting the equipment to the power source.
This equipment should not be operated in an area with an ambient temperature exceed-
ing the maximum recommended: 55°C (131°F).
To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ven-
tilation openings.
During periods of lightning activity, do not work on the equipment or connect or dis-
connect cables.
S/N:MT0744X00012
P/N:MHQA19-XTR REV: X1
Made in IL
MAC: 0002C90142 FC

Overview
Rev 1.0
Mellanox Technologies
14
4. Copper Cable Connecting/Disconnecting
5. Equipment Installation
6. Equipment Disposal
7. Local and National Electrical Codes
Some copper cables are heavy and not flexible, as such they should be carefully
attached to or detached from the connectors. Refer to the cable manufacturer for spe-
cial warnings and instructions.
This equipment should be installed, replaced, or serviced only by trained and qualified
personnel.
Disposal of this equipment should be in accordance to all national laws and regula-
tions.
This equipment should be installed in compliance with local and national electrical
codes.

ConnectX-2 Card User Manual Rev 1.0
Mellanox Technologies 15
2Adapter Card Interfaces
2.1 I/O Interfaces
Each card includes the following interfaces:
• I/O panel LEDs
•I
2C compatible connector (for debug)
2.1.1 InfiniBand Interface
The ConnectX®-2 device is compliant with the InfiniBand Architecture Specification, Release 1.2.1.
VPI adapter cards (listed in Table 3 on page 11) based on this device provide access to its port by
means of a QSFP connector.
2.1.2 VPI Port Configuration
The VPI port is auto-sensing but can be manually configured using a script.
Port type management:
By default ConnectX®-2 the VPI port is initialized as an Infiniband port. If you wish to change the
port type use the connectx_port_config script after the driver is loaded.
The script is installed as part of the Mellanox OFED for Linux package (under /sbin). See the Mel-
lanox OFED for Linux User’s Manual available at http://www.mellanox.com => Home > Products
> Adapter IB/VPI SW.
Running "/sbin/connectx_port_config -s" will show the current port configuration for all Con-
nectX®-2 devices.
Port configuration is saved in the file: /etc/infiniband/connectx.conf. This saved configuration is
restored at driver restart only if done via "/etc/init.d/openibd restart".
Possible port types are:
• "eth" - Always Ethernet
• "ib" - Always Infiniband
• "auto" - Link sensing mode - detect port type based on the attached network type. If no link is
detected, the driver retries link sensing every few seconds.
The port link type can be configured for each device in the system at run time using the "/sbin/
connectx_port_config" script. This utility will prompt for the PCI device to be modified (if there is
only one it will be selected automatically). At the next stage the user will be prompted for the
desired mode for each port.

Adapter Card Interfaces
Rev 1.0
Mellanox Technologies
16
The desired port configuration will then be set for the selected device. Note: This utility also has a
non interactive mode:
"/sbin/connectx_port_config [[-d|--device <PCI device ID>] -c|--conf <port1,port2>]".
2.1.3 PCI Express Interface
The ConnectX®-2 adapter cards support PCI Express 2.0 (1.1 compatible) through an x8 edge con-
nector. The device can be either a master initiating the PCI Express bus operations or a slave
responding to PCI bus operations.
2.1.4 LED Assignment
The board has I/O LEDs located on the I/O panel. The green LED, when lit, indicates that the
driver is running and a valid physical connection between nodes exists. If the green LED is blink-
ing, it indicates a problem with the physical link. The yellow LED when lit, indicates a valid data
activity link, this is the logical link. The yellow LED lights up when the network is discovered
over the physical link. A valid data activity link without data transfer is designated by a constant
yellow LED indication. A valid data activity link with data transfer is designated by a blinking yel-
low LED indication. If the LEDs are not active, either the physical link or the logical link (or both)
connections have not been established.
Figure 3: Physical and Logical Link Indications
The short bracket has the same port and LED footprint as the tall bracket.
Table 5 - LEDs
LED Name
Physical Link - Green
Constant on indicates a good physical link
Blinking indicates a problem with the Physical link
Data Activity - Yellow
Blinking indicates Data Transfer
Constant on indicates no Data Transfer

ConnectX-2 Card User Manual Rev 1.0
Mellanox Technologies 17
2.1.5 I2C Compatible Interface
A three-pin header on the adapter card is provided as the I2C compatible interface. See Figure 9 on
page 32 for the location on the board.
Figure 4: I2C Connector
2.2 Power
All adapter cards receive 12V and 3.3V power from the PCI Express Edge connector. All other
required power voltages are generated by on-board switch mode regulators. See “Specifications”
on page 31.
2.3 QSFP Power Level
The card supports power level 3, according to SFF Committee INF-8438i Specification for QSFP
(Quad Small Formfactor Pluggable) Transceiver.
2.4 Memory
The adapter cards support multiple memory devices through the PCI Flash, and I2C compatible
interfaces.
2.4.1 System Memory
The adapter card utilizes the PCI Express interface to store and access fabric and/or Ethernet fabric
connection information and packet data on the system memory.
2.4.2 Flash
Each of the adapter cards include one 16MB SPI Flash device accessible via the Flash interface of
the MT25414B0 ConnectX®-2 VPI device.
There is a jumper on each adapter card that indicates to the device whether an on-board Flash
device is to be used. Table 6 provides information on this jumper. See the schematic in Figure 9 on
page 32 for the jumper location.

Adapter Card Interfaces
Rev 1.0
Mellanox Technologies
18
2.4.3 EEPROM
Each board incorporates an EEPROM that is accessible through the I2C compatible interface. The
EEPROM is used for storing the Vital Product Data (VPD). The EEPROM capacity is 4Kb.
2.5 VPD Layout
The PCI VPD (Vital Product Data) layout, for each of the described Mellanox Technologies VPI
adapter cards complies with the format defined in the PCI 2.3 Specification, Appendix I. All
adapter cards have the same PCI VPD layout.
Table 6 - Jumper Configuration
Description Option Card Default
Configuration Comments
Flash present/
not present connection open – Flash present connection shorted – Flash not present
Figure 5: Flash Jumper
connection open –
Flash present Header 1x2
Table 7 - MHQA19X[STN]R
Offset (Decimal) Item Value Format Description
0 Large Resource Type ID String Tag
(0x02) 0x82
1 Length [7:0] LSB 0x10
2 Length [15:8] MSB 0x0
3 Data KESTREL MINI
QDR STR
19 Large Resource Type VPD-R Tag
(0x10) 0x90
20 Length [7:0] LSB 0x4F
21 Length [15:8] MSB 0x00
22 VPD Keyword PN STR Add in Card Part Number
24 Length 0x15
25 PN PN %STR_SPC
46 VPD Keyword EC STR Engineering Change Level of the card
(rev)
48 Length 0x2
49 Revision RV %STR PCB revision
51 VPD Keyword SN STR Serial Number
53 Length 0x18
54 SerialNumber %STR_SPC “00..00XXXX..XX”

ConnectX-2 Card User Manual Rev 1.0
Mellanox Technologies 19
78 VPD Keyword V0 STR Misc Information
80 Length 0x10
81 Data PCIe Gen2 x8 STR_SPC
97 VPD Keyword RV STR
99 Length 0x1
100 Data 0,99 %CS0
101 Large Resource Type VPD-W Tag
(0x11) 0x91
102 Length [7:0] LSB 0x97
103 Length [15:8] MSB 0x00
104 VPD Keyword V1 STR EFI Driver version
106 Length 0x6
107 Data N/A STR_SPC
113 VPD Keyword YA STR Asset Tag
115 Length 0x20
116 Data N/A STR_SPC “N/A”
148 VPD Keyword RW STR Remaining read/write area
150 Length 0x68
151 Data STR_ZERO Reserved (0x00)
255 Small Resource Type END Tag (0x11) 0x78
Table 7 - MHQA19X[STN]R
Offset (Decimal) Item Value Format Description

Driver Software and Firmware
Rev 1.0
Mellanox Technologies
20
3Driver Software and Firmware
3.1 Driver Software
3.1.1 Linux
For Linux, download and install the latest OpenFabrics Enterprise Distribution (OFED) software
package available via the Mellanox Web site at:
http://www.mellanox.com => Downloads > InfiniBand/VPI SW/Drivers. Follow the installation
instructions included in the download package.
3.1.2 Windows
For Windows, download and install the latest Mellanox WinOF VPI for Windows software pack-
age available via the Mellanox Web site at: http://www.mellanox.com => Downloads > Infini-
Band/VPI SW/Drivers. Follow the installation instructions included in the download package.
3.2 FlexBoot
FlexBoot enables remote boot over Ethernet or InfiniBand using Boot over InfiniBand (BoIB),
Boot over Ethernet (BoE), or Boot over iSCSI (Bo-iSCSI). This technology is based on the Pre-
boot Execution Environment (PXE) standard specification, and FlexBoot software is based on the
open source EtherBoot/gPXE project (see www.etherboot.org). For more information go to
http://www.mellanox.com => Products > Adapter IB/VPI SW/Flexboot.
3.3 Updating Adapter Card Firmware
Each card is shipped with the latest version of qualified firmware at the time of manufacturing.
Firmware is updated occasionally, and the most recent firmware can be obtained from:
http://www.mellanox.com => Downloads > Firmware. Check that the firmware on your card is the
latest found on the Mellanox site, if not update to the latest version found on the Mellanox website.
Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firm-
ware Tools (MFT) package. This package is available for download, along with its user manual,
from the Mellanox Firmware Tools page. See http://www.mellanox.com => Downloads > Firm-
ware Tools.
A firmware binaries table lists a binary file percard. The file name of each such binary is com-
posed by combining the firmware name, the firmware release version, and the card part number.
Please contact Mellanox or your assigned Field Application Engineer if you cannot find the firm-
ware binary for your adapter card.
To check the latest firmware:
1. Go to Mellanox web site to check current firmware version.
Go to: http://www.mellanox.com/supportdownloader
2. Enter your card PSID and compare the firmware versions.