Mellanox Technologies ConnectX MNEH28-XSC User manual

Mellanox Technologies
ConnectX®EN Dual Port Ethernet
Network Interface Cards User’s Manual
P/N: MNEH28-XSC, MNEH28-XTC, MNEH29-XSC, MNEH29-XTC
Rev 0.30

2
Rev 0.30 Mellanox Technologies
© Copyright 2008. Mellanox Technologies, Inc. All Rights Reserved.
Mellanox Technologies, InfiniHost®, and ConnectX®are registered trademarks for Mellanox Technologies, Inc.
All other marks and names mentioned herein may be trademarks of their respective companies.
ConnectX®EN Dual Port Ethernet Adapter Cards User’s Manual
Document Number: 2844
2900 Stender Way
Santa Clara, CA 95054
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies Ltd.
PO Box 586 Hermon Building
Yokneam 20692
Israel
Tel: [phone]
Fax: [phone]

ConnectX EN Dual Port Ethernet Network Interface Cards With PCI Express x8 User’s Manual 3
Rev 0.30
Mellanox Technologies
Table of Contents
Revision History 6
About this Manual 7
Chapter 1 Overview 8
1.1 Adapter Cards 8
1.2 Mellanox Part Numbering Legend 9
1.3 Finding the Mac Address and Serial Number on the Adapter Card 10
Chapter 2 AdapterCard Installation 11
2.1 Hardware and Software Requirements 11
2.2 Installation Instructions 12
Chapter 3 Driver Softwareand Firmware 14
3.1 Driver Software 14
3.2 Updating Adapter Card Firmware 14
3.3 Single Adapter Card Firmware Update 14
Chapter 4 AdapterCardInterfaces 15
4.1 I/O Interfaces 15
4.2 I2C Compatible Interface 17
4.3 Power 17
4.4 Memory 17
4.5 VPDs 18
AppendixA Specifications 22
A.1 Board Mechanical Drawing and Dimensions 22
A.2 EMCCertificationStatements 22
A.3 MNEH28-[XTC,XSC] Specifications 25
A.4 MNEH29-[XTC,XSC] Specifications 26
AppendixB Interface Connectors Pinout 27
B.1 I2C-compatibleConnector Pinout 27
B.2 EthernetConnector Pinout 27
B.3 PCI Express x8 Connector Pinout 28
AppendixC Replacing Existing Bracket on Adapter Cards 29
C.1 RemovingBracket 29
C.2 Placing a Kapton® Polyimide Label 31
C.3 Assembling a New Bracket 33
AppendixD Avertissements de sécurité d’installation 37

4
Rev 0.30 Mellanox Technologies
List of Tables
Table 1: Revision History Table 6
Table 2: Documents List 7
Table 3: Ethernet CX4 Network Interface Cards 8
Table 4: Mellanox Adapter Cards Part Numbering Key 9
Table 5: Hardware and Software Requirements 11
Table 6: Jumper Configuration 17
Table 7: VPD Layout for MNEH28-XTC 18
Table 8: VPD Layout for MNEH28-XSC 19
Table 9: VPD Layout for MNEH29-XTC 19
Table 10: VPD Layout for MNEH29-XSC 21
Table 11: Adapter Cards EMC certification Status 22
Table 12: Specifications for MNEH28-[XTC XSC] 25
Table 13: Specifications for MNEH29-[XTC XSC] 26
Table 14: I2C-Compatible Connector Pinout 27
Table 15: Connector Pin To Port Signal Name 28

ConnectX EN Dual Port Ethernet Network Interface Cards With PCI Express x8 User’s Manual 5
Rev 0.30
Mellanox Technologies
List of Figures
Figure 1: Port Numbering 15
Figure 2: Physical and Logical Link Indications 16
Figure 3: I2C Connector 17
Figure 4: Schematic of the Ethernet Adapter Card With CX4 Connectors 22
Figure 5: I2C-Compatible Connector 27
Figure 6: ConnectX CX4 Copper Connector Pinout 27
Figure 7: Tall Bracket of a Dual IB Port Adapter Card 29
Figure 8: Connector Retention Clip 29
Figure 9: Extracting Connector Clip 30
Figure 10: Unscrew Bracket Screws 30
Figure 11: Rotate the Bracket to Detach it From the Card 31
Figure 12: Hold Kapton Label With Pincers 31
Figure 13: Place Label on Print Side With Label’s and Card’s Holes Aligned 32
Figure 14: Ensure That Label is Well-attached 32
Figure 15: Adapter Card Ready for New Bracket 33
Figure 16: Place Bracket onto Card 33
Figure 17: Attach Bracket onto Card using Screws 34
Figure 18: Sliding Connector Clip Evenly 34
Figure 19: Fix Clip Hooks into Place Using Screwdriver 35
Figure 20: Spread the Hooks to Slide on the Clip 35
Figure 21: Assembled Bracket View 35
Figure 22: Print Side View After Bracket Assembly With Kapton Label 36

6
Rev 0.30 Mellanox Technologies
Revision History
This document was printed on 8/7/08.
Table 1 - Revision History Table
Date Rev Comments/Changes
May 2008 0.3 changes according to MRD Requested by Tar.
April 2008 0.20 Changed link to MLNX OFED page and removed Confidential in footer.
May 2007 0.10 Preliminary first revision
August 2008 1.0 Initial Release

ConnectX EN Dual Port Ethernet Network Interface Cards With PCI Express x8 User’s Manual 7
Rev 0.30
Mellanox Technologies
About this Manual
This User’s Manual describes Mellanox Technologies Ethernet PCI Express Network Interface Cards. It provides
details as to the interfaces of the card, specifications, required software and firmware for operating the card, and rele-
vant documentation.
Intended Audience
This manual is intended for the installer and user of these cards.
The manual assumes basic familiarity with Ethernet networks.
Related Documentation
Table 2 - Documents List
Online Resources
• Mellanox Technologies Web pages: http://www.mellanox.com
• Mellanox Technologies Firmware download Web page: http://www.mellanox.com/ under Firmware downloads
• Mellanox Technologies Document Distribution System (DDS): http://docs.mellanox.com (requires a customer
login account)
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega bytes. The use of
Mb or Mbits (small b) indicates size in mega bits.
ConnectX®EN (MTNIC) PRM
Document Number: DOC18348
Reference describing the interface used by developers to write a device
driver.
ConnectX®EN Hardware Reference Manual
Document Number: 2788HM
Reference for hardware engineers responsible for designing systems and
boards.
Mellanox Firmware Tools (MFT) User’s Manual
Document Number: 2204UG User’s Manual describing the set of MFT firmware management tools. See http://www.mellanox.com under ‘Firmware’ downloads.
PCI Express 2.0 Specifications Industry Standard PCI Express 2.0 Card Electromechanical Specifica-
tion, Rev 1.3.
SFP+ Module Spec sheet
Document Number: 2957 Reference for the Mellanox SFP + Module

8
Rev 0.30 Mellanox Technologies
1 Overview
This document is a User’s Manual for Mellanox Technologies Ethernet Network Interface Cards (NICs) based on the
MT25408 ConnectX®EN integrated circuit device. The cards described in this manual have the following main fea-
tures:
• IEEE 802.3ak compliant
• Two 10GBASE-CX4 copper ports for connecting Ethernet traffic
• PCI Express 2.0 (1.1 compatible) through an x8 edge connector up to 5GT/s
• ‘Media detect circuit’ with powered connectors supporting the use of active cables and external PHY fiber solu-
tions
• EU Restriction of Hazardous Substances (RoHS) compliant
The cards differ in:
• Bracket height: short or tall
• PCI Express 2.0 with SerDes speed: 2.5 GT/s or 5.0 GT/s
1.1 Adapter Cards
Table 3 on page 8 lists the Ethernet NICs described in this manual.
Table 3 - Ethernet CX4 Network Interface Cards
Ordering Part
Number (OPN)
PCI
Express
SERDES
Speed
Data
Transmissio
n Rate
Short /
Tall
Bracket
RoHS
Compliance Adapter IC Part Number NIC Photo (1)
1. The NICs have a similar form and fit. The main visible difference is in the bracket height.
MNEH28-XSC 2.5 GT/s
PCIe
Gen1
20 + 20 Gb/s Short RoHS-R5
(with
exemption)
MT25408A0-FCC-SE
MNEH28-XTC 2.5 GT/s
PCIe
Gen1
Tall
MNEH29-XSC 5.0 GT/s
PCIe
Gen2
40 + 40 Gb/s Short RoHS-R5
(with
exemption)
MT25408A0-FCC-TE
MNEH29-XTC 5.0 GT/s
PCIe
Gen2
Tall
Short Tall

ConnectX EN Dual Port Ethernet Network Interface Cards With PCI Express x8 User’s Manual 9
Rev 0.30
Mellanox Technologies
1.2 Mellanox Part Numbering Legend
Table 4 describes the Mellanox Technologies adapter cards part numbering legend.
For example, the part number MNEH28-XTC describes Mellanox Technologies’ ConnectX EN NIC with dual
10GBASE-CX4 ports, a PCIe2.0 x8 2.5GT/s interface, no on-board memory (mem-free), a tall PCI bracket, and
RoHS R5 compliance. Using the legend,
• field M =M to indicate a Mellanox Technologies product,
• field H = N to indicate a Network Interface Card,
• field T =E to indicate 10GBase-CX4 (copper CX4 with media adapter support),
• field S =H to indicate the ConnectX family,
• field # =2 to indicate two ports,
• field I = 8 to indicate PCI Express 2.0 x8 running at 2.5GT/s,
• field X = X to indicate no on-board memory,
• field B = T to indicate a tall bracket, and
• field R = C to indicate RoHS R5 (w/ Exemptions) compliance
Table 4 - Mellanox Adapter Cards Part Numbering Key
Adapter Card
OPN
MHTS#I-XBR Field Decoder
M Mellanox
Technologies
H Adapter Type H = InfiniBand Host Channel Adapter, N = Ethernet Network Interface Card, S = Express
Module
T Media E = 10GBASE-CX4*, G = 10GBASE-CX4*, K = 10GBASE-SR (XFP),
T = 10GBASE- UTP (Twisted Pair), P = 10GBASE-SR(SFP+)
* = with powered connector
S Silicon H = ConnectX
# # ports 1 = 1, 2 = 2,
I Host Interface 8 = PCIe x8, 9 = PCIe (SerDes @ 5.0 GT/s)
G Generation <blank> = Initial product generation B = 2nd Generation board
- Separator
X Memory Size X = MemFree
B Bracket S = Short, T = Tall, N = None
R RoHS <blank> = non RoHS, C = RoHS w/ Exemption, R = RoHS Lead-Free

10
Rev 0.30 Mellanox Technologies
1.3 Finding the Mac Address and Serial Number on the Adapter Card
All Mellanox Ethernet NICs have a label on the printed side of the adapter card that has the card serial number and
the card MAC address.
Serial
number
MAC
address

ConnectX EN Dual Port Ethernet Network Interface Cards With PCI Express x8 User’s Manual 11
Rev 0.30
Mellanox Technologies
2 Adapter Card Installation
2.1 Hardware and Software Requirements
Before installing the NIC, please make sure that the system meets the hardware and software requirements listed in
Table 5.
Table 5 - Hardware and Software Requirements
Requirement Description
Hardware • Minimum 3 GB of available space
• PCI Express x8 or x16 slots
Software
Operating
Systems/Distributions
• For Windows see WinIB ReadMe at https://docs.mellanox.com/dm/WinIB/
ReadMe.html
• For Linux see Mellanox OpenFabrics Enterprise Distribution (OFED) software
package available via the Mellanox OpenFabrics Web site http://www.mel-
lanox.com/products/ofed.php
• Novell SuSE Linux Enterprise server (SLES), Red Hat Enterprise Linux (RHEL),
and other Linux distributions
• Microsoft Windows Server2003/2008, Windows Compute Cluster Server 2003

12
Rev 0.30 Mellanox Technologies
2.2 Installation Instructions
2.2.1 Installation Safety Warnings
1. Installation Instructions
2. Over-temperature
3. During Lightning - Electrical Hazard
4. Copper InfiniBand Cable Connecting/Disconnecting
5. Equipment Installation
6. Equipment Disposal
7. Local and National Electrical Codes
Read all installation instructions before connecting the equipment to the power source.
This equipment should not be operated in an area with an ambient temperature exceed-
ing the maximum recommended:55°C (131°F). Moreover, to guarantee proper air flow,
allow at least 8cm (3 inches) of clearance around the ventilation openings.
During periods of lightning activity, do not work on the equipment or connect or discon-
nect cables.
Copper InfiniBand cables are heavy and not flexible, as such they should be carefully
attached to or detached from the connectors. Refer to the cable manufacturer for special
warnings and instructions.
This equipment should be installed, replaced, or serviced only by trained and qualified
personnel.
Disposal of this equipment should be in accordance to all national laws and regulations.
This equipment should be installed in compliance with local and national electrical
codes.

ConnectX EN Dual Port Ethernet Network Interface Cards With PCI Express x8 User’s Manual 13
Rev 0.30
Mellanox Technologies
2.2.2 Installation Instructions
The adapter cards listed in Table 3 on page 8 are standard PCI Express x8 cards each with a standard x8 edge connec-
tor. Please consult the host machine documentation for instructions on how to install a PCI Express card.
Note: When more than one PCI Slot is available first make sure to use the PCI slot with the proper configu-
ration.
Any PCI slot with the proper configuration is acceptable for connection. If the card is installed in a PCI slot with less
lanes than the card requires then the adapter card will not provide the optimum data transfer.

14
Rev 0.30 Mellanox Technologies
3 Driver Software and Firmware
Drivers and Firmware can be downloaded and installed by using the Mellanox OFED download site within the Mel-
lanox Website.
Note: The installation requires administrator privileges on the target machine.
3.1 Driver Software
For Linux, download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available
via the Mellanox OpenFabrics Web site at http://www.mellanox.com/products/ofed.php. Follow the installation
instructions included in the download package.
For Windows, download the appropriate software from https://docs.mellanox.com/dm/WinIB/ReadMe.html.
3.2 Updating Adapter Card Firmware
Each adapter card is shipped with the latest version of qualified firmware at the time of manufacturing. Firmware is
updated occasionally, and the most recent firmware can be obtained from http://www.mellanox.com through the
‘Firmware’ downloads link.
3.3 Single Adapter Card Firmware Update
Firmware can be updated on the stand-alone single card using the flint tool of the Mellanox Firmware Tools (MFT)
package. This package is available for download, along with its user’s manual, from the single adapter card firmware
update page. See http://www.mellanox.com under ‘Firmware’ downloads.
A firmware binaries table lists a binary file per adapter card. The file name of each such binary is composed by com-
bining the firmware name, the firmware release version, and the card part number.
Note: Please contact your assigned Field Application Engineer if you cannot find the firmware binary for
your adapter card. This may happen if the product is not yet available for general distribution.

ConnectX EN Dual Port Ethernet Network Interface Cards With PCI Express x8 User’s Manual 15
Rev 0.30
Mellanox Technologies
4 Adapter Card Interfaces
4.1 I/O Interfaces
Each adapter card includes the following interfaces:
• Two 4X copper connectors
• PCI Express x8 edge connector
• I/O panel LEDs
•I
2C compatible connector (for debug)
4.1.1 Ethernet CX4 Interface
The ConnectX®EN 10GBase-CX4 device is compliant with the IEEE Std 802.3ak 10GBASE-CX4. It has two com-
pliant 4X ports (1 and 2), each having four Tx/Rx pairs of SerDes. Each of the cards (listed in Table 3 on page 8)
based on this device provides access to these ports by means of two 4X connectors for external copper cables. Con-
nector 1 connects to port 1 of the device, while connector 2 connects to port 2.
Figure 1: Port Numbering
Each of the NIC cards is embedded with a ‘media detect circuit’ that supports active cables and external fiber solu-
tions to be connected to the port connectors. Fiber Solutions require the use of active media converters.
4.1.2 PCI Express Interface
The ConnectX®EN 10GBASE-CX4 adapter cards support the PCI Express 2.0 x8 interface, 1.1 compatible. The
NICs can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations.
4.1.3 LED Assignment
The board has four LEDs located on the I/O panel - 2 LEDs per port. The green LED, when lit, indicates that the
driver is running and a valid physical connection between nodes exists. The green LED, when blinking, indicates that
the physical connection between nodes is problematic. The yellow LED when lit, indicates a valid data activity link,
this is the logical link. The yellow LED illuminates when the network is discovered over the physical link. A valid
Port 1
Port 2

16
Rev 0.30 Mellanox Technologies
data activity link without data transfer is designated by a constant yellow LED indication. A valid data activity link
with data transfer is designated by a blinking yellow LED indication. If the LEDs are not active, either the physical
link or the logical link (or both) connections have not been established.
Figure 2: Physical and Logical Link Indications
Note: The short bracket has the same port and LED footprint as the tall bracket.
Port 1
Port 2
Port Number LED Name
Port 1 Physical Link - Green
Constant on indicates a good physical link
Blinking indicates a problem with the Physical link
Data Activity - Yellow
Blinking indicates Data Transfer
Constant on indicates no Data Transfer
Port 2 Physical Link - Green
Constant on indicates a good physical link
Blinking indicates a problem with the Physical link
Data Activity - Yellow
Blinking indicates Data Transfer
Constant on indicates no Data Transfer

ConnectX EN Dual Port Ethernet Network Interface Cards With PCI Express x8 User’s Manual 17
Rev 0.30
Mellanox Technologies
4.2 I2C Compatible Interface
A three-pin header on the adapter card is provided as the I2C compatible interface. See Figure 4 on page 22 for the
location on the board.
Figure 3: I2C Connector
4.3 Power
All adapter cards receive power from the PCI Express Edge connector. All other required power voltages are gener-
ated by on-board switch mode regulators. For power consumption see Specifications starting on page 22.
4.4 Memory
The NICs support multiple memory devices through the PCI Express, Flash, and I2C-compatible interfaces.
4.4.1 System Memory
Each of the NICs utilize the PCI Express interface to store and access connection information and packet data on the
system memory.
4.4.2 Flash
Each of the NICs includes one 2MB SPI Flash device (P/N M25P16-VME6G by ST Microelectronics) accessible via
the Flash interface of the MT25408 ConnectX EN device.
There is a jumper on each adapter card that indicates to the device whether an on-board Flash device exists (or is to be
used). Table 6 provides information on this jumper. See Figure 4 on page 22 for the jumper location.
4.4.3 EEPROM
Each board incorporates an EEPROM that is accessible through the I2C-compatible interface. The EEPROM is used
for storing the Vital Product Data (VPD). The VPD format adheres to the PCI Local Bus specification rev 2.3 VPD
definition (see “VPDs” on page 18). The EEPROM capacity is 512 bytes.
Table 6 - Jumper Configuration
Description Option Card Default
Configuration Comments
Flash present/ not present connection open – Flash present
connection shorted – Flash not present connection open – Flash present Header 1x2

18
Rev 0.30 Mellanox Technologies
4.5 VPDs
• The PCI VPD (Vital Product Data) layout for each of the described Mellanox Technologies ConnectX®EN
10GBASE-CX4 NICs cards comply with the format defined in the PCI 2.3.
Table 7 - VPD Layout for MNEH28-XTC
Offset
(Decimal) Item Value Format Description
0 Large Resource Type ID String Tag (0x02) 0x82
1 Length [7:0] LSB 0xD
2 Length [15:8] MSB 0x0
3DataEagle10G-ETHSTR
16 Large Resource Type VPD-R Tag (0x10) 0x90
17 Length [7:0] LSB 0x4F
18 Length [15:8] MSB 0x00
19 VPD Keyword PN STR Add in Card Part Number
21 Length 0x15
22 PN MNEH28-XTC %STR_SP
C
43 VPD Keyword EC STR Engineering Change Level of
the card (rev)
45 Length 0x2
46 Revision A1 %STR PCB revision
48 VPD Keyword SN STR Serial Number
50 Length 0x18
51 Serial Number %STR_SP
C“00..00XXXX..XX”
75 VPD Keyword V0 STR Misc. Information
77 Length 0x10
78 Data PCIe x8 STR_SPC
94 VPD Keyword RV STR
96 Length 0x1
97 Data 0,96 %CS0
98 Large Resource Type VPD-W Tag (0x11) 0x91
99 Length [7:0] LSB 0x9A
100 Length [15:8] MSB 0x00
101 VPD Keyword V1 STR EFI Driver version
103 Length 0x6
104 Data N/A STR_SPC
110 VPD Keyword YA STR Asset Tag
112 Length 0x20
113 Data N/A STR_SPC “N/A”
145 VPD Keyword RW STR Remaining read/write area
147 Length 0x6B
148 Data STR_ZER
OReserved (0x00)
255 Small Resource Type END Tag (0x11) 0x78

ConnectX EN Dual Port Ethernet Network Interface Cards With PCI Express x8 User’s Manual 19
Rev 0.30
Mellanox Technologies
Table 8 - VPD Layout for MNEH28-XSC
Offset
(Decimal) Item Value Format Description
0 Large Resource Type ID String Tag (0x02) 0x82
1 Length [7:0] LSB 0xD
2 Length [15:8] MSB 0x0
3 Data Eagle 10G-ETH STR
16 Large Resource Type VPD-R Tag (0x10) 0x90
17 Length [7:0] LSB 0x4F
18 Length [15:8] MSB 0x00
19 VPD Keyword PN STR Add in Card Part Number
21 Length 0x15
22 PN MNEH28-XSC %STR_SPC
43 VPD Keyword EC STR Engineering Change Level of the
card (rev)
45 Length 0x2
46 Revision A1 %STR PCB revision
48 VPD Keyword SN STR Serial Number
50 Length 0x18
51 Serial Number %STR_SPC “00..00XXXX..XX”
75 VPD Keyword V0 STR Misc. Information
77 Length 0x10
78 Data PCIe x8 STR_SPC
94 VPD Keyword RV STR
96 Length 0x1
97 Data 0,96 %CS0
98 Large Resource Type VPD-W Tag (0x11) 0x91
99 Length [7:0] LSB 0x9E
100 Length [15:8] MSB 0x00
101 VPD Keyword V1 STR EFI Driver version
103 Length 0x6
104 Data N/A STR_SPC
110 VPD Keyword YA STR Asset Tag
112 Length 0x20
113 Data N/A STR_SPC “N/A”
145 VPD Keyword RW STR Remaining read/write area
147 Length 0x6F
148 Data STR_ZERO Reserved (0x00)
259 Small Resource Type END Tag (0x11) 0x78
Table 9 - VPD Layout for MNEH29-XTC
Offset
(Decimal) Item Value Format Description
0 Large Resource Type ID String Tag
(0x02) 0x82

20
Rev 0.30 Mellanox Technologies
1 Length [7:0] LSB 0xD
2 Length [15:8] MSB 0x0
3 Data Eagle 10G-
ETH STR
16 Large Resource Type VPD-R Tag (0x10) 0x90
17 Length [7:0] LSB 0x4F
18 Length [15:8] MSB 0x00
19 VPD Keyword PN STR Add in Card Part Number
21 Length 0x15
22 PN PN %STR_SPC
43 VPD Keyword EC STR Engineering Change Level of the card
(rev)
45 Length 0x2
46 Revision X1 %STR PCB revision
48 VPD Keyword SN STR Serial Number
50 Length 0x18
51 Serial Number %STR_SPC “00..00XXXX..XX”
75 VPD Keyword V0 STR Misc. Information
77 Length 0x10
78 Data PCIe Gen2 x8 STR_SPC
94 VPD Keyword RV STR
96 Length 0x1
97 Data 0,96 %CS0
98 Large Resource Type VPD-W Tag (0x11) 0x91
99 Length [7:0] LSB 0x9A
100 Length [15:8] MSB 0x00
101 VPD Keyword V1 STR EFI Driver version
103 Length 0x6
104 Data N/A STR_SPC
110 VPD Keyword YA STR Asset Tag
112 Length 0x20
113 Data N/A STR_SPC “N/A”
145 VPD Keyword RW STR Remaining read/write area
147 Length 0x6B
148 Data STR_ZERO Reserved (0x00)
255 Small Resource Type END Tag (0x11) 0x78
Table 9 - VPD Layout for MNEH29-XTC (Continued)
Offset
(Decimal) Item Value Format Description