Nvidia Mellanox ConnectX-6 MCX651105A-EDAT User manual

Exported on Nov/09/2023 12:30 PM
https://docs.nvidia.com/networking/x/zYWmBQ







NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards User
Manual



2
Table of Contents
Introduction........................................................................................................................................... 11
Product Overview ...............................................................................................................................................11
ConnectX-6 PCIe x8 Card .................................................................................................................................. 12
ConnectX-6 PCIe x16 Card ................................................................................................................................. 14
ConnectX-6 DE PCIe x16 Card ............................................................................................................................. 15
ConnectX-6 for Liquid-Cooled Intel® Server System D50TNP Platforms............................................................................ 16
ConnectX-6 Socket Direct™ Cards ........................................................................................................................ 17
ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe x16) ......................................................................................... 17
ConnectX-6 Single-slot Socket Direct Cards (2x PCIe x8 in a row) .............................................................................. 19
Package Contents ...............................................................................................................................................20
ConnectX-6 PCIe x8/x16 Adapter Cards ................................................................................................................. 20
ConnectX-6 PCIe x16 Adapter Card for liquid-cooled Intel® Server System D50TNP Platforms ................................................ 21
ConnectX-6 Socket Direct Cards (2x PCIe x16) ......................................................................................................... 21
Features and Benefits ..........................................................................................................................................22
Operating Systems/Distributions .............................................................................................................................25
Connectivity .....................................................................................................................................................25
Manageability....................................................................................................................................................26
Interfaces ............................................................................................................................................. 27
InfiniBand Interface ............................................................................................................................................27

3
Ethernet Interfaces.............................................................................................................................................27
PCI Express Interface...........................................................................................................................................27
LED Interface ....................................................................................................................................................28
Heatsink Interface ..............................................................................................................................................30
SMBus Interface .................................................................................................................................................30
Voltage Regulators ..............................................................................................................................................30
Hardware Installation ............................................................................................................................... 31
Safety Warnings .................................................................................................................................................31
Installation Procedure Overview..............................................................................................................................32
System Requirements ..........................................................................................................................................33
Hardware Requirements ................................................................................................................................... 33
Airflow Requirements ..........................................................................................................................................34
Software Requirements .................................................................................................................................... 35
Safety Precautions ..............................................................................................................................................35
Pre-Installation Checklist ......................................................................................................................................36
Bracket Replacement Instructions ...........................................................................................................................36
Installation Instructions........................................................................................................................................37
Cables and Modules.........................................................................................................................................38
Identifying the Card in Your System..........................................................................................................................39
PCIe x8/16 Cards Installation Instructions ..................................................................................................................42
Installing the Card .......................................................................................................................................... 42

4
Uninstalling the Card.......................................................................................................................................44
Socket Direct (2x PCIe x16) Cards Installation Instructions ..............................................................................................46
Installing the Card .......................................................................................................................................... 47
Uninstalling the Card.......................................................................................................................................57
Cards for Intel Liquid-Cooled Platforms Installation Instructions .......................................................................................59
Installing the Card .......................................................................................................................................... 59
Driver Installation.................................................................................................................................... 65
Linux Driver Installation .......................................................................................................................................65
Prerequisites................................................................................................................................................. 65
Downloading NVIDIA OFED ................................................................................................................................. 66
Installing MLNX_OFED ...................................................................................................................................... 67
Installation Script ....................................................................................................................................... 67
Installation Procedure.................................................................................................................................. 70
Installation Results ..................................................................................................................................... 73
Installation Logging..................................................................................................................................... 73
Driver Load Upon System Boot............................................................................................................................ 74
mlnxofedinstall Return Codes......................................................................................................................... 74
Installation Logging..................................................................................................................................... 75
Uninstalling MLNX_OFED ............................................................................................................................... 75
Additional Installation Procedures ....................................................................................................................... 76
Installing MLNX_OFED Using YUM..................................................................................................................... 76

5
Installing MLNX_OFED Using apt-get ................................................................................................................. 82
Performance Tuning ........................................................................................................................................ 86
Windows Driver Installation ...................................................................................................................................86
Software Requirements .................................................................................................................................... 87
Downloading WinOF-2 Driver.............................................................................................................................. 87
Installing WinOF-2 Driver .................................................................................................................................. 88
Attended Installation................................................................................................................................... 88
Unattended Installation................................................................................................................................ 98
Firmware Upgrade ........................................................................................................................................ 100
VMware Driver Installation .................................................................................................................................. 100
Software Requirements .................................................................................................................................. 100
Installing NATIVE ESXi Driver for VMware vSphere ................................................................................................... 101
Removing Earlier NVIDIA Drivers ........................................................................................................................ 101
Firmware Programming .................................................................................................................................. 102
Troubleshooting .....................................................................................................................................103
General Troubleshooting..................................................................................................................................... 103
Linux Troubleshooting ........................................................................................................................................ 104
Windows Troubleshooting.................................................................................................................................... 105
Updating Adapter Firmware .......................................................................................................................106
Monitoring............................................................................................................................................108
Specifications........................................................................................................................................109

6
MCX651105A-EDAT Specifications........................................................................................................................... 109
MCX653105A-HDAT Specifications .......................................................................................................................... 111
MCX653106A-HDAT Specifications .......................................................................................................................... 113
MCX653105A-HDAL Specifications .......................................................................................................................... 115
MCX653106A-HDAL Specifications .......................................................................................................................... 117
MCX653105A-ECAT Specifications........................................................................................................................... 119
MCX653106A-ECAT Specifications........................................................................................................................... 121
MCX654105A-HCAT Specifications .......................................................................................................................... 123
MCX654106A-HCAT Specifications .......................................................................................................................... 125
MCX654106A-ECAT Specifications........................................................................................................................... 128
MCX653105A-EFAT Specifications ........................................................................................................................... 130
MCX653106A-EFAT Specifications ........................................................................................................................... 132
MCX683105AN-HDAT Specifications ........................................................................................................................ 134
Adapter Card and Bracket Mechanical Drawings and Dimensions ..................................................................................... 136
Adapter Cards ............................................................................................................................................. 136
Brackets Dimensions...................................................................................................................................... 138
PCI Express Pinouts Description for Single-Slot Socket Direct Card .........................................................................140
Finding the GUID/MAC on the Adapter Card ....................................................................................................141
Document Revision History ........................................................................................................................143



7
About This Manual
This User Manual describes NVIDIA® ConnectX®-6 InfiniBand/Ethernet adapter cards. It provides details as to the interfaces of the board, specifications,
required software and firmware for operating the board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 InfiniBand/Ethernet adapter cards.
NVIDIA SKU Legacy OPN Marketing Description Lifecycle
900-9X603-0016-DT0 MCX653105A-EFAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE), single-port QSFP56,
PCIe3.0/4.0 Socket Direct 2x8 in a row, tall bracket
Mass Production
900-9X603-0056-DT0 MCX653106A-EFAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IBand100GbE), dual-port QSFP56, PCIe3.0/4.0
Socket Direct 2x8 in a row, tall bracket
Mass Production
900-9X628-0016-ST0 MCX651105A-EDAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE, single-port QSFP56, PCIe4.0 x8,
tall bracket
Mass Production
900-9X6AF-0016-ST1 MCX653105A-ECAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE), single-port QSFP56,
PCIe3.0/4.0 x16, tall bracket
Mass Production
900-9X6AF-0056-MT1 MCX653106A-ECAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE), dual-port QSFP56, PCIe3.0/4.0
x16, tall bracket
Mass Production
900-9X6AF-0018-MT2 MCX653105A-HDAT ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, single-port QSFP56, PCIe3.0/4.0 x16,
tall bracket
Mass Production
900-9X6AF-0058-ST1 MCX653106A-HDAT ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, dual-port QSFP56, PCIe3.0/4.0 x16,
tall bracket
Mass Production
900-9X6B4-0058-DT0 MCX654106A-HCAT ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, dual-port QSFP56, Socket Direct 2x
PCIe3.0/4.0x16, tall bracket
Mass Production

8
NVIDIA SKU Legacy OPN Marketing Description Lifecycle
900-9X6AF-0018-SS0 MCX653105A-HDAL ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, single-port QSFP56, PCIe3.0/4.0 x16,
cold plate for liquid-cooledIntel® Server System D50TNP
platforms, tall bracket, ROHS R6
Mass Production
900-9X6AF-0058-SS0 MCX653106A-HDAL ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, dual-port QSFP56, PCIe3.0/4.0 x16,
cold plate for liquid-cooledIntel® Server System D50TNP
platforms, tall bracket, ROHS R6
Mass Production
900-9X0BC-001H-ST1 MCX683105AN-HDAT ConnectX®-6 DE adapter card, HDR IB (200Gb/s), single-port
QSFP, PCIe4.0 x16,no crypto, tall bracket
Mass Production
EOL'ed (End of Life) Ordering Part Numbers
NVIDIA SKU Legacy OPN Marketing Description Lifecycle
900-9X6B4-0056-DT0 MCX654106A-ECAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR InfiniBand and 100GbE), dual-port QSFP56,
Socket Direct 2x PCIe 3.0/4.0 x16, tall bracket
End of Life
900-9X6B4-0018-DT2 MCX654105A-HCAT ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, single-port QSFP56, Socket Direct 2x
PCIe3.0/4.0x16, tall bracket
End of Life
Intended Audience
This manual is intended for the installer and user of these cards.
The manual assumes basic familiarity with InfiniBand and Ethernet network and architecture specifications.
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough the following methods:

9
•
•
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchased NVIDIA M-1 Global Support Services, please see your contract fordetails regarding Technical Support.
Customers who purchased NVIDIA products through a NVIDIA approved reseller should first seekassistance through their reseller.
Related Documentation
MLNX_OFED for Linux User Manual and Release Notes User Manual describing OFED features, performance, band diagnostic, tools content, and configuration.
SeeMLNX_OFED for Linux Documentation.
WinOF-2 for WindowsUser Manualand Release Notes User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools content, and configuration.
SeeWinOF-2 for Windows Documentation.
NVIDIA VMware for Ethernet User Manual User Manual and release notes describing the various components of the NVIDIA ConnectX® NATIVE ESXi stack.
SeeVMware® ESXi Drivers Documentation.
NVIDIA Firmware Utility (mlxup) User Manual and Release Notes NVIDIA firmware update and query utility used to update the firmware. Refer toFirmware Utility (mlxup)
Documentation.
NVIDIA Firmware Tools (MFT) User Manual User Manual describing the set of MFT firmware management tools for a single node. SeeMFT User Manual.
InfiniBand Architecture Specification Release 1.2.1, Vol 2 - Release 1.3 InfiniBand Specifications
IEEE Std 802.3 Specification IEEE Ethernet Specifications
PCI Express Specifications Industry Standard PCI Express Base and Card Electromechanical Specifications. Refer toPCI-SIG Specifications.
LinkX Interconnect Solutions LinkX InfiniBand cables and transceivers are designed to maximize the performance of High-Performance
Computing networks, requiring high-bandwidth, low-latency connections between compute nodes and switch
nodes.NVIDIA offers one of the industry’s broadest portfolio ofQDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/HDR100
(100Gb/s), HDR (200Gb/s)and NDR (400Gb/s) cables, includingDirect Attach Copper cables (DACs), copper splitter
cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths from 0.5m to 10km.In addition to
meeting IBTA standards, NVIDIAtests every product in an end-to-end environment ensuring a Bit Error Rate of less
than 1E-15.Read more atLinkX Cables and Transceivers.
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in MegaBytes. The use of Mb or Mbits (small b) indicates the size in
MegaBits. In this document, PCIe is used to mean PCI Express.
Revision History

10
A list of the changes made to this document are provided inDocument Revision History.