Nvidia Mellanox ConnectX-6 MCX654106A-ECAT User manual

Exported onOct/22/2020 07:16 PM
https://docs.mellanox.com/x/NCQuAQ
NVIDIA Mellanox ConnectX-6 InfiniBand/VPI
Adapter Cards User Manual

2
Table of Contents
Table of Contents .......................................................................................................2
Introduction................................................................................................................9
Product Overview .................................................................................................................... 9
ConnectX-6 PCIe x8 Card...................................................................................................... 10
ConnectX-6 PCIe x16 Card.................................................................................................... 11
ConnectX-6 Socket Direct™ Cards....................................................................................... 11
ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe x16) .............................................. 12
ConnectX-6 Single-slot Socket Direct Cards (2x PCIe x8 in a row) ............................... 13
Package Contents ................................................................................................................. 14
ConnectX-6 PCIe x8/x16 Adapter Cards.......................................................................... 14
ConnectX-6 Socket Direct Cards (2x PCIe x16)............................................................... 14
Features and Benefits........................................................................................................... 15
Operating Systems/Distributions .................................................................................... 17
Connectivity ...................................................................................................................... 17
Manageability ................................................................................................................... 17
Interfaces .................................................................................................................18
InfiniBand Interface............................................................................................................... 18
Ethernet QSFP56 Interfaces ................................................................................................. 18
PCI Express Interface ........................................................................................................... 18
LED Interface......................................................................................................................... 18
Heat Sink Interface................................................................................................................ 19
SMBus Interface.................................................................................................................... 20
Voltage Regulators................................................................................................................ 20
Hardware Installation ..............................................................................................21
Safety Warnings..................................................................................................................... 21
Installation Procedure Overview........................................................................................... 21
System Requirements........................................................................................................... 22
Hardware Requirements ................................................................................................. 22
Airflow Requirements ........................................................................................................... 22
Software Requirements ................................................................................................... 23
Safety Precautions ................................................................................................................ 23
Pre-Installation Checklist..................................................................................................... 24

3
Bracket Replacement Instructions ...................................................................................... 24
Installation Instructions........................................................................................................ 25
Cables and Modules......................................................................................................... 25
Identifying the Card in Your System ..................................................................................... 26
ConnectX-6 PCIe x8/16 Installation Instructions................................................................. 27
Installing the Card............................................................................................................ 27
Uninstalling the Card ....................................................................................................... 29
ConnectX-6 Socket Direct (2x PCIe x16) Installation Instructions ...................................... 30
Installing the Card............................................................................................................ 31
Uninstalling the Card ....................................................................................................... 37
Driver Installation ....................................................................................................39
Linux Driver Installation........................................................................................................ 39
Prerequisites .................................................................................................................... 39
Downloading Mellanox OFED .......................................................................................... 39
Installing Mellanox OFED ................................................................................................ 41
Installation Script ........................................................................................................ 41
Installation Procedure ................................................................................................ 42
Installation Results .....................................................................................................44
Installation Logs..........................................................................................................44
openibd Script..............................................................................................................44
Driver Load Upon System Boot ..................................................................................45
mlnxofedinstall Return Codes....................................................................................45
Uninstalling MLNX_OFED...........................................................................................46
Installing MLNX_OFED Using YUM ................................................................................. 46
Setting up MLNX_OFED YUM Repository...................................................................46
Installing MLNX_OFED Using the YUM Tool .............................................................. 47
Uninstalling MLNX_OFED Using the YUM Tool .........................................................48
Installing MLNX_OFED Using apt-get Tool..................................................................... 48
Setting up MLNX_OFED apt-get Repository ..............................................................48
Installing MLNX_OFED Using the apt-get Tool..........................................................48
Uninstalling MLNX_OFED Using the apt-get Tool.....................................................49
Updating Firmware After Installation.............................................................................. 49
Updating the Device Online.........................................................................................49
Updating the Device Manually ....................................................................................50

4
Updating the Device Firmware Automatically upon System Boot ............................50
UEFI Secure Boot............................................................................................................. 51
Enrolling Mellanox's x.509 Public Key on Your Systems...........................................51
Removing Signature from kernel Modules ................................................................51
Performance Tuning ........................................................................................................ 52
Windows Driver Installation.................................................................................................. 52
Software Requirements ................................................................................................... 53
Downloading Mellanox WinOF-2 Driver .......................................................................... 53
Attended Installation...................................................................................................54
Unattended Installation...............................................................................................58
Installation Results .....................................................................................................59
Uninstalling Mellanox WinOF-2 Driver............................................................................ 60
Attended Uninstallation ..............................................................................................60
Unattended Uninstallation..........................................................................................60
Extracting Files Without Running Installation ................................................................ 60
Firmware Upgrade ........................................................................................................... 63
VMware Driver Installation ................................................................................................... 63
Hardware and Software Requirements........................................................................... 63
Installing Mellanox NATIVE ESXi Driver for VMware vSphere........................................ 64
Removing Earlier Mellanox Drivers................................................................................. 64
Firmware Programming .................................................................................................. 65
Updating Adapter Firmware ....................................................................................66
Troubleshooting .......................................................................................................67
GeneralTroubleshooting ...................................................................................................... 67
LinuxTroubleshooting .......................................................................................................... 67
WindowsTroubleshooting..................................................................................................... 68
Specifications ...........................................................................................................69
MCX651105A-EDAT Specifications ...................................................................................... 69
MCX653105A-HDAT Specifications....................................................................................... 70
MCX653106A-HDAT Specifications....................................................................................... 71
MCX653105A-ECAT Specifications ....................................................................................... 73
MCX653106A-ECAT Specifications ....................................................................................... 74
MCX654105A-HCAT Specifications....................................................................................... 75
MCX654106A-HCAT Specifications....................................................................................... 77

5
MCX654106A-ECAT Specifications ....................................................................................... 78
MCX653105A-EFAT Specifications........................................................................................ 80
MCX653106A-EFAT Specifications........................................................................................ 81
Adapter Card and Bracket Mechanical Drawings and Dimensions.................................... 82
ConnectX-6 PCIe x16 Adapter Card................................................................................ 83
ConnectX-6 PCIe x8 Adapter Card.................................................................................. 83
Auxiliary PCIe Connection Card...................................................................................... 84
Tall Bracket ...................................................................................................................... 84
Short Bracket ................................................................................................................... 84
PCIe Express Pinouts Description for Single-Slot Socket Direct Card....................85
Finding the GUID/MAC on the Adapter Card ............................................................86
Document Revision History ......................................................................................88

6
•
•
About This Manual
This User Manual describes NVIDIA® Mellanox® ConnectX®-6 InfiniBand/VPI adapter cards. It
provides details as to the interfaces of the board, specifications, required software and firmware for
operating the board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 InfiniBand/VPI
adapter cards.
OPN Marketing Description
MCX654106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR
InfiniBand and 100GbE), dual-port QSFP56, Socket Direct 2x
PCIe 3.0/4.0 x16, tall bracket
MCX653105A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and
100GbE), single-port QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in
a row, tall bracket
MCX653106A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR
IBand100GbE), dual-port QSFP56, PCIe3.0/4.0 Socket Direct
2x8 in a row, tall bracket
MCX651105A-EDAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and
100GbE, single-port QSFP56, PCIe4.0 x8, tall bracket
MCX653105A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and
100GbE), single-port QSFP56, PCIe3.0/4.0 x16, tall bracket
MCX653106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and
100GbE), dual-port QSFP56, PCIe3.0/4.0 x16, tall bracket
MCX653105A-HDAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,
single-port QSFP56, PCIe3.0/4.0 x16, tall bracket
MCX653106A-HDAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,
dual-port QSFP56, PCIe3.0/4.0 x16, tall bracket
MCX654105A-HCAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,
single-port QSFP56, Socket Direct 2x PCIe3.0/4.0x16, tall
bracket
MCX654106A-HCAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,
dual-port QSFP56, Socket Direct 2x PCIe3.0/4.0x16, tall
bracket
Intended Audience
This manual is intended for the installer and user of these cards.
The manual assumes basic familiarity with InfiniBand and Ethernet network and architecture
specifications.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact usthrough
the following methods:
URL:http://www.mellanox.com> Support
E-mail:[email protected]

7
•Tel: [phone]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails
regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seekassistance through their reseller.
Related Documentation
Mellanox OFED
for Linux User
Manual and
Release Notes
User Manual describing OFED features, performance, band diagnostic, tools content and
configuration. SeeMellanox OFED for Linux Documentation.
WinOF-2 for
WindowsUser
Manual and
Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools
content and configuration. SeeWinOF-2 for Windows Documentation.
Mellanox
VMware for
Ethernet User
Manual
User Manual describing the various components of the Mellanox ConnectX® NATIVE
ESXi stack. See http://www.mellanox.comProducts > Software > Ethernet Drivers >
VMware Driver > User Manual
Mellanox
VMware for
Ethernet Release
Notes
Release notes for Mellanox ConnectX® NATIVE ESXi stack. See http://
www.mellanox.comSoftware > Ethernet Drivers > VMware Driver > Release Notes
Mellanox
Firmware Utility
(mlxup) User
Manual and
Release Notes
Mellanox firmware update and query utility used to update the firmware. Seehttp://
www.mellanox.com> Products > Software > Firmware Tools > mlxup Firmware Utility
Mellanox
Firmware Tools
(MFT) User
Manual
User Manual describing the set of MFT firmware management tools for a single node.
SeeMFT User Manual.
IEEE Std 802.3
Specification
IEEE Ethernet specification at http://standards.ieee.org
PCI Express
3.0/4.0
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications at
https://pcisig.com/specifications

8
Mellanox LinkX
Interconnect
Solutions
Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the
performance of High-Performance Computing networks, requiring high-bandwidth, low-
latency connections between compute nodes and switch nodes. Mellanox offers one of
industry’s broadest portfolio of QDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/HDR100
(100Gb/s) and HDR (200Gb/s) cables, including Direct Attach Copper cables (DACs),
copper splitter cables, Active Optical Cables (AOCs) and transceivers in a wide range of
lengths from 0.5m to 10km. In addition to meeting IBTA standards,Mellanox tests every
product in an end-to-end environment ensuring a Bit Error Rate of less than 1E-15.Read
more at https://www.mellanox.com/products/interconnect/infiniband-overview.php
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits.IB is used in this document to mean
InfiniBand. In this document PCIe is used to mean PCI Express.
Revision History
A list of the changes made to this document are provided inDocument Revision History.

9
Introduction
Product Overview
This is the user guide for Mellanox technologies VPI adapter cards based on the ConnectX®-6
integrated circuit device. ConnectX-6 connectivity provides the highest performing low latency and
most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in enterprise datacenters
and high-performance computing environments.
ConnectX-6 Virtual Protocol Interconnect® adapter cards provide up to two ports of 200Gb/s for
InfiniBand and Ethernet connectivity, sub-600ns latency and 200 million messages per second,
enabling the highest performance and most flexible solution for the most demanding High-
Performance Computing (HPC), storage, and datacenter applications.
ConnectX-6 is a groundbreaking addition to the Mellanox ConnectX series of industry-leading adapter
cards. In addition to all the existing innovative features of past ConnectX versions, ConnectX-6 offers a
number of enhancements that further improve the performance and scalability of datacenter
applications.
ConnectX-6 adapter cards are offered in a variety of PCIe configurations, as described in the below
table.
Configuration OPN Marketing Description
ConnectX-6 PCIe x8 Card MCX651105A-EDAT ConnectX®-6 VPI adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE, single-port
QSFP56, PCIe4.0 x8, tall bracket
ConnectX-6 PCIe x16 Card
MCX653105A-HDAT ConnectX®-6 VPI adapter card, HDR IB
(200Gb/s) and 200GbE, single-port QSFP56,
PCIe4.0 x16, tall bracket
MCX653106A-HDAT ConnectX®-6 VPI adapter card, HDR IB
(200Gb/s) and 200GbE, dual-port QSFP56,
PCIe3.0/4.0 x16, tall bracket
MCX653105A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE), single-port
QSFP56, PCIe3.0/4.0 x16, tall bracket
MCX653106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE), dual-port
QSFP56, PCIe3.0/4.0 x16, tall bracket
ConnectX-6 Dual-slot Socket Direct
Cards (2x PCIe x16)
MCX654105A-HCAT ConnectX®-6 VPI adapter card kit, HDR IB
(200Gb/s) and 200GbE, single-port QSFP56,
Socket Direct 2x PCIe3.0 x16, tall brackets
MCX654106A-HCAT ConnectX®-6 VPI adapter card, HDR IB
(200Gb/s) and 200GbE, dual-port QSFP56,
Socket Direct 2x PCIe3.0/4.0x16, tall bracket
Make sure to use a PCIe slot that is capable of supplying the required power and airflow to the
ConnectX-6 as stated in Specifications.


10
Configuration OPN Marketing Description
MCX654106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s
(HDR100, EDR InfiniBand and 100GbE), dual-
port QSFP56, Socket Direct 2x PCIe3.0/4.0
x16, tall bracket
ConnectX-6 Single-slot Socket Direct
Cards (2x PCIe x8 in a row)
MCX653105A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE), single-port
QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in a
row, tall bracket
MCX653106A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s
(HDR100, EDR IBand100GbE), dual-port
QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in a
row, tall bracket
ConnectX-6 PCIe x8 Card
ConnectX-6 with a single PCIe x8 slot can support a bandwidth of up to 100Gb/s in a PCIe Gen 4.0 slot.
Part Number MCX651105A-EDAT
Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm
Data Transmission Rate Ethernet: 10/25/40/50/100 Gb/s
InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100
Network Connector Type Single-port QSFP56
PCIe x8 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s
RoHS RoHS Compliant
Adapter IC Part Number MT28908A0-XCCF-HVM