Nvidia Mellanox ConnectX-6 Dx Series User manual

Exported on Dec/12/2022 02:00 PM







NVIDIA ConnectX-6 Dx Ethernet Adapter
Cards User Manual


Downloaded from Arrow.com.

2
Table of Contents
Introduction..................................................................................... 9
Product Overview ....................................................................................9
Features and Benefits ............................................................................. 13
Operating Systems/Distributions ................................................................ 15
Connectivity ........................................................................................ 15
Interfaces ......................................................................................16
ConnectX-6 Dx IC Interface....................................................................... 16
Encryption ...................................................................................... 17
PCI Express Interface.............................................................................. 17
Networking Ports LEDs Interface ................................................................ 17
Scheme 1: One Bi-Color LED ................................................................. 18
Scheme 2: One Bi-Color LED and one Single Color LED .................................. 18
Voltage Regulators ................................................................................. 19
Hardware Installation ........................................................................20
Safety Warnings ................................................................................ 20
Installation Procedure Overview............................................................. 20
System Requirements ......................................................................... 20
Safety Precautions ............................................................................. 21
Unpacking the Package ....................................................................... 21
Pre-Installation Checklist ..................................................................... 22
Bracket Replacement Instructions .......................................................... 22
Installation Instructions....................................................................... 23
Cables and Modules............................................................................ 24
Identifying the Card in Your System......................................................... 25
Uninstalling the Card.......................................................................... 26
Driver Installation.............................................................................28
Linux Driver Installation .......................................................................... 28
Prerequisites.................................................................................... 28
Downloading MLNX_OFED..................................................................... 28
Installing MLNX_OFED ......................................................................... 29
Driver Load Upon System Boot............................................................... 34
Additional Installation Procedures .......................................................... 36
Downloaded from Arrow.com.Downloaded from Arrow.com.

3
Performance Tuning ........................................................................... 42
Windows Driver Installation ...................................................................... 42
Software Requirements ....................................................................... 42
Downloading WinOF-2 Driver................................................................. 42
Installing WinOF-2 Driver ..................................................................... 43
Firmware Upgrade ............................................................................. 50
VMware Driver Installation ....................................................................... 50
Hardware and Software Requirements ..................................................... 50
Installing NATIVE ESXi Driver for VMware vSphere ........................................ 50
Removing Earlier NVIDIA Drivers ............................................................. 51
Firmware Programming ....................................................................... 51
Updating Adapter Firmware .................................................................52
Troubleshooting ...............................................................................53
General Troubleshooting .......................................................................... 53
Linux Troubleshooting ............................................................................. 54
Windows Troubleshooting......................................................................... 54
Specifications..................................................................................56
MCX621102AC-ADAT / MCX621102AN-ADAT / MCX621102AN-ADAT Specifications ....... 56
MCX623102AC-ADAT / MCX623102AN-ADAT / MCX623102AS-ADAT Specifications........ 57
MCX621202AS-ADAT / MCX621202AC-ADAT Specifications ................................... 58
MCX623102AC-GDAT / MCX623102AE-GDAT / MCX623102AN-GDAT / MCX623102AS-GDAT
Specifications....................................................................................... 59
MCX623105AN-CDAT / MCX623105AE-CDAT / MCX623105AC-CDAT Specifications........ 60
MCX623106AS-CDAT / MCX623106AN-CDAT / MCX623106AC-CDAT / MCX623106AE-CDAT
Specifications....................................................................................... 61
MCX623106PN-CDAT / MCX623106PC-CDAT Specifications................................... 62
MCX623106TC-CDAT / MCX623106TN-CDAT / MCX623106GN-CDAT / MCX623106GC-CDAT
Specifications....................................................................................... 63
MCX623105AC-VDAT / MCX623105AN-VDAT / MCX623105AS-VDAT / MCX623105AE-VDAT
Specifications....................................................................................... 64
Board Mechanical Drawing and Dimensions.................................................... 65
Bracket Mechanical Drawing ..................................................................... 66
Monitoring......................................................................................67
Thermal Sensors.................................................................................... 67
Adapter Card Heatsink ............................................................................ 67
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

4
Finding the MAC on the Adapter Card .....................................................68
Document Revision History ..................................................................69

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

5
About This Manual
This User Manual describes NVIDIA® ConnectX®-6 Dx Ethernet adapter cards. It provides details as
to the interfaces of the board, specifications, required software and firmware for operating the
board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Dx Ethernet
adapter cards.
Configuration NVIDIA SKU Legacy OPN Marketing Description
25GbE Cards 900-9X661-0053-SQ0 MCX621102AN-ADAT ConnectX-6 Dx EN adapter
card,25GbE,Dual-
portSFP28, PCIe 4.0x8, No
Crypto, Tall Bracket
900-9X6AP-0073-ST0 MCX623102AS-ADAT ConnectX-6 Dx EN adapter
card, 25GbE, Dual-port
SFP28,PCIe 4.0 x16,Secure
Boot, No Crypto, Tall
Bracket
25GbE with Active Cooling
Cards
900-9X663-0073-SQ0 MCX621202AS-ADAT ConnectX-6 Dx EN adapter
card, 25GbE, with active
cooling, Dual-port
SFP28,PCIe 4.0 x8,Secure
Boot, No Crypto, Tall Bracket
900-9X663-0083-SQ0 MCX621202AC-ADAT ConnectX-6 Dx EN adapter
card, 25GbE, with active
cooling, Dual-port SFP28,
PCIe 4.0 x8, Crypto and
Secure Boot, Tall Bracket
50GbE Cards 900-9X6AP-0085-ST0 MCX623102AC-GDAT ConnectX-6 Dx EN adapter
card, 50GbE, Dual-port
SFP56, PCIe 4.0 x16, Crypto
and Secure Boot, Tall Bracket
100GbE Cards 900-9X6AG-0016-ST0 MCX623105AN-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Single-port
QSFP56, PCIe 4.0 x16, No
Crypto, Tall Bracket
900-9X6AG-0086-ST0 MCX623106AC-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, PCIe 4.0 x16, Crypto
and Secure Boot, Tall Bracket
900-9X6AG-0056-ST1 MCX623106AN-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, PCIe 4.0 x16, No
Crypto, Tall Bracket
900-9X6AG-0076-ST0 MCX623106AS-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, PCIe 4.0 x16, Secure
Boot, No Crypto, Tall
Bracket
100GbE with PPS In/Out
Cards
900-9X6AJ-0086-ST0 MCX623106PC-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, with PPS In/Out,
PCIe 4.0 x16, Crypto and
Secure Boot, Tall Bracket
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

6
Configuration NVIDIA SKU Legacy OPN Marketing Description
900-9X6AK-0086-SQ0 MCX623106TC-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, Enhanced-SyncE &
PTP, PPS In/Out, PCIe 4.0
x16, Crypto and Secure Boot,
Tall Bracket
900-9X6AK-0086-SQ1 MCX623106GC-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, Enhanced-SyncE &
PTP GM support and GNSS,
PPS Out, PCIe 4.0 x16,
Crypto and Secure Boot, Tall
Bracket
200GbE Cards 900-9X6AG-0048-ST0 MCX623105AC-VDAT ConnectX-6 Dx EN adapter
card, 200GbE, Single-port
QSFP56, PCIe 4.0 x16, Crypto
and Secure Boot, Tall Bracket
900-9X6AG-0018-ST0 MCX623105AN-VDAT ConnectX-6 Dx EN adapter
card, 200GbE, Single-port
QSFP56, PCIe 4.0 x16, No
Crypto, Tall Bracket
Legacy (EOL) Ordering Part Numbers
Legacy OPN Marketing Description
MCX621102AC-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x8, Crypto and Secure Boot, Tall Bracket
MCX623102AC-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623102AN-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX621102AE-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x8, Crypto, No Secure Boot, Tall Bracket
MCX623102AN-GDAT ConnectX-6 Dx EN adapter card, 50GbE , Dual-port SFP56, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX623102AE-GDAT ConnectX-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe
4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623102AS-GDAT ConnectX-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe
4.0 x16, Secure Boot, No Crypto, Tall Bracket
MCX623105AC-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Single-port QSFP56,
PCIe 4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623105AE-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Single-port QSFP56,
PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623106AE-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe
4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623109AC-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Single-port DSFP, PCIe
4.0 x16, Crypto and Secure Boot, Tall Bracket
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

7
•
•
Legacy OPN Marketing Description
MCX623109AN-CDAT ConnectX-6 Dx EN adapter card, 100GbE , Single-port DSFP, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX623106GN-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56,
Enhanced-SyncE & PTP GM support and GNSS, PPS Out, PCIe 4.0
x16, No Crypto, Tall Bracket
MCX623106PC-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623106PE-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623106PN-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, No Crypto, Tall Bracket
MCX623106TN-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, No Crypto, Tall Bracket
MCX623105AE-VDAT ConnectX-6 Dx EN adapter card, 200GbE, Single-port QSFP56,
PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623105AS-VDAT ConnectX-6 Dx EN adapter card, 200GbE, Single-port QSFP56,
PCIe 4.0 x16, Secure Boot, No Crypto, Tall Bracket
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic
familiarity with Ethernet network and architecture specifications.
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough
the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchasedNVIDIA M-1 Global Support Services, please see your contract fordetails
regarding Technical Support.
Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek
assistance through their reseller.
Related Documentation
NVIDIA MLNX_OFED for
Linux User Manual and
Release Notes
User Manual and Release Notes describing MLNx_OFED features, performance,
band diagnostic, tools content, and configuration. See NVIDIA MLNX_OFED for
Linux Documentation.
WinOF-2 for
WindowsUser Manual
and Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic,
tools content, and configuration. See WinOF-2 for Windows Documentation.
NVIDIA VMware for
Ethernet User Manual
and Release Notes
User Manual describing the various components of the NVIDIA ConnectX® NATIVE
ESXi stack. SeeVMware® ESXi Documentation.
NVIDIA Firmware
Update
NVIDIA firmware update and query utility used to update the firmware.
SeeNVIDIA Firmware Utility (mlxup) Documentation.
NVIDIA Firmware Tools
(MFT) User Manual
User Manual describing the set of MFT firmware management tools for a single
node. SeeMFT User Manual.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

8
NVIDIA MLNX_OFED for
Linux User Manual and
Release Notes
User Manual and Release Notes describing MLNx_OFED features, performance,
band diagnostic, tools content, and configuration. See NVIDIA MLNX_OFED for
Linux Documentation.
IEEE Std 802.3
Specification
IEEE Ethernet Specifications
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications.
LinkX Interconnect
Solutions
LinkX Ethernet cables and transceivers are designed to maximize the
performance of High-Performance Computing networks, requiring high-
bandwidth, low-latency connections between compute nodes and switch
nodes.NVIDIA offers one of the industry’s broadest portfolio of 40GbE, 56GbE,
100GbE, 200GbE and400GbE cables, includingDirect Attach Copper cables
(DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a
wide range of lengths from 0.5m to 10km. In addition to meeting
Ethernetstandards, NVIDIA tests every product in an end-to-end environment
ensuring a Bit Error Rate of less than 1E-15.Read more atLinkX Cables and
Transceivers.
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.
Revision History
A list of the changes made to this document are provided in.Document Revision History.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

9
Introduction
Product Overview
This is the User Manual for Ethernet adapter cards based on the ConnectX®-6 Dx integrated circuit
device.
As the world's most advanced cloud SmartNIC, ConnectX-6 Dx provides up to two ports of 25, 50 or
100Gb/s or a single-port of 200Gb/s Ethernet connectivity, powered by 50Gb/s PAM4 SerDes
technology and PCIe Gen 4.0 host connectivity.ConnectX-6 Dx continues among NVIDIA's innovation
path in scalable cloud fabrics, delivering unparalleled performance and efficiency at every scale.
ConnectX-6 Dx's innovative hardware offload engines, including IPsec and TLS inline data-in-motion
encryption, are ideal for enabling secure network connectivity in modern data-center
environments. Please refer toFeature and Benefits for more details.
ConnectX-6 Dx 25GbE Adapter Cards
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX62110
2AC-ADAT
4.89in. x
2.71in
(124.22mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
✔ ✔ ✔ Tall
Bracket
MCX62110
2AN-ADAT
4.89in. x
2.71in
(124.22mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
- - ✔Tall
Bracket
MCX62110
2AN-ADAT
4.89in. x
2.71in
(124.22mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
- - ✔Tall
Bracket
MCX62310
2AC-ADAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX62310
2AN-ADAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX62310
2AS-ADAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
ConnectX-6 Dx 25GbE Adapter Cards with Active Cooling
These cards are optimized for Workstation Environments and include an onboard cooling fan
that meets the acoustic requirements for workstations.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

10
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX62120
2AS-ADAT
6.01in. x
2.71in
(152.90mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
✔-✔Tall
Bracket
MCX62120
2AC-ADAT
6.01in. x
2.71in
(152.90mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
✔-✔Tall
Bracket
ConnectX-6 Dx 50GbE Adapter Cards
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX62310
2AC-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
50/25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX62310
2AE-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
-✔ ✔ Tall
Bracket
MCX62310
2AN-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX62310
2AS-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
ConnectX-6 Dx 100GbE Adapter Cards
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX623105
AN-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX623106
AN-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Dual-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.