Nvidia Mellanox ConnectX-6 Dx Series User manual

Exported on Dec/12/2022 02:00 PM







NVIDIA ConnectX-6 Dx Ethernet Adapter
Cards User Manual


Downloaded from Arrow.com.

2
Table of Contents
Introduction..................................................................................... 9
Product Overview ....................................................................................9
Features and Benefits ............................................................................. 13
Operating Systems/Distributions ................................................................ 15
Connectivity ........................................................................................ 15
Interfaces ......................................................................................16
ConnectX-6 Dx IC Interface....................................................................... 16
Encryption ...................................................................................... 17
PCI Express Interface.............................................................................. 17
Networking Ports LEDs Interface ................................................................ 17
Scheme 1: One Bi-Color LED ................................................................. 18
Scheme 2: One Bi-Color LED and one Single Color LED .................................. 18
Voltage Regulators ................................................................................. 19
Hardware Installation ........................................................................20
Safety Warnings ................................................................................ 20
Installation Procedure Overview............................................................. 20
System Requirements ......................................................................... 20
Safety Precautions ............................................................................. 21
Unpacking the Package ....................................................................... 21
Pre-Installation Checklist ..................................................................... 22
Bracket Replacement Instructions .......................................................... 22
Installation Instructions....................................................................... 23
Cables and Modules............................................................................ 24
Identifying the Card in Your System......................................................... 25
Uninstalling the Card.......................................................................... 26
Driver Installation.............................................................................28
Linux Driver Installation .......................................................................... 28
Prerequisites.................................................................................... 28
Downloading MLNX_OFED..................................................................... 28
Installing MLNX_OFED ......................................................................... 29
Driver Load Upon System Boot............................................................... 34
Additional Installation Procedures .......................................................... 36
Downloaded from Arrow.com.Downloaded from Arrow.com.

3
Performance Tuning ........................................................................... 42
Windows Driver Installation ...................................................................... 42
Software Requirements ....................................................................... 42
Downloading WinOF-2 Driver................................................................. 42
Installing WinOF-2 Driver ..................................................................... 43
Firmware Upgrade ............................................................................. 50
VMware Driver Installation ....................................................................... 50
Hardware and Software Requirements ..................................................... 50
Installing NATIVE ESXi Driver for VMware vSphere ........................................ 50
Removing Earlier NVIDIA Drivers ............................................................. 51
Firmware Programming ....................................................................... 51
Updating Adapter Firmware .................................................................52
Troubleshooting ...............................................................................53
General Troubleshooting .......................................................................... 53
Linux Troubleshooting ............................................................................. 54
Windows Troubleshooting......................................................................... 54
Specifications..................................................................................56
MCX621102AC-ADAT / MCX621102AN-ADAT / MCX621102AN-ADAT Specifications ....... 56
MCX623102AC-ADAT / MCX623102AN-ADAT / MCX623102AS-ADAT Specifications........ 57
MCX621202AS-ADAT / MCX621202AC-ADAT Specifications ................................... 58
MCX623102AC-GDAT / MCX623102AE-GDAT / MCX623102AN-GDAT / MCX623102AS-GDAT
Specifications....................................................................................... 59
MCX623105AN-CDAT / MCX623105AE-CDAT / MCX623105AC-CDAT Specifications........ 60
MCX623106AS-CDAT / MCX623106AN-CDAT / MCX623106AC-CDAT / MCX623106AE-CDAT
Specifications....................................................................................... 61
MCX623106PN-CDAT / MCX623106PC-CDAT Specifications................................... 62
MCX623106TC-CDAT / MCX623106TN-CDAT / MCX623106GN-CDAT / MCX623106GC-CDAT
Specifications....................................................................................... 63
MCX623105AC-VDAT / MCX623105AN-VDAT / MCX623105AS-VDAT / MCX623105AE-VDAT
Specifications....................................................................................... 64
Board Mechanical Drawing and Dimensions.................................................... 65
Bracket Mechanical Drawing ..................................................................... 66
Monitoring......................................................................................67
Thermal Sensors.................................................................................... 67
Adapter Card Heatsink ............................................................................ 67
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

4
Finding the MAC on the Adapter Card .....................................................68
Document Revision History ..................................................................69

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

5
About This Manual
This User Manual describes NVIDIA® ConnectX®-6 Dx Ethernet adapter cards. It provides details as
to the interfaces of the board, specifications, required software and firmware for operating the
board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Dx Ethernet
adapter cards.
Configuration NVIDIA SKU Legacy OPN Marketing Description
25GbE Cards 900-9X661-0053-SQ0 MCX621102AN-ADAT ConnectX-6 Dx EN adapter
card,25GbE,Dual-
portSFP28, PCIe 4.0x8, No
Crypto, Tall Bracket
900-9X6AP-0073-ST0 MCX623102AS-ADAT ConnectX-6 Dx EN adapter
card, 25GbE, Dual-port
SFP28,PCIe 4.0 x16,Secure
Boot, No Crypto, Tall
Bracket
25GbE with Active Cooling
Cards
900-9X663-0073-SQ0 MCX621202AS-ADAT ConnectX-6 Dx EN adapter
card, 25GbE, with active
cooling, Dual-port
SFP28,PCIe 4.0 x8,Secure
Boot, No Crypto, Tall Bracket
900-9X663-0083-SQ0 MCX621202AC-ADAT ConnectX-6 Dx EN adapter
card, 25GbE, with active
cooling, Dual-port SFP28,
PCIe 4.0 x8, Crypto and
Secure Boot, Tall Bracket
50GbE Cards 900-9X6AP-0085-ST0 MCX623102AC-GDAT ConnectX-6 Dx EN adapter
card, 50GbE, Dual-port
SFP56, PCIe 4.0 x16, Crypto
and Secure Boot, Tall Bracket
100GbE Cards 900-9X6AG-0016-ST0 MCX623105AN-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Single-port
QSFP56, PCIe 4.0 x16, No
Crypto, Tall Bracket
900-9X6AG-0086-ST0 MCX623106AC-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, PCIe 4.0 x16, Crypto
and Secure Boot, Tall Bracket
900-9X6AG-0056-ST1 MCX623106AN-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, PCIe 4.0 x16, No
Crypto, Tall Bracket
900-9X6AG-0076-ST0 MCX623106AS-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, PCIe 4.0 x16, Secure
Boot, No Crypto, Tall
Bracket
100GbE with PPS In/Out
Cards
900-9X6AJ-0086-ST0 MCX623106PC-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, with PPS In/Out,
PCIe 4.0 x16, Crypto and
Secure Boot, Tall Bracket
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

6
Configuration NVIDIA SKU Legacy OPN Marketing Description
900-9X6AK-0086-SQ0 MCX623106TC-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, Enhanced-SyncE &
PTP, PPS In/Out, PCIe 4.0
x16, Crypto and Secure Boot,
Tall Bracket
900-9X6AK-0086-SQ1 MCX623106GC-CDAT ConnectX-6 Dx EN adapter
card, 100GbE, Dual-port
QSFP56, Enhanced-SyncE &
PTP GM support and GNSS,
PPS Out, PCIe 4.0 x16,
Crypto and Secure Boot, Tall
Bracket
200GbE Cards 900-9X6AG-0048-ST0 MCX623105AC-VDAT ConnectX-6 Dx EN adapter
card, 200GbE, Single-port
QSFP56, PCIe 4.0 x16, Crypto
and Secure Boot, Tall Bracket
900-9X6AG-0018-ST0 MCX623105AN-VDAT ConnectX-6 Dx EN adapter
card, 200GbE, Single-port
QSFP56, PCIe 4.0 x16, No
Crypto, Tall Bracket
Legacy (EOL) Ordering Part Numbers
Legacy OPN Marketing Description
MCX621102AC-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x8, Crypto and Secure Boot, Tall Bracket
MCX623102AC-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623102AN-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX621102AE-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x8, Crypto, No Secure Boot, Tall Bracket
MCX623102AN-GDAT ConnectX-6 Dx EN adapter card, 50GbE , Dual-port SFP56, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX623102AE-GDAT ConnectX-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe
4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623102AS-GDAT ConnectX-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe
4.0 x16, Secure Boot, No Crypto, Tall Bracket
MCX623105AC-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Single-port QSFP56,
PCIe 4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623105AE-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Single-port QSFP56,
PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623106AE-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe
4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623109AC-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Single-port DSFP, PCIe
4.0 x16, Crypto and Secure Boot, Tall Bracket
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

7
•
•
Legacy OPN Marketing Description
MCX623109AN-CDAT ConnectX-6 Dx EN adapter card, 100GbE , Single-port DSFP, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX623106GN-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56,
Enhanced-SyncE & PTP GM support and GNSS, PPS Out, PCIe 4.0
x16, No Crypto, Tall Bracket
MCX623106PC-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623106PE-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623106PN-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, No Crypto, Tall Bracket
MCX623106TN-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, No Crypto, Tall Bracket
MCX623105AE-VDAT ConnectX-6 Dx EN adapter card, 200GbE, Single-port QSFP56,
PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623105AS-VDAT ConnectX-6 Dx EN adapter card, 200GbE, Single-port QSFP56,
PCIe 4.0 x16, Secure Boot, No Crypto, Tall Bracket
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic
familiarity with Ethernet network and architecture specifications.
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough
the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchasedNVIDIA M-1 Global Support Services, please see your contract fordetails
regarding Technical Support.
Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek
assistance through their reseller.
Related Documentation
NVIDIA MLNX_OFED for
Linux User Manual and
Release Notes
User Manual and Release Notes describing MLNx_OFED features, performance,
band diagnostic, tools content, and configuration. See NVIDIA MLNX_OFED for
Linux Documentation.
WinOF-2 for
WindowsUser Manual
and Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic,
tools content, and configuration. See WinOF-2 for Windows Documentation.
NVIDIA VMware for
Ethernet User Manual
and Release Notes
User Manual describing the various components of the NVIDIA ConnectX® NATIVE
ESXi stack. SeeVMware® ESXi Documentation.
NVIDIA Firmware
Update
NVIDIA firmware update and query utility used to update the firmware.
SeeNVIDIA Firmware Utility (mlxup) Documentation.
NVIDIA Firmware Tools
(MFT) User Manual
User Manual describing the set of MFT firmware management tools for a single
node. SeeMFT User Manual.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

8
NVIDIA MLNX_OFED for
Linux User Manual and
Release Notes
User Manual and Release Notes describing MLNx_OFED features, performance,
band diagnostic, tools content, and configuration. See NVIDIA MLNX_OFED for
Linux Documentation.
IEEE Std 802.3
Specification
IEEE Ethernet Specifications
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications.
LinkX Interconnect
Solutions
LinkX Ethernet cables and transceivers are designed to maximize the
performance of High-Performance Computing networks, requiring high-
bandwidth, low-latency connections between compute nodes and switch
nodes.NVIDIA offers one of the industry’s broadest portfolio of 40GbE, 56GbE,
100GbE, 200GbE and400GbE cables, includingDirect Attach Copper cables
(DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a
wide range of lengths from 0.5m to 10km. In addition to meeting
Ethernetstandards, NVIDIA tests every product in an end-to-end environment
ensuring a Bit Error Rate of less than 1E-15.Read more atLinkX Cables and
Transceivers.
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.
Revision History
A list of the changes made to this document are provided in.Document Revision History.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

9
Introduction
Product Overview
This is the User Manual for Ethernet adapter cards based on the ConnectX®-6 Dx integrated circuit
device.
As the world's most advanced cloud SmartNIC, ConnectX-6 Dx provides up to two ports of 25, 50 or
100Gb/s or a single-port of 200Gb/s Ethernet connectivity, powered by 50Gb/s PAM4 SerDes
technology and PCIe Gen 4.0 host connectivity.ConnectX-6 Dx continues among NVIDIA's innovation
path in scalable cloud fabrics, delivering unparalleled performance and efficiency at every scale.
ConnectX-6 Dx's innovative hardware offload engines, including IPsec and TLS inline data-in-motion
encryption, are ideal for enabling secure network connectivity in modern data-center
environments. Please refer toFeature and Benefits for more details.
ConnectX-6 Dx 25GbE Adapter Cards
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX62110
2AC-ADAT
4.89in. x
2.71in
(124.22mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
✔ ✔ ✔ Tall
Bracket
MCX62110
2AN-ADAT
4.89in. x
2.71in
(124.22mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
- - ✔Tall
Bracket
MCX62110
2AN-ADAT
4.89in. x
2.71in
(124.22mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
- - ✔Tall
Bracket
MCX62310
2AC-ADAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX62310
2AN-ADAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX62310
2AS-ADAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
ConnectX-6 Dx 25GbE Adapter Cards with Active Cooling
These cards are optimized for Workstation Environments and include an onboard cooling fan
that meets the acoustic requirements for workstations.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

10
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX62120
2AS-ADAT
6.01in. x
2.71in
(152.90mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
✔-✔Tall
Bracket
MCX62120
2AC-ADAT
6.01in. x
2.71in
(152.90mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
✔-✔Tall
Bracket
ConnectX-6 Dx 50GbE Adapter Cards
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX62310
2AC-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
50/25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX62310
2AE-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
-✔ ✔ Tall
Bracket
MCX62310
2AN-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX62310
2AS-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
ConnectX-6 Dx 100GbE Adapter Cards
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX623105
AN-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX623106
AN-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Dual-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

11
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX623105
AC-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX623106
AC-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Dual-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX623105
AE-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
-✔ ✔ Tall
Bracket
MCX623106
AE-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Dual-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
-✔ ✔ Tall
Bracket
MCX623106
AS-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Dual-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
ConnectX-6 Dx 100GbE Adapter Cards for Timing and Telecommunication Application SMAs
OPN Form
Factor/
Dimensio
ns
Data
Transmi
ssion
Rate
No. of
Ports
and
Type
PCIe
Interface
Sec
ure
Boo
t
Cry
pto
PPS
In
/
Out
SMA
s
PP
S
OU
T
PP
S
IN
Sycn
E
&
PTP
GM
Supp
ort
GN
SS
Ro
HS
Brack
et
Type
MCX623
106PN-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
- - ✔ ✔ ✔ - - ✔Tall
Bracke
t
MCX623
106PC-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
✔ ✔ ✔ ✔ ✔ - - ✔Tall
Bracke
t
MCX623
106TN-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
- - ✔ ✔ ✔ ✔ -✔Tall
Bracke
t
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

12
OPN Form
Factor/
Dimensio
ns
Data
Transmi
ssion
Rate
No. of
Ports
and
Type
PCIe
Interface
Sec
ure
Boo
t
Cry
pto
PPS
In
/
Out
SMA
s
PP
S
OU
T
PP
S
IN
Sycn
E
&
PTP
GM
Supp
ort
GN
SS
Ro
HS
Brack
et
Type
MCX623
106TC-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
✔ ✔ ✔ ✔ ✔ ✔ -✔Tall
Bracke
t
MCX623
106GN-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
- - ✔ ✔ -✔ ✔ ✔Tall
Bracke
t
MCX623
106GC-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
✔ ✔ ✔ ✔ -✔ ✔ ✔Tall
Bracke
t
ConnectX-6 Dx 200GbE Adapter Cards
OPN Form Factor/
Dimensions
Data
Transmissi
on Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Crypt
o
RoH
S
Bracket
Type
MCX623105
AC-VDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
200/100/50
/25/10/1
GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX623105
AE-VDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
200/100/50
/25/10/1
GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
-✔ ✔ Tall
Bracket
MCX623105
AN-VDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
200/100/50
/25/10/1
GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
--✔Tall
Bracket
MCX623105
AS-VDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
200/100/50
/25/10/1
GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
For more detailed information seeSpecifications.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

13
•
•
•
Features and Benefits
Feature Description
PCI Express
(PCIe)
PCIe Gen 4.0 SERDES@ 8.0GT/s / 16.0GT/s through x8/x16 Edge Connector
Up to
200GbE
NVIDIA adapters comply with the following IEEE 802.3 standards:
200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
– IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
– IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
– IEEE 802.3ba 40 Gigabit Ethernet
– IEEE 802.3by 25 Gigabit Ethernet
– IEEE 802.3ae 10 Gigabit Ethernet
– IEEE 802.3ap based auto-negotiation and KR startup
– IEEE 802.3ad, 802.1AX Link Aggregation
– IEEE 802.1Q, 802.1P VLAN tags and priority
– IEEE 802.1Qau (QCN)
– Congestion Notification
– IEEE 802.1Qaz (ETS)
– IEEE 802.1Qbb (PFC)
– IEEE 802.1Qbg
– IEEE 1588v2
– Jumbo frame support (9.6KB)
Memory PCI Express - stores and accesses Ethernet fabric connection information and packet
data.
SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G device by
Macronix)
Available only in QSFP cards: FRU EEPROM - Stores the parameters and personality of the
card. The EEPROM capacity is 128Kbit. FRU I2C address is (0x50) and is accessible
through the PCIe SMBus (Note: Address 0x58 is reserved.)
Overlay
Networks
In order to better scale their networks, data center operators often create overlay networks
that carry traffic from individual virtual machines over logical tunnels in encapsulated formats
such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet
from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-6 Dx
effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines
that encapsulate and de-capsulate the overlay protocol.
RDMA
overConver
ged
Ethernet
(RoCE)
ConnectX-6 Dx, utilizing RoCE (RDMA over Converged Ethernet) technology, delivers low-latency
and high-performance over Band and Ethernet networks. Leveraging data center bridging (DCB)
capabilities, as well as ConnectX-6 Dx, advanced congestion control hardware mechanisms,
RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks.
NVIDIA
PeerDirect
®
NVIDIA PeerDirect®communication provides high-efficiency RDMA access by eliminating
unnecessary internal data copies between components on the PCIe bus (for example, from GPU
to CPU), and therefore significantly reduces application run time. ConnectX-6 Dx advanced
acceleration technology enables higher cluster efficiency and scalability to tens of thousands of
nodes.
This section describes hardware features and capabilities. Please refer to the relevant
driver and/or firmware release notes for feature availability.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

14
Feature Description
CPU
Offload
Adapter functionality enables reduced CPU overhead leaving more CPU resources available for
computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of
Service
(QoS)
Support for port-based Quality of Service enabling various application requirements for latency
and SLA.
Hardware-
based I/
OVirtualiza
tion
ConnectX-6 Dx provides dedicated adapter resources and guaranteed isolation and protection
for virtual machines within the server.
Storage
Acceleratio
n
A consolidated compute and storage network achieves significant cost-performance advantages
over multi-fabric networks. Standard block and file access protocols can leverage
• RDMA for high-performance storage access
• NVMe over Fabric offloads for the target machine
SR-IOV ConnectX-6 Dx SR-IOV technology provides dedicated adapter resources and guaranteed
isolation and protection for virtual machines (VM) within the server.
High-
Performanc
eAccelerati
ons
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing
Time
Sensitive
Application
s
NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related features
called “5T”. NVIDIA PTP and 5T software solutions are designed to meet the most demanding
PTP profiles. ConnectX-6 Dx incorporates an integrated Hardware Clock (PHC) that allows
ConnectX-6 Dx to achieve sub 20u Sec accuracy and also offers many timing-related functions
such as time-triggered scheduling or time-based SND accelerations (time-based ASAP²).
Furthermore, 5T technology enables the software application to transmit fronthaul (ORAN)
compatible in high bandwidth. The PTP part supports the subordinate clock, master clock, and
boundary clock.
ConnectX-6 Dx PTP solution allows you to run any PTP stack on your host.
Enhanced-
SyncE &
PTP
NVIDIA offers ConnectX-6 Dx cards with SyncE support including an improved holdover to meet
ITU-T G.8273.2 class C.
Enabled in MCX623106TN-CDAT, MCX623106TC-CDAT, MCX623106GN-CDAT, and MCX623106GC-
CDAT.
Grand
Master
(GNSS
Reciver)
NVIDIA offers ConnectX-6 Dx with an integrated GNSS receiver to allow a compact and efficient
solution for a Grand Master at every server.
Enabled in MCX623106G[N/C]-CDAT
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

15
•
•
•
•
•
•
•
•
•
Feature Description
PPS In/Out
SMAs
NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related features
called “5T”. NVIDIA PTP and 5T software solutions are designed to meet the most demanding
PTP profiles. ConnectX-6 Dx incorporates an integrated Hardware Clock (PHC) that allows
ConnectX-6 Dx to achieve sub 20u Sec accuracy and also offers many timing-related functions
such as time-triggered scheduling or time-based SND accelerations (time-based ASAP²).
Furthermore, 5T technology enables the software application to transmit fronthaul (ORAN)
compatible in high bandwidth. The PTP part supports the subordinate clock, master clock, and
boundary clock.
ConnectX-6 Dx PTP solution allows you to run any PTP stack on your host.
With respect to testing and measurements, selected NVIDIA adapters allow you to use the PPS-
out signal from the onboard SMA connecter, ConnectX-6 Dx also allows measuring PTP in scale,
with PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In
signal, and the timestamp will be sent to the SW.
Enabled in MCX623106PN-CDAT, MCX623106PC-CDAT, MCX623106TN-CDAT, MCX623106TC-CDAT,
MCX623106GN-CDAT, and MCX623106GC-CDAT.
Operating Systems/Distributions
RHEL/CentOS
Windows
FreeBSD
VMware
OpenFabrics Enterprise Distribution (MLNX_OFED)
OpenFabrics Windows Distribution (WinOF-2)
Connectivity
Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

16
Interfaces
The below figures show the component side of the ConnectX-6 Dx adapter card. Each numbered
interface that is referenced in the figures is described in the following table with a link to detailed
information.
Callout Item Description
1“ConnectX-6 Dx IC” ConnectX-6 Dx IC on the board.
2“Ethernet SFP28/SFP56/QSFP56
Interface”
Ethernet traffic is transmitted through the adapter's
SFP28/SFP56/QSFP56 connectors. The networking
connectors allow for the use of modules, optical and
passive cable interconnect solutions.
3"PCI Express Interface" PCIe Gen 3.0/4.0 through an x8/x16 edge
connector.
4“Networking Ports LEDs Interface” There are two I/O LEDs per port to indicate speed
and link status.
“SMBus Interface” Allows BMC connectivity using MCTP over SMBus or
MCTP over PCIe protocols.
“Voltage Regulators” Voltage supply pins that feed onboard regulators.
ConnectX-6 Dx IC Interface
The ConnectX®-6 Dx EN family of adapter IC devicesdelivers two ports of 10/25/40/50/100Gb/s or
a single-port of 200Gb/s Ethernet connectivity paired with best-in-class hardware capabilities that
accelerate and secure cloud and data-center workloads.
The below figures are for illustration purposes only and might not reflect the current
revision of the adapter card.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

17
•
•
•
•
•
•
•
Encryption
ConnectX-6 Dx brings security to every end-point, including:
Purpose-built inline acceleration engines that offload IPsec and TLS data-in-motion and XTS-
AES data-at-rest cryptographic operations.
Stateful firewall solution acceleration, powered by Open vSwitch connection tracking and
NVIDIA’s ASAP2 technology.
Embedded hardware root-of-trust and support for RSA-based secure firmware update and
secure boot, providing guaranteed integrity of the network adapter.
Ethernet SFP28 / SFP56 / QSFP56 Interfaces
The network ports of the ConnectX-6 Dx adapter card are compliant with the IEEE 802.3 Ethernet
standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28 /
SFP56 / QSFP56connector on the adaptercard.
PCI Express Interface
ConnectX-6 Dx adapter cards support PCI Express Gen 3.0/4.0 (1.1 and 2.0 compatible) through
x8/16 edge connectors. The device can be either a master initiating the PCI Express bus operations,
or a slave responding to PCI bus operations.
The following lists PCIe interface features:
PCIe Gen 4.0 and 3.0 compliant, 2.0 and 1.1 compatible
2.5, 5.0, 8.0, or 16.0 GT/s link rate x8 or x16 lanes
Auto-negotiates to x16, x8, x4, x2, or x1
Support for MSI/MSI-X mechanisms
Networking Ports LEDs Interface
For the networking ports LEDs description, follow the below table depending on the OPN you have
purchased.
OPN LEDs Scheme
MCX621102A[C/N/E]-ADAT, MCX621202A(C/S)-ADAT Scheme 1: One Bi-Color LED
MCX623102A[C/N]-ADAT, MCX623102A[C/N]-
GDAT,MCX623102A[S/E/N/C]-GDAT
MCX623105A[N/E]-CDAT,MCX623106A[C/N/S/E]-
CDAT,MCX623106P[C/N/E]-CDAT,
MCX623105A[C/N/S/E]-VDAT
Scheme 2: One Bi-Color LED and one Single Color
LED
Applies to Crypto OPNs only.

The adapter card includes special circuits to protect from ESD shocks to the card/server
when plugging copper cables.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

18
Scheme 1: One Bi-Color LED
There is one bicolor(Yellow and Green) I/O LED per port to indicate speed and link status.
Link Indications
State Bi-Color LED (Yellow/Green)
Physical link speed
Beacon command for locating the adapter card 1Hz blinking Yellow
Error 4Hz blinking Yellow Indicates an error with the link. The
error can be one of the following:
Error Type Description LED Behavior
I2CI2C access to the
networking ports
fails
Blinks until error
is fixed
Over-current Over-current
condition of the
networking ports
Blinks until error
is fixed
Physical Activity The Green LED will blink.
Link Up The Green LED will be solid.
Scheme 2: One Bi-Color LED and one Single Color LED
There are two I/O LEDs per port to indicate speed and link status. LED1 is a bicolor LED (Yellow and
green) and LED2 is a single color LED (green).
Link Indications
State Bi-Color LED (Yellow/Green) Single Color LED (Green)
Beacon command for locating the
adapter card
1Hz blinking Yellow OFF
Error 4Hz blinking Yellow Indicates an error
with the link. The error can be one of the
following:
Error
Type
Description LED
Behavior
I2CI2C access to
the
networking
ports fails
Blinks until
error is
fixed
Over-
current
Over-current
condition of
the
networking
ports
Blinks until
error is
fixed
ON
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

19
State Bi-Color LED (Yellow/Green) Single Color LED (Green)
Physical Activity The Green LED will blink. Blinking
Link Up The Green LED will be solid. ON
SMBus Interface
ConnectX-6 Dx technology maintains support for manageability through a BMC. ConnectX-6 Dx PCIe
stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as
if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific
manageability solution in use by the server, please contact NVIDIA Support.
Voltage Regulators
The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These
voltage supply pins feed on-board regulators that provide the necessary power to the various
components on the card.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

20
Hardware Installation
Installation and initialization of ConnectX-6 Dx adapter cards require attention to the mechanical
attributes, power specification, and precautions for electronic equipment.
Safety Warnings
Please observe all safety warnings to avoidinjury and prevent damage to system components.
Unable to render include or excerpt-include. Could not retrieve page.
Installation Procedure Overview
The installation procedure of ConnectX-6 Dx adapter cards involves the following steps:
Step Procedure Direct Link
1Check the system’s hardware and software
requirements.
Refer to System Requirements
2 Pay attention to the airflow consideration within
the host system
Refer to Airflow Requirements
3 Follow the safety precautions Refer to Safety Precautions
4 Unpack the package Refer to Unpack the package
5 Follow the pre-installation checklist Refer to Pre-Installation Checklist
6 (Optional) Replace the full-height mounting
bracket with the supplied short bracket
Refer to Bracket Replacement
Instructions
7 Install the ConnectX-6 Dx card in the system Refer to Installation Instructions
8 Connect cables or modules to the card Refer to Cables and Modules
9 Identify ConnectX-6 Dx in the system Refer to Identifying Your Card
System Requirements
Hardware Requirements
A system with a PCI Express slot of the corresponding bus width is required for installing the card.
Safety warnings are provided here in the English language. For safety warnings in other
languages, refer to the Adapter Installation Safety Instructions.

Unless otherwise specified, NVIDIA products are designed to work in an environmentally
controlled data center with low levels of gaseous and dust (particulate) contamination.
The operation environment should meet severity level G1 as per ISA 71.04 for gaseous
contamination and ISO 14644-1 class 8 for cleanliness level.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.