Nvidia Mellanox MCX621102ACADAT User manual

Exported on Sep/16/2022 01:57 PM







NVIDIA ConnectX-6 Dx Ethernet Adapter
Cards User Manual



2
Table of Contents
Introduction..................................................................................... 9
Product Overview ....................................................................................9
Features and Benefits ............................................................................. 13
Operating Systems/Distributions ................................................................ 15
Connectivity ........................................................................................ 16
Interfaces ......................................................................................17
ConnectX-6 Dx IC Interface....................................................................... 18
Encryption ...................................................................................... 18
PCI Express Interface.............................................................................. 18
Networking Ports LEDs Interface ................................................................ 19
Scheme 1: One Bi-Color LED ................................................................. 19
Scheme 2: One Bi-Color LED and one Single Color LED .................................. 20
Voltage Regulators ................................................................................. 20
Heatsink ............................................................................................. 21
Thermal Sensors.................................................................................... 21
Hardware Installation ........................................................................22
Safety Warnings ................................................................................ 22
Installation Procedure Overview............................................................. 22
System Requirements ......................................................................... 23
Safety Precautions ............................................................................. 23
Unpacking the Package ....................................................................... 24
Pre-Installation Checklist ..................................................................... 24
Bracket Replacement Instructions .......................................................... 25
Installation Instructions....................................................................... 25
Cables and Modules............................................................................ 27
Identifying the Card in Your System......................................................... 28
Uninstalling the Card.......................................................................... 29
Driver Installation.............................................................................31
Linux Driver Installation .......................................................................... 31
Prerequisites.................................................................................... 31
Downloading MLNX_OFED..................................................................... 31
Installing MLNX_OFED ......................................................................... 32

3
Installing MLNX_OFED Using YUM............................................................ 37
Installing MLNX_OFED Using apt-get Tool .................................................. 40
Updating Firmware After Installation....................................................... 41
UEFI Secure Boot............................................................................... 42
Performance Tuning ........................................................................... 44
Windows Driver Installation ...................................................................... 44
Software Requirements ....................................................................... 44
Downloading WinOF-2 Driver................................................................. 45
Uninstalling WinOF-2 Driver.................................................................. 52
Extracting Files Without Running Installation ............................................. 53
Firmware Upgrade ............................................................................. 55
VMware Driver Installation ....................................................................... 55
Hardware and Software Requirements ..................................................... 55
Installing NATIVE ESXi Driver for VMware vSphere ........................................ 56
Removing Earlier NVIDIA Drivers ............................................................. 56
Firmware Programming ....................................................................... 57
Updating Adapter Firmware .................................................................58
Troubleshooting ...............................................................................59
General Troubleshooting .......................................................................... 59
Linux Troubleshooting ............................................................................. 60
Windows Troubleshooting......................................................................... 60
Specifications..................................................................................62
MCX621102AC-ADAT / MCX621102AN-ADAT / MCX621102AN-ADAT Specifications ....... 62
MCX623102AC-ADAT / MCX623102AN-ADAT / MCX623102AS-ADAT Specifications........ 63
MCX621202AS-ADAT / MCX621202AC-ADAT Specifications ................................... 64
MCX623102AC-GDAT / MCX623102AE-GDAT / MCX623102AN-GDAT / MCX623102AS-GDAT
Specifications....................................................................................... 65
MCX623105AN-CDAT / MCX623105AE-CDAT / MCX623105AC-CDAT Specifications........ 66
MCX623106AS-CDAT / MCX623106AN-CDAT / MCX623106AC-CDAT / MCX623106AE-CDAT
Specifications....................................................................................... 67
MCX623106PN-CDAT / MCX623106PC-CDAT Specifications................................... 69
MCX623106TC-CDAT / MCX623106TN-CDAT / MCX623106GN-CDAT / MCX623106GC-CDAT
Specifications....................................................................................... 70
MCX623105AC-VDAT / MCX623105AN-VDAT / MCX623105AS-VDAT / MCX623105AE-VDAT
Specifications....................................................................................... 71

4
Board Mechanical Drawing and Dimensions.................................................... 72
Bracket Mechanical Drawing ..................................................................... 73
Finding the MAC on the Adapter Card .....................................................75
Document Revision History ..................................................................76


5
About This Manual
This User Manual describes NVIDIA® ConnectX®-6 Dx Ethernet adapter cards. It provides details as
to the interfaces of the board, specifications, required software and firmware for operating the
board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Dx Ethernet
adapter cards.
Configuration OPN Marketing Description
25GbE Cards MCX621102AN-ADAT ConnectX-6 Dx EN adapter
card,25GbE,Dual-portSFP28, PCIe
4.0x8, No Crypto, Tall Bracket
MCX623102AS-ADAT ConnectX-6 Dx EN adapter card,
25GbE, Dual-port SFP28,PCIe 4.0
x16,Secure Boot, No Crypto, Tall
Bracket
25GbE with Active Cooling Cards
MCX621202AS-ADAT
ConnectX-6 Dx EN adapter card,
25GbE, with active cooling, Dual-port
SFP28,PCIe 4.0 x8,Secure Boot, No
Crypto, Tall Bracket
MCX621202AC-ADAT
ConnectX-6 Dx EN adapter card,
25GbE, with active cooling, Dual-port
SFP28, PCIe 4.0 x8, Crypto and Secure
Boot, Tall Bracket
50GbE Cards MCX623102AC-GDAT ConnectX-6 Dx EN adapter card,
50GbE, Dual-port SFP56, PCIe 4.0 x16,
Crypto and Secure Boot, Tall Bracket
100GbE Cards MCX623105AN-CDAT ConnectX-6 Dx EN adapter card,
100GbE, Single-port QSFP56, PCIe 4.0
x16, No Crypto, Tall Bracket
MCX623106AC-CDAT ConnectX-6 Dx EN adapter card,
100GbE, Dual-port QSFP56, PCIe 4.0
x16, Crypto and Secure Boot, Tall
Bracket
MCX623106AN-CDAT ConnectX-6 Dx EN adapter card,
100GbE, Dual-port QSFP56, PCIe 4.0
x16, No Crypto, Tall Bracket
MCX623106AS-CDAT ConnectX-6 Dx EN adapter card,
100GbE, Dual-port QSFP56, PCIe 4.0
x16, Secure Boot, No Crypto, Tall
Bracket
100GbE with PPS In/Out Cards MCX623106PC-CDAT ConnectX-6 Dx EN adapter card,
100GbE, Dual-port QSFP56, with PPS
In/Out, PCIe 4.0 x16, Crypto and
Secure Boot, Tall Bracket

6
Configuration OPN Marketing Description
MCX623106TC-CDAT ConnectX-6 Dx EN adapter card,
100GbE, Dual-port QSFP56, Enhanced-
SyncE & PTP, PPS In/Out, PCIe 4.0
x16, Crypto and Secure Boot, Tall
Bracket
MCX623106GC-CDAT
ConnectX-6 Dx EN adapter card,
100GbE, Dual-port QSFP56, Enhanced-
SyncE & PTP GM support and GNSS,
PPS Out, PCIe 4.0 x16, Crypto and
Secure Boot, Tall Bracket
200GbE Cards MCX623105AC-VDAT ConnectX-6 Dx EN adapter card,
200GbE, Single-port QSFP56, PCIe 4.0
x16, Crypto and Secure Boot, Tall
Bracket
MCX623105AN-VDAT ConnectX-6 Dx EN adapter card,
200GbE, Single-port QSFP56, PCIe 4.0
x16, No Crypto, Tall Bracket
Legacy (EOL) Ordering Part Numbers
OPN Marketing Description
MCX621102AC-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x8, Crypto and Secure Boot, Tall Bracket
MCX623102AC-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623102AN-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX621102AE-ADAT ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe
4.0 x8, Crypto, No Secure Boot, Tall Bracket
MCX623102AN-GDAT ConnectX-6 Dx EN adapter card, 50GbE , Dual-port SFP56, PCIe
4.0 x16, No Crypto, Tall Bracket
MCX623102AE-GDAT ConnectX-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe
4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623102AS-GDAT ConnectX-6 Dx EN adapter card, 50GbE, Dual-port SFP56, PCIe
4.0 x16, Secure Boot, No Crypto, Tall Bracket
MCX623105AC-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Single-port QSFP56,
PCIe 4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623105AE-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Single-port QSFP56,
PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623106AE-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, PCIe
4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623109AC-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Single-port DSFP, PCIe
4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623109AN-CDAT ConnectX-6 Dx EN adapter card, 100GbE , Single-port DSFP, PCIe
4.0 x16, No Crypto, Tall Bracket

7
•
•
OPN Marketing Description
MCX623106GN-CDAT
ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56,
Enhanced-SyncE & PTP GM support and GNSS, PPS Out, PCIe 4.0
x16, No Crypto, Tall Bracket
MCX623106PC-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, Crypto and Secure Boot, Tall Bracket
MCX623106PE-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623106PN-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, No Crypto, Tall Bracket
MCX623106TN-CDAT ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with
PPS In/Out, PCIe 4.0 x16, No Crypto, Tall Bracket
MCX623105AE-VDAT ConnectX-6 Dx EN adapter card, 200GbE, Single-port QSFP56,
PCIe 4.0 x16, Crypto, No Secure Boot, Tall Bracket
MCX623105AS-VDAT ConnectX-6 Dx EN adapter card, 200GbE, Single-port QSFP56,
PCIe 4.0 x16, Secure Boot, No Crypto, Tall Bracket
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic
familiarity with Ethernet network and architecture specifications.
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough
the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchasedNVIDIA M-1 Global Support Services, please see your contract fordetails
regarding Technical Support.
Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek
assistance through their reseller.
Related Documentation
NVIDIA MLNX_OFED for
Linux User Manual and
Release Notes
User Manual and Release Notes describing MLNx_OFED features, performance,
band diagnostic, tools content, and configuration. See NVIDIA MLNX_OFED for
Linux Documentation.
WinOF-2 for
WindowsUser Manual
and Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic,
tools content, and configuration. See WinOF-2 for Windows Documentation.
NVIDIA VMware for
Ethernet User Manual
and Release Notes
User Manual describing the various components of the NVIDIA ConnectX® NATIVE
ESXi stack. SeeVMware® ESXi Documentation.

8
NVIDIA MLNX_OFED for
Linux User Manual and
Release Notes
User Manual and Release Notes describing MLNx_OFED features, performance,
band diagnostic, tools content, and configuration. See NVIDIA MLNX_OFED for
Linux Documentation.
NVIDIA Firmware
Update
NVIDIA firmware update and query utility used to update the firmware.
SeeNVIDIA Firmware Utility (mlxup) Documentation.
NVIDIA Firmware Tools
(MFT) User Manual User Manual describing the set of MFT firmware management tools for a single
node. SeeMFT User Manual.
IEEE Std 802.3
Specification
IEEE Ethernet Specifications
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications.
LinkX Interconnect
Solutions LinkX Ethernet cables and transceivers are designed to maximize the
performance of High-Performance Computing networks, requiring high-
bandwidth, low-latency connections between compute nodes and switch
nodes.NVIDIA offers one of the industry’s broadest portfolio of 40GbE, 56GbE,
100GbE, 200GbE and400GbE cables, includingDirect Attach Copper cables
(DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a
wide range of lengths from 0.5m to 10km. In addition to meeting
Ethernetstandards, NVIDIA tests every product in an end-to-end environment
ensuring a Bit Error Rate of less than 1E-15.Read more atLinkX Cables and
Transceivers.
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.
Revision History
A list of the changes made to this document are provided in.Document Revision History.

9
Introduction
Product Overview
This is the User Manual for Ethernet adapter cards based on the ConnectX®-6 Dx integrated circuit
device.
As the world's most advanced cloud SmartNIC, ConnectX-6 Dx provides up to two ports of 25, 50 or
100Gb/s or a single-port of 200Gb/s Ethernet connectivity, powered by 50Gb/s PAM4 SerDes
technology and PCIe Gen 4.0 host connectivity.ConnectX-6 Dx continues among NVIDIA's innovation
path in scalable cloud fabrics, delivering unparalleled performance and efficiency at every scale.
ConnectX-6 Dx's innovative hardware offload engines, including IPsec and TLS inline data-in-motion
encryption, are ideal for enabling secure network connectivity in modern data-center
environments. Please refer toFeature and Benefits for more details.
ConnectX-6 Dx 25GbE Adapter Cards
OPN Form
Factor/
Dimensions
Data
Transmiss
ion Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Cry
pto
Ro
HS
Bracket
Type
MCX62110
2AC-ADAT
4.89in. x
2.71in
(124.22mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
✔ ✔ ✔ Tall
Bracket
MCX62110
2AN-ADAT
4.89in. x
2.71in
(124.22mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
- - ✔Tall
Bracket
MCX62110
2AN-ADAT
4.89in. x
2.71in
(124.22mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
- - ✔Tall
Bracket
MCX62310
2AC-ADAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX62310
2AN-ADAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX62310
2AS-ADAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
ConnectX-6 Dx 25GbE Adapter Cards with Active Cooling

10
OPN Form
Factor/
Dimensions
Data
Transmiss
ion Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Cry
pto
Ro
HS
Bracket
Type
MCX62120
2AS-ADAT
6.01in. x
2.71in
(152.90mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
✔-✔Tall
Bracket
MCX62120
2AC-ADAT
6.01in. x
2.71in
(152.90mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP28
PCIe Gen 4.0
SERDES @
16.0GT/s x8
✔-✔Tall
Bracket
ConnectX-6 Dx 50GbE Adapter Cards
OPN Form
Factor/
Dimensions
Data
Transmiss
ion Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Cry
pto
Ro
HS
Bracket
Type
MCX62310
2AC-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
50/25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX62310
2AE-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
-✔ ✔ Tall
Bracket
MCX62310
2AN-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX62310
2AS-GDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
25/10/1
GbE
Dual-port
SFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
ConnectX-6 Dx 100GbE Adapter Cards
These cards are optimized for Workstation Environments and include an onboard cooling fan
that meets the acoustic requirements for workstations.


11
OPN Form
Factor/
Dimensions
Data
Transmiss
ion Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Cry
pto
Ro
HS
Bracket
Type
MCX623105
AN-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX623106
AN-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Dual-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
- - ✔Tall
Bracket
MCX623105
AC-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX623106
AC-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Dual-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX623105
AE-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
-✔ ✔ Tall
Bracket
MCX623106
AE-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Dual-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
-✔ ✔ Tall
Bracket
MCX623106
AS-CDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
100/50/25/
10/1 GbE
Dual-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
ConnectX-6 Dx 100GbE Adapter Cards for Timing and Telecommunication Application SMAs
OPN Form
Factor/
Dimensi
ons
Data
Trans
missio
n Rate
No. of
Ports
and
Type
PCIe
Interfac
e
Se
cu
re
Bo
ot
Cr
yp
to
PP
S
In
/
O
ut
SM
As
P
P
S
O
U
T
P
P
S
I
N
Syc
nE
&
PTP
GM
Sup
port
G
N
SS

R
o
H
S
Brac
ket
Type
MCX623
106PN-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
- - ✔ ✔ ✔ - - ✔Tall
Bracke
t

12
OPN Form
Factor/
Dimensi
ons
Data
Trans
missio
n Rate
No. of
Ports
and
Type
PCIe
Interfac
e
Se
cu
re
Bo
ot
Cr
yp
to
PP
S
In
/
O
ut
SM
As
P
P
S
O
U
T
P
P
S
I
N
Syc
nE
&
PTP
GM
Sup
port
G
N
SS

R
o
H
S
Brac
ket
Type
MCX623
106PC-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
✔ ✔ ✔ ✔ ✔ - - ✔Tall
Bracke
t
MCX623
106TN-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
- - ✔ ✔ ✔ ✔ -✔Tall
Bracke
t
MCX623
106TC-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
✔ ✔ ✔ ✔ ✔ ✔ -✔Tall
Bracke
t
MCX623
106GN-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
- - ✔ ✔ -✔ ✔ ✔Tall
Bracke
t
MCX623
106GC-
CDAT
5.59in. x
2.71in
(142.00m
m x
68.90mm)
100/50/
25/10/1
GbE
Dual-
port
QSFP56
PCIe Gen
4.0
SERDES @
16.0GT/s
x16
✔ ✔ ✔ ✔ -✔ ✔ ✔Tall
Bracke
t
ConnectX-6 Dx 200GbE Adapter Cards
OPN Form
Factor/
Dimensions
Data
Transmiss
ion Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Cry
pto
Ro
HS
Bracket
Type
MCX623105
AC-VDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
200/100/50
/25/10/1
GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔ ✔ ✔ Tall
Bracket
MCX623105
AE-VDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
200/100/50
/25/10/1
GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
-✔ ✔ Tall
Bracket
MCX623105
AN-VDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
200/100/50
/25/10/1
GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
--✔Tall
Bracket

13
•
•
•
OPN Form
Factor/
Dimensions
Data
Transmiss
ion Rate
No. of
Ports and
Type
PCIe
Interface
Secure
Boot
Cry
pto
Ro
HS
Bracket
Type
MCX623105
AS-VDAT
5.59in. x
2.71in
(142.00mm x
68.90mm)
200/100/50
/25/10/1
GbE
Single-port
QSFP56
PCIe Gen 4.0
SERDES @
16.0GT/s x16
✔-✔Tall
Bracket
For more detailed information seeSpecifications.
Features and Benefits
Feature Description
PCI Express
(PCIe) PCIe Gen 4.0 SERDES@ 8.0GT/s / 16.0GT/s through x8/x16 Edge Connector
Up to
200GbE NVIDIA adapters comply with the following IEEE 802.3 standards:
200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
– IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
– IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
– IEEE 802.3ba 40 Gigabit Ethernet
– IEEE 802.3by 25 Gigabit Ethernet
– IEEE 802.3ae 10 Gigabit Ethernet
– IEEE 802.3ap based auto-negotiation and KR startup
– IEEE 802.3ad, 802.1AX Link Aggregation
– IEEE 802.1Q, 802.1P VLAN tags and priority
– IEEE 802.1Qau (QCN)
– Congestion Notification
– IEEE 802.1Qaz (ETS)
– IEEE 802.1Qbb (PFC)
– IEEE 802.1Qbg
– IEEE 1588v2
– Jumbo frame support (9.6KB)
Memory PCI Express - stores and accesses Ethernet fabric connection information and packet
data.
SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G device by
Macronix)
Available only in QSFP cards: FRU EEPROM - Stores the parameters and personality of
the card. The EEPROM capacity is 128Kbit. FRU I2C address is (0x50) and is accessible
through the PCIe SMBus (Note: Address 0x58 is reserved.)
This section describes hardware features and capabilities. Please refer to the relevant
driver and/or firmware release notes for feature availability.


14
Feature Description
Overlay
Networks
In order to better scale their networks, data center operators often create overlay networks
that carry traffic from individual virtual machines over logical tunnels in encapsulated formats
such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet
from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-6 Dx
effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading
engines that encapsulate and de-capsulate the overlay protocol.
RDMA
overConver
ged
Ethernet
(RoCE)
ConnectX-6 Dx, utilizing RoCE (RDMA over Converged Ethernet) technology, delivers low-latency
and high-performance over Band and Ethernet networks. Leveraging data center bridging (DCB)
capabilities, as well as ConnectX-6 Dx, advanced congestion control hardware mechanisms,
RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks.
NVIDIA
PeerDirect®
NVIDIA PeerDirect®communication provides high-efficiency RDMA access by eliminating
unnecessary internal data copies between components on the PCIe bus (for example, from GPU
to CPU), and therefore significantly reduces application run time. ConnectX-6 Dx advanced
acceleration technology enables higher cluster efficiency and scalability to tens of thousands of
nodes.
CPU Offload
Adapter functionality enables reduced CPU overhead leaving more CPU resources available for
computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of
Service
(QoS)
Support for port-based Quality of Service enabling various application requirements for latency
and SLA.
Hardware-
based I/
OVirtualizat
ion
ConnectX-6 Dx provides dedicated adapter resources and guaranteed isolation and protection
for virtual machines within the server.
Storage
Acceleration A consolidated compute and storage network achieves significant cost-performance advantages
over multi-fabric networks. Standard block and file access protocols can leverage
• RDMA for high-performance storage access
• NVMe over Fabric offloads for the target machine
SR-IOV ConnectX-6 Dx SR-IOV technology provides dedicated adapter resources and guaranteed
isolation and protection for virtual machines (VM) within the server.
High-
Performance
Acceleratio
ns
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing

15
•
•
•
•
•
•
Feature Description
Time
Sensitive
Applications
NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related
features called “5T”. NVIDIA PTP and 5T software solutions are designed to meet the most
demanding PTP profiles. ConnectX-6 Dx incorporates an integrated Hardware Clock (PHC) that
allows ConnectX-6 Dx to achieve sub 20u Sec accuracy and also offers many timing-related
functions such as time-triggered scheduling or time-based SND accelerations (time-based
ASAP²).
Furthermore, 5T technology enables the software application to transmit fronthaul (ORAN)
compatible in high bandwidth. The PTP part supports the subordinate clock, master clock, and
boundary clock.
ConnectX-6 Dx PTP solution allows you to run any PTP stack on your host.
Enhanced-
SyncE &
PTP
NVIDIA offers ConnectX-6 Dx cards with SyncE support including an improved holdover to meet
ITU-T G.8273.2 class C.
Enabled in MCX623106TN-CDAT, MCX623106TC-CDAT, MCX623106GN-CDAT, and MCX623106GC-
CDAT.
Grand
Master
(GNSS
Reciver)
NVIDIA offers ConnectX-6 Dx with an integrated GNSS receiver to allow a compact and efficient
solution for a Grand Master at every server.
Enabled in MCX623106G[N/C]-CDAT
PPS In/Out
SMAs NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related
features called “5T”. NVIDIA PTP and 5T software solutions are designed to meet the most
demanding PTP profiles. ConnectX-6 Dx incorporates an integrated Hardware Clock (PHC) that
allows ConnectX-6 Dx to achieve sub 20u Sec accuracy and also offers many timing-related
functions such as time-triggered scheduling or time-based SND accelerations (time-based
ASAP²). Furthermore, 5T technology enables the software application to transmit fronthaul
(ORAN) compatible in high bandwidth. The PTP part supports the subordinate clock, master
clock, and boundary clock.
ConnectX-6 Dx PTP solution allows you to run any PTP stack on your host.
With respect to testing and measurements, selected NVIDIA adapters allow you to use the PPS-
out signal from the onboard SMA connecter, ConnectX-6 Dx also allows measuring PTP in scale,
with PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In
signal, and the timestamp will be sent to the SW.
Enabled in MCX623106PN-CDAT, MCX623106PC-CDAT, MCX623106TN-CDAT, MCX623106TC-CDAT,
MCX623106GN-CDAT, and MCX623106GC-CDAT.
Operating Systems/Distributions
RHEL/CentOS
Windows
FreeBSD
VMware
OpenFabrics Enterprise Distribution (MLNX_OFED)
OpenFabrics Windows Distribution (WinOF-2)

16
•
•
•
Connectivity
Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support

17
Interfaces
The below figures show the component side of the ConnectX-6 Dx adapter card. Each numbered
interface that is referenced in the figures is described in the following table with a link to detailed
information.
Callout Item Description
1“ConnectX-6 Dx IC”
ConnectX-6 Dx IC on the board.
2“Ethernet SFP28/SFP56/QSFP56
Interface”
Ethernet traffic is transmitted through the adapter's
SFP28/SFP56/QSFP56 connectors. The networking
connectors allow for the use of modules, optical and
passive cable interconnect solutions.
3"PCI Express Interface" PCIe Gen 3.0/4.0 through an x8/x16 edge
connector.
4“Networking Ports LEDs Interface” There are two I/O LEDs per port to indicate speed
and link status.
“SMBus Interface” Allows BMC connectivity using MCTP over SMBus or
MCTP over PCIe protocols.
“Voltage Regulators” Voltage supply pins that feed onboard regulators.
The below figures are for illustration purposes only and might not reflect the current
revision of the adapter card.


18
•
•
•
•
•
•
•
ConnectX-6 Dx IC Interface
The ConnectX®-6 Dx EN family of adapter IC devicesdelivers two ports of 10/25/40/50/100Gb/s or
a single-port of 200Gb/s Ethernet connectivity paired with best-in-class hardware capabilities that
accelerate and secure cloud and data-center workloads.
Encryption
ConnectX-6 Dx brings security to every end-point, including:
Purpose-built inline acceleration engines that offload IPsec and TLS data-in-motion and XTS-
AES data-at-rest cryptographic operations.
Stateful firewall solution acceleration, powered by Open vSwitch connection tracking and
NVIDIA’s ASAP2 technology.
Embedded hardware root-of-trust and support for RSA-based secure firmware update and
secure boot, providing guaranteed integrity of the network adapter.
Ethernet SFP28 / SFP56 / QSFP56 Interfaces
The network ports of the ConnectX-6 Dx adapter card are compliant with the IEEE 802.3 Ethernet
standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28 /
SFP56 / QSFP56connector on the adaptercard.
PCI Express Interface
ConnectX-6 Dx adapter cards support PCI Express Gen 3.0/4.0 (1.1 and 2.0 compatible) through
x8/16 edge connectors. The device can be either a master initiating the PCI Express bus operations,
or a slave responding to PCI bus operations.
The following lists PCIe interface features:
PCIe Gen 4.0 and 3.0 compliant, 2.0 and 1.1 compatible
2.5, 5.0, 8.0, or 16.0 GT/s link rate x8 or x16 lanes
Auto-negotiates to x16, x8, x4, x2, or x1
Support for MSI/MSI-X mechanisms
Applies to Crypto OPNs only.

The adapter card includes special circuits to protect from ESD shocks to the card/server
when plugging copper cables.


19
Networking Ports LEDs Interface
For the networking ports LEDs description, follow the below table depending on the OPN you have
purchased.
OPN LEDs Scheme
MCX621102A[C/N/E]-ADAT, MCX621202A(C/S)-ADAT Scheme 1: One Bi-Color LED
MCX623102A[C/N]-ADAT, MCX623102A[C/N]-
GDAT,MCX623102A[S/E/N/C]-GDAT
MCX623105A[N/E]-CDAT,MCX623106A[C/N/S/E]-
CDAT,MCX623106P[C/N/E]-CDAT,
MCX623105A[C/N/S/E]-VDAT
Scheme 2: One Bi-Color LED and one Single Color
LED
Scheme 1: One Bi-Color LED
There is one bicolor(Yellow and Green) I/O LED per port to indicate speed and link status.
Link Indications
State Bi-Color LED (Yellow/Green)
Physical link speed
Beacon command for locating the adapter card 1Hz blinking Yellow
Error
4Hz blinking Yellow Indicates an error with the link. The
error can be one of the following:
Error Type Description LED Behavior
I2C
I2C access to the
networking ports
fails
Blinks until error
is fixed
Over-current
Over-current
condition of the
networking ports
Blinks until error
is fixed
Physical Activity
The Green LED will blink.
Link Up
The Green LED will be solid.

20
Scheme 2: One Bi-Color LED and one Single Color LED
There are two I/O LEDs per port to indicate speed and link status. LED1 is a bicolor LED (Yellow and
green) and LED2 is a single color LED (green).
Link Indications
State Bi-Color LED (Yellow/Green) Single Color LED (Green)
Beacon command for locating the
adapter card
1Hz blinking Yellow OFF
Error
4Hz blinking Yellow Indicates an error
with the link. The error can be one of the
following:
Error
Type
Description LED
Behavior
I2C
I2C access to
the
networking
ports fails
Blinks until
error is
fixed
Over-
current
Over-current
condition of
the
networking
ports
Blinks until
error is
fixed
ON
Physical Activity
The Green LED will blink.
Blinking
Link Up
The Green LED will be solid.
ON
SMBus Interface
ConnectX-6 Dx technology maintains support for manageability through a BMC. ConnectX-6 Dx PCIe
stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as
if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific
manageability solution in use by the server, please contact NVIDIA Support.
Voltage Regulators
The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These
voltage supply pins feed on-board regulators that provide the necessary power to the various
components on the card.