Nvidia Mellanox ConnectX-6 Dx Series User manual

Exported on Dec/12/2022 02:23 PM







NVIDIA ConnectX-6 Dx Ethernet Adapter
Cards for OCP Spec 3.0 User Manual


Downloaded from Arrow.com.

2
Table of Contents
Introduction..................................................................................... 9
Products Overview ...................................................................................9
ConnectX-6 Dx 25GbE Adapter Cards for OCP 3.0...........................................9
ConnectX-6 Dx 100GbE Adapter Cards for OCP 3.0 ....................................... 10
ConnectX-6 Dx 200GbE Adapter Cards for OCP 3.0 ....................................... 11
Features and Benefits ............................................................................. 12
Operating Systems/Distributions ................................................................ 14
Connectivity ........................................................................................ 14
Interfaces ......................................................................................15
ConnectX-6 Dx IC................................................................................... 16
Encryption ...................................................................................... 16
NVIDIA Multi-HostTM Support ................................................................ 16
Ethernet Networking Interfaces ................................................................. 17
PCI Express Interface.............................................................................. 17
Networking Ports LEDs Interface ................................................................ 17
SMBus Interface .................................................................................... 18
Voltage Regulators ................................................................................. 19
CPLD Interface ..................................................................................... 19
Heatsink ............................................................................................. 19
Hardware Installation ........................................................................20
Safety Warnings .................................................................................... 20
Installation Procedure Overview................................................................. 21
System Requirements ............................................................................. 21
Hardware Requirements ...................................................................... 21
Airflow Requirements ......................................................................... 21
Software Requirements ....................................................................... 22
Safety Precautions ................................................................................. 22
Pre-Installation Checklist ......................................................................... 22
OCP 3.0 Bracket Replacement Instructions .................................................... 23
OCP 3.0 Adapter Card Installation Instructions................................................ 23
Cards with Thumbscrew (Pull-tab) Bracket................................................ 23
Cards with Internal Lock Bracket............................................................ 24
Downloaded from Arrow.com.Downloaded from Arrow.com.

3
Cards with Ejector-Latch Bracket ........................................................... 24
Cables and Modules................................................................................ 25
Identifying the Card in Your System............................................................. 26
Card Extraction Instructions ..................................................................... 27
Safety Precautions ............................................................................. 22
Cards Extraction Instructions ................................................................ 27
Driver Installation.............................................................................29
Windows Driver Installation ...................................................................... 29
Software Requirements ....................................................................... 29
Downloading WinOF-2 Driver................................................................. 29
Installing WinOF-2 Driver ..................................................................... 30
Firmware Upgrade ............................................................................. 37
Linux Driver Installation .......................................................................... 37
Prerequisites.................................................................................... 37
Downloading NVIDIA OFED .................................................................... 37
Installing MLNX_OFED ......................................................................... 38
Driver Load Upon System Boot............................................................... 43
Additional Installation Procedures .......................................................... 45
Performance Tuning ........................................................................... 51
VMware Driver Installation ....................................................................... 51
Hardware and Software Requirements ..................................................... 51
Installing NATIVE ESXi Driver for VMware vSphere ........................................ 51
Removing Earlier NVIDIA Drivers ............................................................. 52
Firmware Programming ....................................................................... 52
Updating Adapter Firmware .................................................................53
Troubleshooting ...............................................................................54
General Troubleshooting .......................................................................... 54
Linux Troubleshooting ............................................................................. 55
Windows Troubleshooting......................................................................... 55
Specifications..................................................................................57
MCX623432AE-ADAB / MCX623432AN-ADAB / MCX623432AC-ADAB / MCX623432AS-ADAB
Specifications....................................................................................... 57
MCX623432AC-GDAB / MCX623432AN-GDAB / MCX623432AS-GDAI Specifications ....... 58
MCX623435MN-CDAB Specifications ............................................................. 59
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

4
MCX623439MC-CDAB / MCX623439MN-CDAB Specifications ................................. 60
MCX623436AC-CDAB / MCX623436AN-CDAB / MCX623436AS-CDAB / MCX623436MC-
CDAB / MCX623436MN-CDAB / MCX623436MS-CDAB / MCX623436AS-CDAI
Specifications....................................................................................... 61
MCX623430MS-CDAB Specifications ............................................................. 62
MCX623435AC-CDAB / MCX623435AE-CDAB / MCX623435AN-CDAB / MCX623435AC-
VDAB / MCX623435AE-VDAB / MCX623435AN-VDAB / MCX623435AS-VDAI
Specifications....................................................................................... 63
Board Mechanical Drawing and Dimensions.................................................... 64
Bracket Mechanical Drawings and Dimensions ................................................ 64
Monitoring......................................................................................66
Thermal Sensors.................................................................................... 66
Adapter Card Heatsink ............................................................................ 66
Finding the MAC and Serial Number on the Adapter Card ..............................67
Document Revision History ..................................................................70

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

5
About This Manual
This is the User Guide for Ethernet NVIDIA® ConnectX®-6 Dx adapter cards for Open Compute
Project Spec 3.0. These adapters' connectivity provides the highest performing low latency and most
flexible interconnect solution for servers supporting OCP spec 3.0 adapter cards used in Enterprise
Data Centers and High-Performance Computing environments.
Important Note:
In some of the OCP 3.0 prototype samples, the insertion force that is required to install the card
into a particular PCI connector may exceed the maximum insertion force that is allowed by the
connector’s spec. NVIDIA is updating the OCP 3.0 cards to ensure the proper insertion process.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Dx Ethernet
adapter cards for OCP Spec 3.0.
NVIDIA SKU OPN Marketing Description
900-9X624-0073-SB0 MCX623432AS-ADAB ConnectX®-6 Dx EN adapter card,
25GbE, OCP3.0, With Host management,
Dual-port SFP28, PCIe 4.0 x16,Secure
Boot, No Crypto,Thumbscrew (Pull Tab)
Bracket
900-9X624-0085-SB0 MCX623432AC-GDAB ConnectX®-6 Dx EN adapter card,
50GbE, OCP3.0, With Host Management,
Dual-port SFP56, PCIe 4.0 x16, Crypto
and Secure Boot, Thumbscrew (Pull Tab)
Bracket
900-9X658-0016-SB0 MCX623435AN-CDAB ConnectX®-6 Dx EN adapter card,
100GbE, OCP3.0, With Host
management, Single-port QSFP56, PCIe
4.0 x16, No Crypto, Thumbscrew (Pull
Tab) Bracket
900-9X658-0046-SB0 MCX623435AC-CDAB ConnectX-6 Dx EN adapter card,
100GbE, OCP3.0, With Host
Management, Single-portQSFP56, PCIe
4.0 x16, Crypto and Secure Boot,
Thumbscrew (Pull Tab) Bracket
900-9X658-0056-MB0 MCX623436MN-CDAB ConnectX®-6 Dx EN adapter card,
100GbE, OCP3.0, With Host
management, Dual-port QSFP56, Multi-
Host or Socket Direct, PCIe 4.0 x16, No
Crypto, Thumbscrew (Pull Tab) Bracket
900-9X658-0076-MB0 MCX623436MS-CDAB ConnectX®-6 Dx EN adapter card,
100GbE, OCP3.0, With Host
management, Dual-port QSFP56,Multi-
Host or Socket Direct, PCIe 4.0 x16,
Secure Boot, No Crypto, Thumbscrew
(Pull Tab) Bracket
900-9X675-0076-MB0 MCX623430MS-CDAB ConnectX-6 Dx EN adapter card, 100GbE
OCP3.0, With Host Management,
Dual-port DSFP, Multi-Host or Socket
Direct, PCIe 4.0 x16, Secure Boot, No
Crypto, Thumbscrew (Pull Tab) Bracket
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

6
NVIDIA SKU OPN Marketing Description
900-9X658-0086-SB0 MCX623436AC-CDAB ConnectX®-6 Dx EN adapter card,
100GbE, OCP3.0, With Host
Management, Dual-port QSFP56, PCIe
4.0 x16, Crypto and Secure Boot,
Thumbscrew (Pull Tab) Bracket
900-9X658-0056-SB1 MCX623436AN-CDAB ConnectX®-6 Dx EN adapter card,
100GbE, OCP3.0, With Host
Management, Dual-port QSFP56, PCIe
4.0 x16, No Crypto, Thumbscrew (Pull
Tab) Bracket
900-9X658-0046-SI0 MCX623435AC-CDAI ConnectX-6 Dx EN adapter card,
100GbE, OCP3.0, With Host
Management, Single-port QSFP56, PCIe
4.0 x16, Crypto and Secure Boot,
Internal Lock Bracket
900-9X658-0018-SB0 MCX623435AN-VDAB ConnectX®-6 Dx EN adapter card,
200GbE, OCP3.0, With Host
Management, Single-port QSFP56, PCIe
4.0 x16, No Crypto, Thumbscrew (Pull
Tab) Bracket
Legacy (EOL) Ordering Part Numbers
NVIDIA SKU OPN Marketing Description
900-9X624-0063-SB0 MCX623432AE-ADAB ConnectX®-6 Dx EN adapter card, 25GbE, OCP3.0,
With Host Management, Dual-port SFP28, PCIe 4.0
x16, Crypto, No Secure Boot, Thumbscrew (Pull
Tab) Bracket
900-9X624-0075-SI0 MCX623432AS-GDAI ConnectX®-6 Dx EN adapter card, 50GbE, OCP3.0,
With Host Management, Dual-port SFP56, PCIe 4.0
x16, Secure Boot, No Crypto, Internal Lock Bracket
900-9X658-0026-SB0 MCX623435AE-CDAB ConnectX®-6 Dx EN adapter card, 100GbE,
OCP3.0, With Host Management, Single-port
QSFP56, PCIe 4.0 x16, Crypto, No Secure Boot,
Thumbscrew (Pull Tab) Bracket
900-9X658-0038-SI0 MCX623435AS-VDAI ConnectX®-6 Dx EN adapter card, 200GbE,
OCP3.0, With Host Management, Single-port
QSFP56, PCIe 4.0 x16, Secure Boot, No Crypto,
Internal Lock Bracket
900-9X658-0066-SB0 MCX623436AE-CDAB ConnectX-6 Dx EN adapter card, 100GbE, OCP3.0,
With Host Management, Dual-port QSFP56, PCIe
4.0 x16, Crypto, No Secure Boot, Thumbscrew
(Pull Tab) Bracket
900-9X658-0076-SI0 MCX623436AS-CDAI ConnectX®-6 Dx EN adapter card, 100GbE OCP3.0,
With Host Management, Dual-port QSFP56, PCIe
4.0 x16, Secure Boot, No Crypto, Internal Lock
Bracket
900-9X675-0046-MB0 MCX623439MC-CDAB ConnectX®-6 Dx EN adapter card, 100GbE,
OCP3.0,With Host Management, Single-port
QSFP56, Multi-Host or Socket Direct, PCIe 4.0
x16,Crypto and Secure Boot, Thumbscrew(Pull
Tab)Bracket
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

7
•
•
NVIDIA SKU OPN Marketing Description
900-9X624-0083-SB0 MCX623432AC-ADAB ConnectX®-6 Dx EN adapter card, 25GbE, OCP3.0,
With Host Management, Dual-port SFP28, PCIe 4.0
x16, Crypto and Secure Boot, Thumbscrew (Pull
Tab) Bracket
900-9X624-0003-SB0 MCX623432AN-ADAB ConnectX®-6 Dx EN adapter card, 25GbE, OCP3.0,
With Host management, Dual-port SFP28, PCIe 4.0
x16, No Crypto, Thumbscrew (Pull Tab) Bracket
900-9X624-0055-SB0 MCX623432AN-GDAB ConnectX®-6 Dx EN adapter card, 50GbE, OCP3.0,
With Host Management, Dual-port SFP56, PCIe 4.0
x16, No Crypto, Thumbscrew (Pull Tab) Bracket
900-9X658-0048-SB0 MCX623435AC-VDAB ConnectX®-6 Dx EN adapter card, 200GbE,
OCP3.0, With Host Management, Single-port
QSFP56, PCIe 4.0 x16, Crypto and Secure Boot,
Thumbscrew (Pull Tab) Bracket
900-9X658-0016-MB0 MCX623435MN-CDAB ConnectX®-6 Dx EN adapter card, 100GbE,
OCP3.0, With Host Management, Single-port
QSFP56, Multi-Host or Socket Direct, PCIe 4.0 x16,
No Crypto, Thumbscrew (Pull Tab) Bracket
Intended Audience
This manual is intended for the installer and user of these cards. The manual assumes basic
familiarity with the Ethernet network and architecture specifications.
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough
the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchased NVIDIA Global Support Services, please see your contract for details
regarding Technical Support.
Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek
assistance through their reseller.
Related Documentation
NVIDIAOFED for
Linux User Manual
and Release Notes
User Manual describing OFED features, performance, band diagnostic, tools content,
and configuration. See NVIDIA OFED for Linux Documentation.
WinOF-2 for
WindowsUser
Manual and
Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools
content, and configuration. See WinOF-2 for Windows Documentation.
NVIDIAVMware for
Ethernet User
Manual and
Release Notes
User Manual describing the various components of theNVIDIA ConnectX® NATIVE ESXi
stack. See http://www.nvidia.comProducts > Software > Ethernet Drivers > VMware
Driver
NVIDIAFirmware
Utility (mlxup)
User Manual and
Release Notes
NVIDIAfirmware update and query utility used to update the firmware. Seehttp://
www.nvidia.com> Products > Software > Firmware Tools >mlxup Firmware Utility
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

8
NVIDIAFirmware
Tools (MFT) User
Manual
User Manual describing the set of MFT firmware management tools for a single node.
SeeMFT User Manual.
IEEE Std 802.3
Specification
IEEE Ethernet specification at http://standards.ieee.org/
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications at
https://pcisig.com/specifications
Open Compute
Project 3.0
Specification
https://www.opencompute.org/
NVIDIALinkX
Interconnect
Solutions
NVIDIA LinkX cables and transceivers are designed to maximize the performance of
High-Performance Computing networks, requiring high-bandwidth, low-latency
connections between compute nodes and switch nodes. NVIDIA offers one of the
industry’s broadest portfolios of 40Gb/s, 56Gb/s, 100Gb/s, and 200Gb/s cables,
including Direct Attach Copper cables (DACs), copper splitter cables, Active Optical
Cables (AOCs), and transceivers in a wide range of lengths from 0.5m to 10km. NVIDIA
tests every product in an end-to-end environment ensuring a Bit Error Rate of less
than 1E-15.Read more at https://www.nvidia.com/products/interconnect/ethernet-
overview.php
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in MegaBytes.
The use of Mb or Mbits (small b) indicates the size in Megabits. In this document, PCIe is used to
mean PCI Express.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

9
Introduction
This is the User Guide for Ethernet adapter cards based on the ConnectX®-6 Dx integrated circuit
device for OCP Spec 3.0. These adapters connectivity provide the highest performing low latency
and most flexible interconnect solution for servers supporting OCP 3.0used in Enterprise Data
Centers and High-Performance Computing environments.
Important Note:
In some of the OCP 3.0 prototype samples, the insertion force that is required to install the card
into a particular PCI connector may exceed the maximum insertion force that is allowed by the
connector’s spec.NVIDIA is updating the OCP 3.0 cards to ensure the proper insertion process.
The following table provides the ordering part number, port speed, number of ports, and PCI Express
speed.
Products Overview
ConnectX-6 Dx 25GbE Adapter Cards for OCP 3.0
OPN MCX623432AN-
ADAB
MCX623432AC-
ADAB
MCX623432AE-
ADAB
MCX623432AS-
ADAB
Network Connector
Type
Dual-port SFP28 Dual-port SFP28 Dual-port SFP28 Dual-port SFP28
Ethernet Data Rate 10/25 Gb/s
PCI Express
Connectors
PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Retention Mechanism Thumbscrew(Pull Tab) Bracket
RoHS RoHS Compliant RoHS Compliant RoHS Compliant RoHS Compliant
IC Part Number MT28928A0-XCCF-CE MT28928A0-XCCF-
CE
MT28928A0-XCCF-CE MT28928A0-XCCF-CE
Host Management ✔ ✔ ✔ ✔
Crypto -✔ ✔ -
Secure Boot ✔ ✔ -✔
Device ID 4125 for Physical Function (PF)
4126 for Virtual Function (VF)
ConnectX-6 Dx 50GbE Adapter Cards for OCP 3.0
OPN MCX623432AN-GDAB MCX623432AC-GDAB MCX623432AS-GDAI
Network Connector Type Dual-port SFP56 Dual-port SFP56 Dual-port SFP56
Ethernet Data Rate 10/25/50 Gb/s
Please note the following OPNs are shipped with an assembled thumbscrew(pull tab)
bracket. For other retention mechanism brackets, please contact NVIDIA.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

10
OPN MCX623432AN-GDAB MCX623432AC-GDAB MCX623432AS-GDAI
PCI Express Connectors PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Retention Mechanism Thumbscrew(Pull Tab)
Bracket
Thumbscrew(Pull Tab)
Bracket
Internal Lock Bracket
RoHS RoHS Compliant RoHS Compliant RoHS Compliant
IC Part Number MT28928A0-XCCF-CE MT28928A0-XCCF-CE MT28928A0-XCCF-CE
Host Management ✔ ✔ ✔
Crypto -✔-
Secure Boot -✔ ✔
Device ID 4125 for Physical Function (PF)
4126 for Virtual Function (VF)
ConnectX-6 Dx 100GbE Adapter Cards for OCP 3.0
OPN MCX62
3436A
N-
CDAB
MCX62
3436A
C-
CDAB
MCX62
3435M
N-
CDAB
MCX6
23436
MN-
CDAB
MCX6
23436
MS-
CDAB
MCX6
23436
AS-
CDAI
MCX6
23435
AC-
CDAI
MCX6
23435
AC-
CDAB
MCX6
23435
AE-
CDAB
MCX6
23435
AN-
CDAB
MCX62
3439M
C[/N]-
CDAB
MCX6
23430
MS-
CDAB
Netw
ork
Conn
ector
Type
Dual-
port
QSFP56
Dual-
port
QSFP56
Single-
port
QSFP56
Dual-
port
QSFP5
6
Dual-
port
QSFP5
6
Dual-
port
QSFP5
6
Single-
port
QSFP5
6
Single-
port
QSFP5
6
Single-
port
QSFP5
6
Single-
port
QSFP5
6
Single-
port
QSFP56
Dual-
port
QSFP5
6
Ether
net
Data
Rate
10/25/40/50/100 Gb/s
PCI
Expr
ess
Conn
ector
s
PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dime
nsion
s
Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Rete
ntion
Mech
anis
m
Thumb
screw(
Pull
Tab)
Bracke
t
Thumb
screw(
Pull
Tab)
Bracke
t
Thumb
screw(
Pull
Tab)
Bracke
t
Thumb
screw
(Pull
Tab)
Bracke
t
Thumb
screw
(Pull
Tab)
Bracke
t
Intern
al
Lock
Bracke
t
Intern
al
Lock
Bracke
t
Thumb
screw
(Pull
Tab)
Bracke
t
Thumb
screw
(Pull
Tab)
Bracke
t
Thumb
screw
(Pull
Tab)
Bracke
t
Thumb
screw
(Pull
Tab)
Bracke
t
Thumb
screw
(Pull
Tab)
Bracke
t
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

11
OPN MCX62
3436A
N-
CDAB
MCX62
3436A
C-
CDAB
MCX62
3435M
N-
CDAB
MCX6
23436
MN-
CDAB
MCX6
23436
MS-
CDAB
MCX6
23436
AS-
CDAI
MCX6
23435
AC-
CDAI
MCX6
23435
AC-
CDAB
MCX6
23435
AE-
CDAB
MCX6
23435
AN-
CDAB
MCX62
3439M
C[/N]-
CDAB
MCX6
23430
MS-
CDAB
RoHS RoHS
Compli
ant
RoHS
Compli
ant
RoHS
Compli
ant
RoHS
Compli
ant
RoHS
Compl
iant
RoHS
Compl
iant
RoHS
Compl
iant
RoHS
Compl
iant
RoHS
Compl
iant
RoHS
Compli
ant
RoHS
Compli
ant
RoHS
Compl
iant
IC
Part
Num
ber
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
MT289
28A0-
XCCF-
CE
Host
Mana
geme
nt
✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
Cryp
to
-✔- - - - ✔ ✔ ✔ -✔ ✔
Secu
re
Boot
-✔- - ✔ ✔ ✔ ✔ - - ✔ ✔
Mutli
-Host
or
Sock
et
Direc
t
- - ✔ ✔ ✔ - - - - - ✔-
Devi
ce ID
4125 for Physical Function (PF)
4126 for Virtual Function (VF)
ConnectX-6 Dx 200GbEAdapter Cards for OCP 3.0
OPN MCX623435AN-VDAB MCX623435AC-VDAB MCX623435AS-VDAI
Network Connector Type Single-port QSFP56 Single-port QSFP56 Single-port QSFP56
Ethernet Data Rate 10/25/40/50/100/200 Gb/s
PCI Express Connectors PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Retention Mechanism Thumbscrew(Pull Tab)
Bracket
Thumbscrew(Pull Tab)
Bracket
Internal Lock Bracket
RoHS RoHS Compliant RoHS Compliant RoHS Compliant
IC Part Number MT28928A0-XCCF-CE MT28928A0-XCCF-CE MT28928A0-XCCF-CE
Host Management ✔ ✔ ✔
Crypto -✔-
Secure Boot -✔ ✔
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

12
•
•
OPN MCX623435AN-VDAB MCX623435AC-VDAB MCX623435AS-VDAI
Device ID 4125 for Physical Function (PF)
4126 for Virtual Function (VF)
For more detailed information seeSpecifications.
Features and Benefits
Feature Description
PCI Express (PCIe) Uses PCIe Gen 3.0 (8GT/s) or Gen 4.0 (16GT/s) through x16 edge connector.
Up to 200 Gigabit
Ethernet
NVIDIA adapters comply with the following IEEE 802.3 standards:
• 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
Memory Components EEPROM - The EEPROM capacity is 32Kbit. FRU I2C address is (0x50) and is
accessible through the PCIe SMBus (Note: Address 0x58 is reserved.)
SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G
device by Macronix)
Overlay Networks In order to better scale their networks, data center operators often create overlay
networks that carry traffic from individual virtual machines over logical tunnels in
encapsulated formats such as NVGRE and VXLAN. While this solves network scalability
issues, it hides the TCP packet from the hardware offloading engines, placing higher
loads on the host CPU. ConnectX®-6 Dx effectively addresses this by providing
advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-
capsulate the overlay protocol.
RDMA and RDMA
overConverged
Ethernet (RoCE)
ConnectX®-6 Dx, utilizing RDMA (Remote Data Memory Access) and RoCE (RDMA over
Converged Ethernet) technology, delivers low-latency and high-performance over
Band and Ethernet networks. Leveraging data center bridging (DCB) capabilities as
well as ConnectX®-6 Dx advanced congestion control hardware mechanisms, RoCE
provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks.
This section describes hardware features and capabilities. Please refer to the relevant
driver and/or firmware release notes for feature availability.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

13
Feature Description
NVIDIA PeerDirect® PeerDirect®communication provides high-efficiency RDMA access by eliminating
unnecessary internal data copies between components on the PCIe bus (for example,
from GPU to CPU), and therefore significantly reduces application run time.
ConnectX®-6 Dx advanced acceleration technology enables higher cluster efficiency
and scalability to tens of thousands of nodes.
CPU Offload Adapter functionality enables reduced CPU overhead leaving more CPU resources
available for computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of Service
(QoS)
Support for port-based Quality of Service enabling various application requirements
for latency and SLA.
Hardware-based I/
OVirtualization
ConnectX®-6 Dx provides dedicated adapter resources and guaranteed isolation and
protection for virtual machines within the server.
Storage Acceleration A consolidated compute and storage network achieves significant cost-performance
advantages over multi-fabric networks. Standard block and file access protocols can
leverage RDMA for high-performance storage access.
• NVMe over Fabric offloads for the target machine
SR-IOV ConnectX®-6 Dx SR-IOV technology provides dedicated adapter resources and
guaranteed isolation and protection for virtual machines (VM) within the server.
NC-SI The adapter supports a Network Controller Sideband Interface (NC-SI), MCTP over
SMBus and MCTP over PCIe - Baseboard Management Controller interface.
High-
PerformanceAccelera
tions
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing
Host Management NVIDIA host management sideband implementations enable remote monitor and
control capabilities using RBT, MCTP over SMBus, and MCTP over PCIe – Baseboard
Management Controller (BMC) interface, supporting both NC-SI and PLDM management
protocols using these interfaces. NVIDIA OCP 3.0 adapters support these protocols to
offer numerous Host Management features such as PLDM for Firmware Update,
network boot in UEFI driver, UEFI secure boot, and more.
Secure Boot Hardware Root-of-Trust (RoT) Secure Boot and secure firmware update using RSA
cryptography, and cloning-protection, via a device-unique secret key.
Crypto Crypto – IPsec and TLS data-in-motion inline encryption and decryption offload and
AES-XTS block-level data-at-rest encryption and decryption offload.
Wake-on-LAN (WoL) The adapter supported Wake-on-LAN (WoL), a computer networkingstandard that
allows an adapter to be turned on orawakenedby a network message.
In STBY mode, only port0 is available.
Reset-on-LAN (RoL) Supported
NVIDIA Multi-Host NVIDIA®Mellanox®Multi-Host technology enables next-generation Cloud, Web 2.0 and
high-performance data centers to design and build new scale-out heterogeneous
compute and storage racks with direct connectivity between multiple hosts and the
centralized network controller. This enables direct data access with the lowest
latency to significantly improve densities and maximizes data transfer rates. For more
information, please visit NVIDIA Multi-Host Solutions.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

14
•
•
•
•
•
•
•
•
•
Operating Systems/Distributions
RHEL/CentOS
Windows
FreeBSD
VMware
OpenFabrics Enterprise Distribution (OFED)
OpenFabrics Windows Distribution (WinOF-2)
Connectivity
Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

15
Interfaces
The below figures show the component side of the NVIDIA ConnectX-6 Dx adapter card. Each
numbered interface referenced in the figure is described in the following table with a link to
detailed information.
NVIDIAConenctX-6 Dx for OCP 3.0 Adapter Card Interfaces - Component Side
Call
out
Item Description
1ConnectX-6 Dx IC NVIDIAConnectX-6 Dx IC on the board.
2PCI Express
Interface
PCIe Gen 3.0/4.0 through an x16 edge connector.
3Ethernet
Networking
Interface
Ethernet traffic is transmitted through the adapter's SFP28/SFP56/QSFP56
connectors. The networking connectors allow for the use of modules, optical and
passive cable interconnect solutions.
4Networking Ports
LEDs Interface
There are two I/O LEDs, LED0 and LED1, per port to indicate speed and link
status.
FRU EEPROM FRU EEPROM capacity 4Kb
SMBus Interface Allows BMC connectivity using MCTP over SMBus or MCTP over PCIe protocols.
Voltage Regulators Voltage supply pins that feed onboard regulators.
CPLD Interface Controls the networking port LEDs.
The ConnectX-6 Dx adapter card includes special circuits to protect from ESD shocks to the
card/server when plugging copper cables.

The below figures are for illustration purposes only and might not reflect the current
revision of the adapter card.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

16
•
•
•
•
•
•
ConnectX-6 Dx IC
The ConnectX®-6 Dx EN family of adapter IC devices delivers two ports of 10/25/40/50/100Gb/s or
a single port of 200Gb/s Ethernet connectivity paired with best-in-class hardware capabilities that
accelerate and secure cloud and data-center workloads.
Encryption
ConnectX-6 Dx brings security to every end-point, including:
Purpose-built inline acceleration engines that offload IPsec and TLS data-in-motion and XTS-
AES data-at-rest cryptographic operations.
Stateful firewall solution acceleration, powered by Open vSwitch connection tracking and
NVIDIA's ASAP2 technology.
Embedded hardware root-of-trust and support for RSA-based secure firmware update and
secure boot, guaranteeing the network adapter's integrity.
NVIDIA Multi-HostTM Support
In addition to building exceptionally high bandwidth to the data center, the ConnectX-6 Dx device
enables leveraging this speed across the entire data center utilizing its NVIDIA Multi-Host feature.
Using its 32-lane PCI Express interface, a single ConnectX-6 Dx device can provide 200GbE
interconnect for up to four independent hosts without any performance degradation.
The figure below shows a ConnectX-6 Dx device with NVIDIA Multi-Host connected to four separate
hosts, each with a PCIe x4 interface, on one side to a switch on the other side.
The below bifurcation are optional for the adapter's x16 PCIe interface:
x1 PCIe x16, x1 PCIe x8, x1 PCIe x4
x2 PCIe x8, x2 PCIe x4, x2 PCIe x2, x2 PCIe x1
x4 PCIe x4, x4 PCIe x2, x4 PCIe x1
Applies to Crypto-enabled OPNs.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

17
Multi-host capable cards also support Socket-Direct applications and work as regular Single-Host
cards, depending on the type of server they are plugged into, assuming the server complies with the
OCP 3.0 spec.
According to the OCP 3.0 spec, the adapter card advertises its capability through the PRSNTB[3:0]#
pins. The server determines the configuration through the BIF[2:0]# pins, which it drives to the
adapter card.
TheNVIDIA OCP3.0 card has an internal logic that uses the BIF[2:0]# data and determines the
correct operating mode to boot at. The combination of the PRSNTB[3:0]# and BIF[2:0]# pins
deterministically sets the PCIe lane width for a given combination of OCP 3.0 cards and baseboard.
The logic and the decoding table can be found in the OCP 3.0 spec (Chapter 3.5 PCIe Bifurcation
Mechanism.
For example:
the NVIDIA OCP 3.0 Multi-host adapter drives0100 on PRSNTB[3:0]# to the server.
If Server Drivers Adapter PCIe Mode
000 Single-Host Mode: x1 PCIe x16
001 Socket Direct Mode: x2 PCIe x8
010 Socket Direct Mode: x4 PCIe x4
101 Multi-Host Mode: x2 PCIe x8
110 Multi-Host Mode: x4 PCIe x4
Ethernet Networking Interfaces
The network ports of the ConnectX®-6 Dx adapter card are compliant with the IEEE 802.3 Ethernet
standards listed inFeatures and Benefits. Ethernet traffic is transmitted through the networking
connectors on the adapter card.
PCI Express Interface
The table below describes the supported PCIe interface in ConnectX®-6 Dx adapter cards.
Supported PCIe Interface Features
PCIe Gen 3.0/4.0 (1.1 and 2.0 compatible) through x16
edge connectors
Link Rates: 2.5. 5.0, 8.0 or 16GT/s.
Auto Negotiation to: x16, x8, x4, x2 or x1.
Support for MSI/MSI-X mechanisms.
Networking Ports LEDs Interface
There are two I/O LEDs, LED0 and LED1, per port to indicate speed and link status. LED0 is a bicolor
(yellow and green) LED, and LED1 is a single color (green) LED.
The adapter card includes special circuits to protect from ESD shocks to the card/server
when plugging copper cables.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

18
•
•
•
•
•
Link Indications
LED and State Description
1Hz blinking Yellow Beacon command for locating the adapter card
4Hz blinking Yellow Indicates an error with the link. The error can be one of the
following:
Error Type Description LED Behavior
I2CI2C access to the
networking ports fails
Blinks until the error
is fixed
Over-current Over-current condition
of the networking
ports
Blinks until the error
is fixed
LED0 -
Link Speed
A constant Green indicates a link withthe maximum
networking speed.
A constant Yellow indicates a link withless than the
maximum networking speed.
If LED0 is off, then the link has not been established.
LED1 -
Activity
A blinking Green indicates a valid link with data
transfer.
If LED1 is off, then there is no activity
FRU EEPROM
FRU EEPROM allows the baseboard to identify different types of Mezzanine cards. FRU EEPROM is
accessible through SMCLK and SMDATA. FRU EEPROM address is defined according to SLOT_ID0 and
SLOT_ID1, and its capacity is 4Kb.
SMBus Interface
ConnectX®-6 Dx technology maintains support for manageability through a BMC. ConnectX®-6 Dx
OCP 3.0 adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as
if it is a standard NVIDIA OCP 3.0 adapter. For configuring the adapter for the server's specific
manageability solution, please contact NVIDIA Support.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

19
Voltage Regulators
The voltage regulator power is derived from the OCP 3.0 edge connector 12V and 3.3V supply pins.
These voltage supply pins feed onboard regulators that provide the necessary power to the various
components on the card.
CPLDInterface
The adapter card incorporates a CPLD device that controls the networking port LEDs and the scan
chain. It draws its power supply from 3.3V_EDGE.
Heatsink
A heatsink is attached to the ConnectX®-6 Dx IC to dissipate the heat. It is connected by four spring-
loaded push pins that insert into four mounting holes.
ConnectX®-6 Dx IC has a thermal shutdown safety mechanism that automatically shuts down the
ConnectX®-6 Dx card in case of a high-temperature event, improper thermal coupling, or heatsink
removal.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

20
•
•
Hardware Installation
Installation and initialization of ConnectX-6 Dx adapter cards for OCP Spec 3.0 require attention to
electronic equipment's mechanical attributes, power specification, and precautions.
Safety Warnings
Note that not all warnings are relevant to all models.
General Installation Instructions
Read all installation instructions before connecting the equipment to the power source.
Jewelry Removal Warning
Before you install or remove equipment that is connected to power lines, remove jewelry such as
bracelets, necklaces, rings, watches, and so on. Metal objects heat up when connected to power
and ground and can meltdown, causing serious burns and/or welding the metal object to the
terminals.
Over-temperature
This equipment should not be operated in an area with an ambient temperature exceeding the
maximum recommended: 55°C (131°F).An airflow of 200LFM at this maximum ambient
temperature is required for HCA cards and NICs.To guarantee proper airflow, allow at least 8cm (3
inches) of clearance around the ventilation openings.
During Lightning - Electrical Hazard
During periods of lightning activity, do not work on the equipment or connect or disconnect cables.
Copper Cable Connecting/Disconnecting
Some copper cables are heavy and not flexible, as such, they should be carefully attached to or
detached from the connectors. Refer to the cable manufacturer for special warnings and
instructions.
Equipment Installation
This equipment should be installed, replaced, or serviced only by trained and qualified personnel.
Equipment Disposal
The disposal of this equipment should be in accordance to all national laws and regulations.
Local and National Electrical Codes
This equipment should be installed in compliance with local and national electrical codes.
Hazardous Radiation Exposure
Caution – Use of controls or adjustment or performance of procedures other than
those specified herein may result in hazardous radiation exposure.For products with
optical ports.
CLASS 1 LASER PRODUCT and reference to the most recent laser standards:
IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20
Safety warnings are provided here in the English language. For safety warnings in other
languages, refer to the Adapter Installation Safety Instructions.

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.