Nvidia Mellanox ConnectX-6 Dx Series User manual

Exported onOct/22/2020 06:48 PM
https://docs.mellanox.com/x/2wPjAQ
NVIDIA Mellanox ConnectX-6 Dx Ethernet
Adapter Cards for OCP Spec 3.0 User Manual

2
Table of Contents
Table of Contents .......................................................................................................2
Introduction................................................................................................................9
Products Overview................................................................................................................... 9
ConnectX-6 Dx 25GbE Adapter Cards for OCP 3.0........................................................... 9
ConnectX-6 Dx 100GbE Adapter Cards for OCP 3.0....................................................... 10
ConnectX-6 Dx 200GbEAdapter Cards for OCP 3.0....................................................... 11
Features and Benefits........................................................................................................... 11
Operating Systems/Distributions ......................................................................................... 13
Connectivity ........................................................................................................................... 13
Interfaces .................................................................................................................15
ConnectX-6 Dx IC ................................................................................................................. 16
Encryption......................................................................................................................... 16
Ethernet SFP28/SFP56/QSFP56 Interfaces ........................................................................ 16
PCI Express Interface ........................................................................................................... 16
Networking Ports LEDs Interface ........................................................................................ 17
SMBus Interface.................................................................................................................... 18
Voltage Regulators................................................................................................................ 18
CPLDInterface ...................................................................................................................... 18
Heatsink................................................................................................................................. 18
Hardware Installation ..............................................................................................19
Safety Warnings..................................................................................................................... 19
Installation Procedure Overview........................................................................................... 19
System Requirements........................................................................................................... 20
Hardware Requirements ................................................................................................. 20
Airflow Requirements ...................................................................................................... 20
Software Requirements ................................................................................................... 20
Safety Precautions ................................................................................................................ 21
Pre-Installation Checklist..................................................................................................... 21
OCP 3.0 Bracket Replacement Instructions ........................................................................ 22
OCP 3.0 Adapter Card Installation Instructions.............................................................. 22
Cards with Thumbscrew (Pull-tab) Bracket .............................................................22
Cards with Internal Lock Bracket ..............................................................................23

3
Cables and Modules......................................................................................................... 24
Identifying the Card in Your System ................................................................................ 24
Extraction Instructions..................................................................................................... 25
Safety Precautions ......................................................................................................21
Extraction Instructions................................................................................................25
Driver Installation ....................................................................................................27
Windows Driver Installation.................................................................................................. 27
Hardware and Software Requirements........................................................................... 27
Downloading Mellanox WinOF-2 Driver .......................................................................... 27
Installing Mellanox WinOF-2 Driver ................................................................................ 28
Attended Installation...................................................................................................28
Unattended Installation...............................................................................................32
Installation Results .......................................................................................................... 33
Extracting Files Without Running Installation ................................................................ 33
Uninstalling Mellanox WinOF-2 Driver............................................................................ 35
Attended Uninstallation ..............................................................................................35
Unattended Uninstallation..........................................................................................35
Firmware Upgrade ........................................................................................................... 35
Deploying the Driver on a Nano Server........................................................................... 35
Offline Installation.......................................................................................................35
Online Update..............................................................................................................36
Linux Driver Installation........................................................................................................ 37
Prerequisites .................................................................................................................... 37
Downloading Mellanox OFED .......................................................................................... 37
Installing Mellanox OFED ................................................................................................ 38
Installation Script ........................................................................................................ 38
Installation Procedure ................................................................................................ 39
Installation Results .....................................................................................................41
Installation Logs..........................................................................................................41
openibd Script..............................................................................................................41
Driver Load Upon System Boot ..................................................................................42
mlnxofedinstall Return Codes.................................................................................... 42
Uninstalling MLNX_OFED...........................................................................................43
Installing MLNX_OFED Using YUM ................................................................................. 43

4
Setting up MLNX_OFED YUM Repository...................................................................43
Installing MLNX_OFED Using the YUM Tool ..............................................................44
Uninstalling MLNX_OFED Using the YUM Tool .........................................................45
Installing MLNX_OFED Using apt-get Tool..................................................................... 45
Setting up MLNX_OFED apt-get Repository ..............................................................45
Installing MLNX_OFED Using the apt-get Tool..........................................................45
Uninstalling MLNX_OFED Using the apt-get Tool.....................................................46
Updating Firmware After Installation.............................................................................. 46
Updating the Device Online.........................................................................................46
Updating the Device Manually ....................................................................................47
Updating the Device Firmware Automatically upon System Boot ............................47
UEFI Secure Boot............................................................................................................. 48
Enrolling Mellanox's x.509 Public Key on Your Systems...........................................48
Removing Signature from kernel Modules ................................................................ 48
Performance Tuning ........................................................................................................ 49
VMware Driver Installation ................................................................................................... 49
Hardware and Software Requirements........................................................................... 49
Installing Mellanox NATIVE ESXi Driver for VMware vSphere........................................ 50
Removing Earlier Mellanox Drivers................................................................................. 50
Firmware Programming .................................................................................................. 51
Updating Adapter Firmware ....................................................................................52
Troubleshooting .......................................................................................................53
GeneralTroubleshooting ...................................................................................................... 53
LinuxTroubleshooting .......................................................................................................... 53
WindowsTroubleshooting..................................................................................................... 54
Specifications ...........................................................................................................55
MCX623432A[E/N/C]-ADAB Specifications .......................................................................... 55
MCX623432A[N/C]-GDAB/MCX623432AS-GDAISpecifications........................................... 56
MCX623435MN-CDABSpecifications................................................................................... 57
MCX623436A[N/C]-CDAB/MCX623436AS-CDAISpecifications........................................... 59
Board Mechanical Drawing and Dimensions....................................................................... 61
Bracket Mechanical Drawings and Dimensions................................................................. 63
Dual-port SFP28/SFP56 Thumbscrew Bracket ............................................................. 63
Dual-port QSFP56 Thumbscrew Bracket....................................................................... 63

5
Single-port QSFP56 Thumbscrew Bracket.................................................................... 63
Finding the MAC and Serial Number on the Adapter Card ......................................64
Document Revision History ......................................................................................66

6
About This Manual
This is the User Guide for NVIDIA® Mellanox® Ethernet ConnectX®-6 Dx adapter cards for Open
Compute Project Spec 3.0. These adapters connectivity provide the highest performing low latency and
most flexible interconnect solution for servers supporting OCP spec 3.0 adapter cards used in
Enterprise Data Centers and High-Performance Computing environments.
Important Note:
In some of the OCP 3.0 prototype samples, the insertion force that is required to install the card into a
particular PCI connector may exceed the maximum insertion force that is allowed by the connector’s
spec.Mellanox is updating the OCP 3.0 cards to ensure the proper insertion process.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 Dx Ethernet
adapter cards for OCP Spec 3.0.
OPN Marketing Description
MCX623432A
C-ADAB
ConnectX®-6 Dx EN adapter card, 25GbE OCP3.0, With Host Management, Dual-port SFP28,
PCIe 4.0 x16, Crypto and Secure Boot, Thumbscrew (Pull Tab) Bracket
MCX623432A
E-ADAB
ConnectX®-6 Dx EN adapter card, 25GbE OCP3.0, With Host Management, Dual-port SFP28,
PCIe 4.0 x16, Crypto, No Secure Boot, Thumbscrew (Pull Tab) Bracket
MCX623432A
N-ADAB
ConnectX®-6 Dx EN adapter card, 25GbE OCP3.0, With Host management, Dual-port SFP28,
PCIe 4.0 x16, No Crypto, Thumbscrew (Pull Tab) Bracket
MCX623432A
N-GDAB
ConnectX®-6 Dx EN adapter card, 50GbE OCP3.0, With Host Management, Dual-port SFP56,
PCIe 4.0 x16, No Crypto, Thumbscrew (Pull Tab) Bracket
MCX623432A
C-GDAB
ConnectX®-6 Dx EN adapter card, 50GbE OCP3.0, With Host Management, Dual-port SFP56,
PCIe 4.0 x16, Crypto and Secure Boot, Thumbscrew (Pull Tab) Bracket
MCX623432A
S-GDAI
ConnectX®-6 Dx EN adapter card, 50GbE OCP3.0, With Host Management, Dual-port SFP56,
PCIe 4.0 x16, Secure Boot, No Crypto, Internal Lock Bracket
MCX623435M
N-CDAB
ConnectX®-6 Dx EN adapter card, 100GbE, OCP3.0, With Host Management, Single-port
QSFP56, Multi-Host or Socket Direct, PCIe 4.0 x16, No Crypto, Thumbscrew (Pull Tab) Bracket
MCX623436A
C-CDAB
ConnectX®-6 Dx EN adapter card, 100GbE, OCP3.0, With Host Management, Dual-port QSFP56,
PCIe 4.0 x16, Crypto and Secure Boot, Thumbscrew (Pull Tab) Bracket
MCX623436A
N-CDAB
ConnectX®-6 Dx EN adapter card, 100GbE, OCP3.0, With Host Management, Dual-port QSFP56,
PCIe 4.0 x16, No Crypto, Thumbscrew (Pull Tab) Bracket
MCX623436A
S-CDAI
ConnectX®-6 Dx EN adapter card, 100GbE OCP3.0, With Host Management, Dual-port QSFP56,
PCIe 4.0 x16, Secure Boot, No Crypto, Internal Lock Bracket
MCX623435A
C-VDAB
ConnectX®-6 Dx EN adapter card, 200GbE, OCP3.0, With Host Management, Single-port
QSFP56, PCIe 4.0 x16, Crypto and Secure Boot, Thumbscrew (Pull Tab) Bracket
MCX623435A
N-VDAB
ConnectX®-6 Dx EN adapter card, 200GbE, OCP3.0, With Host Management, Single-port
QSFP56, PCIe 4.0 x16, No Crypto, Thumbscrew (Pull Tab) Bracket
MCX623435A
S-VDAI
ConnectX®-6 Dx EN adapter card, 200GbE, OCP3.0, With Host Management, Single-port
QSFP56, PCIe 4.0 x16, Secure Boot, No Crypto, Internal Lock Bracket

7
•
•
•
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic familiarity
with Ethernet network and architecture specifications.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact usthrough
the following methods:
URL:http://www.mellanox.com> Support
E-mail:[email protected]
Tel: [phone]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails
regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seekassistance through their reseller.
Related Documentation
Mellanox OFED for
Linux User Manual
and Release Notes
User Manual describing OFED features, performance, band diagnostic, tools content
and configuration. SeeMellanox OFED for Linux Documentation.
WinOF-2 for
WindowsUser
Manual and Release
Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools
content and configuration. SeeWinOF-2 for Windows Documentation.
Mellanox VMware for
Ethernet User
Manual and Release
Notes
User Manual describing the various components of the Mellanox ConnectX® NATIVE
ESXi stack. See http://www.mellanox.comProducts > Software > Ethernet Drivers >
VMware Driver
Mellanox Firmware
Utility (mlxup) User
Manual and Release
Notes
Mellanox firmware update and query utility used to update the firmware. Seehttp://
www.mellanox.com> Products > Software > Firmware Tools >mlxup Firmware Utility
Mellanox Firmware
Tools (MFT) User
Manual
User Manual describing the set of MFT firmware management tools for a single node.
SeeMFT User Manual.
IEEE Std 802.3
Specification
IEEE Ethernet specification at http://standards.ieee.org/
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications at
https://pcisig.com/specifications
Open Compute
Project 3.0
Specification
https://www.opencompute.org/

8
Mellanox LinkX
Interconnect
Solutions
Mellanox LinkX cables and transceivers are designed to maximize the performance of
High-Performance Computing networks, requiring high-bandwidth, low-latency
connections between compute nodes and switch nodes. Mellanox offers one of
industry’s broadest portfolio of 40Gb/s, 56Gb/s, 100Gb/s and 200Gb/s cables, including
Direct Attach Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs)
and transceivers in a wide range of lengths from 0.5m to 10km. Mellanox tests every
product in an end-to-end environment ensuring a Bit Error Rate of less than 1E-15.Rea
d more at https://www.mellanox.com/products/interconnect/infiniband-overview.php
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.

9
Introduction
This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the ConnectX®-6
Dx integrated circuit device for OCP Spec 3.0. These adapters connectivity provide the highest
performing low latency and most flexible interconnect solution for servers supporting OCP 3.0used in
Enterprise Data Centers and High-Performance Computing environments.
Important Note:
In some of the OCP 3.0 prototype samples, the insertion force that is required to install the card into a
particular PCI connector may exceed the maximum insertion force that is allowed by the connector’s
spec.Mellanox is updating the OCP 3.0 cards to ensure the proper insertion process.
The following table provides the ordering part number, port speed, number of ports, and PCI Express
speed.
Products Overview
ConnectX-6 Dx 25GbE Adapter Cards for OCP 3.0
OPN MCX623432AN-ADAB MCX623432AC-ADAB MCX623432AE-ADAB
Network Connector Type Dual-port SFP28 Dual-port SFP28 Dual-port SFP28
Ethernet Data Rate 10/25 Gb/s
PCI Express Connectors PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Retention Mechanism Thumbscrew(Pull Tab) Bracket
RoHS RoHS Compliant RoHS Compliant RoHS Compliant
IC Part Number MT28928A0-XCCF-CE MT28928A0-XCCF-CE MT28928A0-XCCF-CE
Host Management √ √ √
Crypto - √ √
Secure Boot √ √ -
Device ID 4125 for Physical Function (PF)
4126 for Virtual Function (VF)
ConnectX-6 Dx 50GbE Adapter Cards for OCP 3.0
Please note the following OPNs include a thumbscrew(pull tab) bracket. For other retention
mechanism brackets, please contact Mellanox.


10
OPN MCX623432AN-GDAB MCX623432AC-GDAB MCX623432AS-GDAI
Network Connector Type Dual-port SFP56 Dual-port SFP56 Dual-port SFP56
Ethernet Data Rate 10/25/40/50 Gb/s
PCI Express Connectors PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Retention Mechanism Thumbscrew(Pull Tab)
Bracket
Thumbscrew(Pull Tab)
Bracket
Internal Lock Bracket
RoHS RoHS Compliant RoHS Compliant RoHS Compliant
IC Part Number MT28928A0-XCCF-CE MT28928A0-XCCF-CE MT28928A0-XCCF-CE
Host Management √ √ √
Crypto - √ -
Secure Boot - √ √
Device ID 4125 for Physical Function (PF)
4126 for Virtual Function (VF)
ConnectX-6 Dx 100GbE Adapter Cards for OCP 3.0
OPN MCX623436AN-
CDAB
MCX623436AC-
CDAB
MCX623435MN-
CDAB
MCX623436AS-
CDAI
Network Connector
Type
Dual-port QSFP56 Dual-port QSFP56 Single-port QSFP56 Dual-port QSFP56
Ethernet Data Rate 10/25/40/50/100 Gb/s
PCI Express
Connectors
PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Retention
Mechanism
Thumbscrew(Pull
Tab) Bracket
Thumbscrew(Pull
Tab) Bracket
Thumbscrew(Pull
Tab) Bracket
Internal Lock
Bracket
RoHS RoHS Compliant RoHS Compliant RoHS Compliant RoHS Compliant
IC Part Number MT28928A0-XCCF-
CE
MT28928A0-XCCF-
CE
MT28928A0-XCCF-
CE
MT28928A0-XCCF-
CE
Host Management √ √ √ √
Crypto - √ - -
Secure Boot - √ - √
Mutli-Host or Socket
Direct
- - √ -

11
OPN MCX623436AN-
CDAB
MCX623436AC-
CDAB
MCX623435MN-
CDAB
MCX623436AS-
CDAI
Device ID 4125 for Physical Function (PF)
4126 for Virtual Function (VF)
ConnectX-6 Dx 200GbEAdapter Cards for OCP 3.0
OPN MCX623435AN-VDAB MCX623435AC-VDAB MCX623435AS-VDAI
Network Connector Type Single-port QSFP56 Single-port QSFP56 Single-port QSFP56
Ethernet Data Rate 10/25/40/50/100/200 Gb/s
PCI Express Connectors PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Retention Mechanism Thumbscrew(Pull Tab)
Bracket
Thumbscrew(Pull Tab)
Bracket
Internal Lock Bracket
RoHS RoHS Compliant RoHS Compliant RoHS Compliant
IC Part Number MT28928A0-XCCF-CE MT28928A0-XCCF-CE MT28928A0-XCCF-CE
Host Management √ √ √
Crypto - √ -
Secure Boot - √ √
Device ID 4125 for Physical Function (PF)
4126 for Virtual Function (VF)
For more detailed information seeSpecifications.
Features and Benefits
Feature Description
PCI Express (PCIe) Uses PCIe Gen 3.0 (8GT/s) or Gen 4.0 (16GT/s) through x16 edge connector.
This section describes hardware features and capabilities. Please refer to the relevant driver
and/or firmware release notes for feature availability.


12
•
•
Feature Description
Up to 200 Gigabit
Ethernet
Mellanox adapters comply with the following IEEE 802.3 standards:
• 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
Memory Components EEPROM - The EEPROM capacity is 32Kbit
SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G
device by Macronix)
Overlay Networks In order to better scale their networks, data center operators often create overlay
networks that carry traffic from individual virtual machines over logical tunnels in
encapsulated formats such as NVGRE and VXLAN. While this solves network scalability
issues, it hides the TCP packet from the hardware offloading engines, placing higher
loads on the host CPU. ConnectX®-6 Dx effectively addresses this by providing
advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-
capsulate the overlay protocol.
RDMA and RDMA
overConverged
Ethernet (RoCE)
ConnectX®-6 Dx, utilizing RDMA (Remote Data Memory Access) and RoCE (RDMA over
Converged Ethernet) technology, delivers low-latency and high-performance over Band
and Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as
ConnectX®-6 Dx advanced congestion control hardware mechanisms, RoCE provides
efficient low-latency RDMA services over Layer 2 and Layer 3 networks.
Mellanox
PeerDirect®
PeerDirect® communication provides high-efficiency RDMA access by eliminating
unnecessary internal data copies between components on the PCIe bus (for example,
from GPU to CPU), and therefore significantly reduces application run time.
ConnectX®-6 Dx advanced acceleration technology enables higher cluster efficiency
and scalability to tens of thousands of nodes.
CPU Offload Adapter functionality enabling reduced CPU overhead allowing more available CPU for
computation tasks.
Open vSwitch (OVS) offload using ASAP2 - Accelerated Switch and Packet Processing®
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of Service
(QoS)
Support for port-based Quality of Service enabling various application requirements for
latency and SLA.
Hardware-based I/
OVirtualization
ConnectX®-6 Dx provides dedicated adapter resources and guaranteed isolation and
protection for virtual machines within the server.

13
•
•
•
•
•
•
•
•
•
Feature Description
Storage Acceleration A consolidated compute and storage network achieves significant cost-performance
advantages over multi-fabric networks. Standard block and file access protocols can
leverage RDMA for high-performance storage access.
• NVMe over Fabric offloads for the target machine
• Erasure Coding
• T10-DIF Signature Handover
SR-IOV ConnectX®-6 Dx SR-IOV technology provides dedicated adapter resources and
guaranteed isolation and protection for virtual machines (VM) within the server.
NC-SI The adapter supports a Network Controller Sideband Interface (NC-SI), MCTP over
SMBus and MCTP over PCIe - Baseboard Management Controller interface.
High-
PerformanceAcceler
ations
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing
Host Management Mellanox host management sideband implementations enable remote monitor and
control capabilities using RBT, MCTP over SMBus, and MCTP over PCIe – Baseboard
Management Controller (BMC) interface, supporting both NC-SI and PLDM
management protocols using these interfaces. Mellanox OCP 3.0 adapters support
these protocols to offer numerous Host Management features such as PLDM for
Firmware Update, network boot in UEFI driver, UEFI secure boot, and more.
Multi-Host or Socket
Direct
TBD: Arnon
Secure Boot Hardware Root-of-Trust (RoT) Secure Boot and secure firmware update using RSA
cryptography, and cloning-protection, via a device-unique secret key.
Crypto Crypto – IPsec and TLS data-in-motion inline encryption and decryption offload and
AES-XTS block-level data-at-rest encryption and decryption offload.
Wake-on-LAN (WoL) The adapter supported Wake-on-LAN (WoL), a computer networkingstandard that
allows an adapter to be turned on orawakenedby a network message.
In STBY mode, only port0 is available.
Reset-on-LAN (RoL) Supported
Operating Systems/Distributions
RHEL/CentOS
Windows
FreeBSD
VMware
OpenFabrics Enterprise Distribution (OFED)
OpenFabrics Windows Distribution (WinOF-2)
Connectivity
Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support

14

15
Interfaces
The below figures show the component side of the Mellanox ConnectX-6 Dx adapter card. Each
numbered interface that is referenced in the figures is described in the following table with a link to
detailed information.
Mellanox ConenctX-6 Dx for OCP 3.0 Adapter Card Interfaces - Component Side
Cal
lou
t
Item Description
1“ConnectX-6 Dx IC” Mellanox ConnectX-6 Dx IC on the board.
2“PCI Express
Interface"
PCIe Gen 3.0/4.0 through an x16 edge connector.
3“Ethernet SFP28/
SFP56/QSFP56
Interface”
Ethernet traffic is transmitted through the adapter's SFP28/SFP56/QSFP56
connectors. The networking connectors allow for the use of modules, optical and
passive cable interconnect solutions.
4“Networking Ports
LEDs Interface”
There are two I/O LEDs, LED0 and LED1, per port to indicate speed and link status.
“FRU EEPROM” FRU EEPROM capacity 4Kb
The ConnectX-6 Dx adapter card includes special circuits to protect from ESD shocks to the
card/server when plugging copper cables

The below figures are for illustration purposes only and might not reflect the current revision
of the adapter card.


16
•
•
•
Cal
lou
t
Item Description
“SMBus Interface” Allows BMC connectivity using MCTP over SMBus or MCTP over PCIe protocols.
“Voltage
Regulators”
Voltage supply pins that feed onboard regulators.
“CPLD Interface” Controls the networking port LEDs.
ConnectX-6 Dx IC
The ConnectX®-6 Dx EN family of adapter IC devices delivers two ports of 10/25/40/50/100Gb/s or a
single-port of 200Gb/s Ethernet connectivity paired with best-in-class hardware capabilities that
accelerate and secure cloud and data-center workloads.
Encryption
ConnectX-6 Dx brings security to every end-point, including:
Purpose-built inline acceleration engines that offload IPsec and TLS data-in-motion and XTS-
AES data-at-rest cryptographic operations.
Stateful firewall solution acceleration, powered by Open vSwitch connection tracking and
Mellanox’s ASAP2 technology.
Embedded hardware root-of-trust and support for RSA-based secure firmware update and
secure boot, providing guaranteed integrity of the network adapter.
Ethernet SFP28/SFP56/QSFP56 Interfaces
The network ports of the ConnectX®-6 Dx adapter card are compliant with the IEEE 802.3 Ethernet
standards listed inFeatures and Benefits. Ethernet traffic is transmitted through the QSFP56
connectors on the adaptercard.
PCI Express Interface
The table below describes the supported PCIe interface in ConnectX®-6 Dx adapter cards.
Applies to Crypto-enabled OPNs.

The adapter card includes special circuits to protect from ESD shocks to the card/server when
plugging copper cables.


17
•
•
•
•
•
Supported PCIe Interface Features
PCIe Gen 3.0/4.0 (1.1 and 2.0 compatible) through x16
edge connectors
Link Rates: 2.5. 5.0, 8.0 or 16GT/s.
Auto Negotiation to: x16, x8, x4, x2 or x1.
Support for MSI/MSI-X mechanisms.
Networking Ports LEDs Interface
There are two I/O LEDs, LED0 and LED1, per port to indicate speed and link status. LED0 is bicolor
(yellow and green) LED and LED1 is a single color (green) LED.
Link Indications
LED and State Description
1Hz blinking Yellow Beacon command for locating the adapter card
4Hz blinking Yellow Indicates an error with the link. The error can be one of the
following:
Error Type Description LED Behavior
I2C I2C access to the
networking ports fails
Blinks until the error
is fixed
Over-current Over-current condition
of the networking ports
Blinks until the error
is fixed
LED0 -
Link Speed
A constant Green indicates a link withthe maximum
networking speed.
A constant Yellow indicates a link withless than the
maximum networking speed
If LED0 is off, then the link has not been established.
LED1 -
Activity
A blinking Green indicates a valid link with data transfer.
If LED1 is off, then there is no activity
FRU EEPROM
FRU EEPROM allows the baseboard to identify different types of Mezzanine cards. FRU EEPROM is
accessible through SMCLK and SMDATA. FRU EEPROM address is defined according to SLOT_ID0 and
SLOT_ID1 and its capacity is 4Kb.

18
SMBus Interface
ConnectX®-6 Dx technology maintains support for manageability through a BMC. ConnectX®-6 Dx
OCP 3.0 adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as
if it is a standard Mellanox OCP 3.0 adapter. For configuring the adapter for the specific manageability
solution in use by the server, please contact Mellanox Support.
Voltage Regulators
The voltage regulator power is derived from the OCP 3.0 edge connector 12V and 3.3V supply pins.
These voltage supply pins feed onboard regulators that provide the necessary power to the various
components on the card.
CPLDInterface
The adapter card incorporates a CPLD device that controls the networking port LEDs and the scan
chain. It draws its power supply from 3.3V_EDGE.
Heatsink
A heatsink is attached to the ConnectX®-6 Dx IC in order to dissipate the heat. It is attached either by
using four spring-loaded push pins that insert into four mounting holes.
ConnectX®-6 Dx IC has a thermal shutdown safety mechanism that automatically shuts down the
ConnectX®-6 Dx card in case of a high-temperature event, improper thermal coupling or heatsink
removal.

19
Hardware Installation
Installation and initialization of ConnectX-6 Dx adapter cards for OCP Spec 3.0 require attention to the
mechanical attributes, power specification, and precautions for electronic equipment.
Safety Warnings
Unable to render include or excerpt-include. Could not retrieve page.
Installation Procedure Overview
The installation procedure of ConnectX-6 Dx adapter cards for OCP Spec 3.0 involves the following
steps:
St
ep
Procedure Direct Link
1 Check the system’s hardware and software
requirements.
Refer to System Requirements
2 Pay attention to the airflow consideration within the
host system
Refer to Airflow Requirements
3 Follow the safety precautions Refer to Safety Precautions
4 Follow the pre-installation checklist Refer to Pre-Installation Checklist
5 (Optional) Replace the assembled OCP 3.0 bracket
with the desired form factor bracket
Refer toOCP 3.0 Bracket Replacement
Instructions
6 Install ConnectX-6 Dx adapter card for OCP spec 3.0 in
the system
Refer to Cards with Internal Lock Bracket
Installation
Refer to Cards with Internal Lock Bracket
Installation
7 Connect cables or modules to the card Refer to Cables and Modules
8 Identify ConnectX-6 Dx adapter card in the system Refer to Identify the Card in Your System
Safety warnings are provided here in the English language. For safety warnings in other
languages, refer to the Adapter Installation Safety Instructions document available on
mellanox.com.


20
•
System Requirements
Hardware Requirements
A system with a PCI Express x16 slot for OCP spec 3.0 is required for installing the card.
Airflow Requirements
ConnectX-6 Dx adapter cards are offered with two airflow patterns: from the heatsink to the network
ports, and vice versa, as shown below.
Please refer to the "Specifications" chapter for airflow numbers for each specific card model.
Hot Aisle Cooling
Heatsink-to-port AirflowDirection
Cold Aisle Cooling
Port-to-heatsink Airflow Direction
Software Requirements
See Operating Systems/Distributions section under the Introduction section.
Unless otherwise specified, Mellanox products are designed to work in an environmentally
controlled data center with low levels of gaseous and dust (particulate) contamination.
The operating environment should meet severity level G1 as per ISA 71.04 for gaseous
contamination and ISO 14644-1 class 8 for cleanliness level.

For proper operation and performance, please make sure to use a PCIe slot with a
corresponding bus width and that can supply sufficient power to your card. Refer to the
Specifications section of the manual for more power requirements.

All cards in the system should be planned with the same airflow direction.
