Nvidia Mellanox ConnectX-5 User manual

Exported on Sep/19/2022 09:36 AM







NVIDIA ConnectX-5 Ethernet Adapter Cards
for OCP Spec 3.0 User Manual



2
Table of Contents
Introduction..................................................................................... 7
Features and Benefits ...............................................................................9
Operating Systems/Distributions ................................................................ 11
Connectivity ........................................................................................ 11
Interfaces ......................................................................................12
Ethernet SFP28 and QSFP28 Interfaces ......................................................... 12
Networking Port LEDS Interface ................................................................. 12
PCI Express Interface.............................................................................. 13
FRU EEPROM ........................................................................................ 13
Heat Sink Interface ................................................................................ 13
SMBus Interface .................................................................................... 14
Voltage Regulators ................................................................................. 14
CPLD Interface ..................................................................................... 14
Thermal Sensors.................................................................................... 14
Hardware Installation ........................................................................15
Safety Warnings .................................................................................... 15
Installation Procedure Overview................................................................. 16
System Requirements ............................................................................. 17
Hardware Requirements ...................................................................... 17
Airflow Requirements ......................................................................... 17
Software Requirements ....................................................................... 17
Safety Precautions ................................................................................. 18
Pre-Installation Checklist ......................................................................... 18
OCP 3.0 Bracket Replacement Instructions .................................................... 18
OCP 3.0 Adapter Card Installation Instructions................................................ 21
Cables and Modules................................................................................ 22
Identifying the Card in Your System............................................................. 22
Cards Extraction Instructions .................................................................... 24
Driver Installation.............................................................................25
Windows Driver Installation ...................................................................... 25
Software Requirements ....................................................................... 25
Downloading NVIDIA WinOF-2 Driver ........................................................ 25

3
Installing NVIDIA WinOF-2 Driver ............................................................ 26
Uninstalling NVIDIA WinOF-2 Driver ......................................................... 33
Extracting Files Without Running Installation ............................................. 34
Firmware Upgrade ............................................................................. 36
Linux Driver Installation .......................................................................... 36
Prerequisites.................................................................................... 36
Downloading NVIDIA OFED .................................................................... 37
Installing NVIDIA OFED ........................................................................ 37
Installing MLNX_OFED Using YUM............................................................ 43
Installing MLNX_OFED Using apt-get Tool .................................................. 45
Updating Firmware After Installation....................................................... 46
UEFI Secure Boot............................................................................... 48
Performance Tuning ........................................................................... 49
VMware Driver Installation ....................................................................... 49
Hardware and Software Requirements ..................................................... 49
Installing NVIDIA NATIVE ESXi Driver for VMware vSphere ............................... 50
Removing Earlier NVIDIA Drivers ............................................................. 50
Firmware Programming ....................................................................... 51
Updating Adapter Firmware .................................................................52
Troubleshooting ...............................................................................53
General Troubleshooting.......................................................................... 53
Linux Troubleshooting ............................................................................. 53
Windows Troubleshooting......................................................................... 54
Specifications..................................................................................55
MCX562A-ACAI / MCX562A-ACAB Specifications............................................... 55
MCX566A-CCAI Specifications .................................................................... 57
MCX565A-CCAB Specifications ................................................................... 59
MCX566A-CDAB / MCX566A-CDAI Specifications............................................... 61
MCX565M-CDAI / MCX565M-CDAB Specifications .............................................. 63
Board Mechanical Drawing and Dimensions.................................................... 65
Brackets Mechanical Drawing and Dimensions ................................................ 66
Finding MAC and Serial Number on the Adapter Card ...................................67
Document Revision History ..................................................................69


4
About This Manual
This is the User Guide for NVIDIA® Ethernet adapter cards based on the ConnectX®-5 integrated
circuit device for Open Compute Project Spec 3.0. These adapters' connectivity provide the highest
performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers
used in Enterprise Data Centers and High-Performance Computing environments.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-5 Ex and
ConnectX-5 Ethernet adapter cards for OCP Spec 3.0.
IC in Use OPN Marketing Description
ConnectX®-5 MCX562A-ACAI ConnectX®-5 EN network interface card for OCP 3.0, with host
management, 25GbE Dual-port SFP28, PCIe3.0 x16, Internal Lock bracket
MCX562A-ACAB ConnectX®-5 EN network interface card for OCP 3.0, with host
management, 25GbE Dual-port SFP28, PCIe3.0 x16, Thumbscrew (Pull
Tab) bracket
MCX566A-CCAI ConnectX®-5 EN network interface card for OCP 3.0, with host
management, 100GbE Dual-port QSFP28, PCIe3.0 x16, Internal Lock
bracket
MCX565A-CCAB ConnectX®-5 EN network interface card for OCP 3.0, with host
management, 100GbE Single-port QSFP28, PCIe3.0 x16, Thumbscrew
(Pull Tab) bracket
ConnectX®-5 Ex MCX566M-GDAI ConnectX®-5 Ex EN network interface card for OCP 3.0 with Multi-Host
or Socket Direct and host management, 50GbE Dual-port QSFP28, PCIe
4.0/3.0 x16, Internal Lock bracket
MCX565M-CDAI
ConnectX®-5 Ex EN network interface card for OCP 3.0, with Multi-Host
or Socket Direct and host management, 100GbE Single-port QSFP28,
PCIe4.0 x16, Internal Lock bracket
MCX565M-CDAB ConnectX®-5 Ex EN network interface card for OCP 3.0, with Multi-Host
or Socket Direct and host management, 100GbE Single-port QSFP28,
PCIe4.0 x16, Thumbscrew (Pull Tab) bracket
MCX566A-CDAI ConnectX®-5 Ex EN network interface card for OCP 3.0 with host
management, 100GbE Dual-port QSFP28, PCIe4.0 x16, Internal Lock
bracket
MCX566A-CDAB
ConnectX®-5 Ex EN network interface card for OCP 3.0, with host
management, 100GbE Dual-port QSFP28, PCIe4.0 x16, Thumbscrew (Pull
Tab) bracket
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic
familiarity with Ethernet network and architecture specifications.
Technical Support

5
•
•
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough
the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchased NVIDIA Global Support Services, please see your contract for details
regarding Technical Support.
Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek
assistance through their reseller.
Related Documentation
NVIDIAMLNX_OFED for Linux User Manual
and Release Notes
User Manual and Release Notesdescribing
MLNx_OFEDfeatures, performance, band diagnostic,
tools content and configuration. SeeNVIDIA MLNX_OFED
for Linux Documentation.
WinOF-2 for WindowsUser Manual and
Release Notes
User Manual describing WinOF-2 features, performance,
Ethernet diagnostic, tools content and configuration.
SeeWinOF-2 for Windows Documentation.
NVIDIA VMware for Ethernet User Manual and
Release Notes
User Manual describing the various components of the
NVIDIA ConnectX® NATIVE ESXi stack.SeeVMware® ESXi
Documentation.
NVIDIA Firmware Utility (mlxup) User Manual
and Release Notes
NVIDIA firmware update and query utility used to update
the firmware.SeeNVIDIA Firmware Utility (mlxup)
Documentation.
NVIDIA Firmware Tools (MFT) User Manual User Manual describing the set of MFT firmware
management tools for a single node. SeeMFT User
Manual.
Open Compute Project 3.0 Specifications https://www.opencompute.org/
IEEE Std 802.3 Specification IEEE Ethernet Specifications
PCI Express Specifications Industry Standard PCI Express Base and Card
Electromechanical Specifications. Refer toPCI-SIG
Specifications.
LinkX Interconnect Solutions LinkX Ethernet cables and transceivers are designed to
maximize the performance of High-Performance
Computing networks, requiring high-bandwidth, low-
latency connections between compute nodes and switch
nodes.NVIDIA offers one of the industry’s broadest
portfolio of 40GbE, 56GbE, 100GbE, 200GbE and400GbE
cables, includingDirect Attach Copper cables (DACs),
copper splitter cables, Active Optical Cables (AOCs) and
transceivers in a wide range of lengths from 0.5m to
10km. In addition to meeting Ethernetstandards, NVIDIA
tests every product in an end-to-end environment
ensuring a Bit Error Rate of less than 1E-15.Read more
atLinkX Cables and Transceivers.

6
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.

7
1.
2.
Introduction
This is the User Guide for Ethernet adapter cards based on the ConnectX®-5 and ConnectX®-5 Ex
integrated circuit device for OCP Spec 3.0. These adapters connectivity provide the highest
performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers
used in Enterprise Data Centers and High-Performance Computing environments.
Important Notes:
ConnectX-5 OCP 3.0 cards were tested for Shock & Vibe in accordance with NVIDIA
specifications and setup, as the OCPspec3.0 available at that time did not contain any S&V
definitions. A newer version of the OCP spec 3.0 has defined S&V specifications and NVIDIA is
in the midst of retesting these cards to comply with OCP spec 3.0.
In some of the OCP 3.0 cards, the insertion force that is required to install the card into a
particular PCI connector may exceed the maximum insertion force that is allowed by the
connector’s spec.NVIDIA is updating the OCP 3.0 cards to ensure the proper insertion
process.
The following tables provide the ordering part number, port speed, number of ports, and PCI Express
speed per ConnectX-5 and ConnectX-5 ExOPNs.
ConnectX-5 Ex Adapter Cards for OCP 3.0
Part Number MCX566M-
GDAI
MCX565M-
CDAI
MCX565M-
CDAB
MCX566A-
CDAI
MCX566A-
CDAB
Ethernet Data Rate
Up to 50Gb/s
Up to 100Gb/s
Up to 100Gb/s Up to 100Gb/s Up to 100Gb/s
Network Connector
Type
x2 QSFP28 ports
x1 QSFP28 port
x1 QSFP28 port x2 QSFP28
ports x2 QSFP28 ports
PCI Express
Connectors
PCIe Gen 3.0 / 4.0 x16 (a), (b)
SERDES @ 8.0GT/s / 16.0GT/s
Bracket
Internal-Lock
Bracket Internal-Lock
Bracket
Pull-tab
Bracket
Internal-Lock
Bracket Pull-tab bracket
Host Management
✔ ✔ ✔ ✔
✔
Multi-Host/ Socket
Direct
✔ ✔ ✔ NA NA
IC Part Number
MT28804A0-
FCCF-CEM
MT28804A0-
FCCF-CEM
MT28804A0-
FCCF-CEM MT28808A0-
FCCF-CE
MT28808A0-
FCCF-CE
Device ID 4121 for Physical Function (PF)
4122 for Virtual Function (VF)

8
Part Number MCX566M-
GDAI
MCX565M-
CDAI
MCX565M-
CDAB
MCX566A-
CDAI
MCX566A-
CDAB
Form Factor
Dimensions
Small Form Factor (SFF)
4.52in. x 2.99in. (115mm x 76mm)
RoHS RoHS Compliant
ConnectX-5 Adapter Cards for OCP 3.0
Part Number MCX562A-
ACAI
MCX562A-
ACAB
MCX566A-CCAI MCX565A-CCAB
Ethernet Data Rate Up to 25Gb/s
Up to 25Gb/s Up to 100Gb/s Up to 100Gb/s
Network Connector
Type
x2 SFP28 ports
x2 SFP28 ports x2 QSFP28 ports x1 QSFP28 ports
PCI Express Connectors PCIe Gen 3.0 x16
SERDES @ 8.0GT/s
Bracket Internal-Lock
Bracket
Pull-tab Bracket Internal-Lock Bracket
Pull-tab Bracket
Host Management ✔ ✔ ✔
✔
IC Part Number MT27808A0-
FCCF-AE
MT27808A0-FCCF-
AE
MT27808A0-FCCF-CE
MT27808A0-FCCF-CE
Device ID 4119 for Physical Function (PF)
4120 for Virtual Function (VF)
Form Factor Dimensions Small Form Factor (SFF)
4.52in. x 2.99in. (115mm x 76mm)
RoHS RoHS Compliant
a. PCIe 4.0 x16 bus can supply a maximum bandwidth of 256Gb/s (=16 *16GT/s, including overhead),
and can support 200Gb/s when both network ports of the card run at 100Gb/s.
b.TheGen4.0 cards have been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be
tested when servers with Gen 4.0 support become available.
For more detailed information seeSpecifications.

9
•
•
Features and Benefits
Feature Description
PCI Express
(PCIe)
Uses PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x8 or x16 edge connector. Gen 1.1
and 2.0 compatible.
Up to 100
Gigabit
Ethernet
NVIDIA adapters comply with the following IEEE 802.3 standards:
• 100GbE/ 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
Memory EEPROM - The EEPROM capacity is 32Kbit.FRU I2C address is (0x50) and is accessible
through the PCIe SMBus (Note: Address 0x58 is reserved.)
SPI - includes 128Mb SPI Flash device (by WINBOND-NUVOTON).
Overlay
Networks
In order to better scale their networks, data center operators often create overlay networks
that carry traffic from individual virtual machines over logical tunnels in encapsulated formats
such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet
from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5
effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines
that encapsulate and de-capsulate the overlay protocol.
RDMA and
RDMA
overConver
ged
Ethernet
(RoCE)
ConnectX-5, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged
Ethernet) technology, delivers low-latency and high-performance over Band and Ethernet
networks. Leveraging data center bridging (DCB) capabilities as well as ConnectX-5 advanced
congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services
over Layer 2 and Layer 3 networks.
NVIDIA
PeerDirect™
PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary
internal data copies between components on the PCIe bus (for example, from GPU to CPU), and
therefore significantly reduces application run time. ConnectX-5 advanced acceleration
technology enables higher cluster efficiency and scalability to tens of thousands of nodes.
This section describes hardware features and capabilities. Please refer to the relevant
driver and/or firmware release notes for feature availability.


10
Feature Description
CPU Offload Adapter functionality enabling reduced CPU overhead allowing more available CPU for
computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of
Service
(QoS)
Support for port-based Quality of Service enabling various application requirements for latency
and SLA.
Hardware-
based I/
OVirtualiza
tion
ConnectX-5 provides dedicated adapter resources and guaranteed isolation and protection for
virtual machines within the server.
Storage
Acceleratio
n
A consolidated compute and storage network achieves significant cost-performance advantages
over multi-fabric networks. Standard block and file access protocols can leverage RDMA for
high-performance storage access.
• NVMe over Fabric offloads for target machine
SR-IOV ConnectX-5 SR-IOV technology provides dedicated adapter resources and guaranteed isolation
and protection for virtual machines (VM) within the server.
NC-SI over
RMII The adapter supports a slave Network Controller Sideband Interface (NC-SI) that can be
connected to a BMC.
High-
Performanc
eAccelerati
ons
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing
Host
Managemen
t
Technology
NVIDIA’s host management technology for standard and multi-host platforms optimizes board
management and power, performance and firmware update management via NC-SI, MCTP over
SMBus and MCTP over PCIe, as well as PLDM for Monitor and Control DSP0248 and PLDM for
Firmware Update DSP0267.
Multi-Host
Technology NVIDIA Multi-Host™ technology, when enabled, allows multiple hosts to be connected into a
single adapter by separating the PCIe interface into multiple and independent interfaces. By
Using NVIDIA Multi Host™, ConnectX-5 lowers the total cost of ownership (TCO) in the data
center by reducing CAPEX (cables, NICs, and switch port expenses), and by reducing OPEX by
cutting down on switch port management and overall power usage. With NVIDIA Multi-Host™
technology powered by a shared buffer architecture, connection-tracking offloads, and RoCE
enhancements, ConnectX-5 offers an extremely flexible solution for today’s demanding data
center and cloud application.
NVIDIA
Socket
Direct™
NVIDIA Socket Direct technology brings improved performance to multi-socket servers by
enabling direct access from each CPU in a multi-socket server to the network through its
dedicated PCIe interface. With this type of configuration, each CPU connects directly to the
network; this enables the interconnect to bypass a QPI (UPI) and the other CPU, optimizing
performance and improving latency. CPU utilization improves as each CPU handles only its own
traffic, and not the traffic from the other CPU. NVIDIA’s OCP 3.0 cards include native support
for socket direct technology for multi-socket servers and can support up to 4 CPU sockets.
Wake-on-
LAN (WoL)
Supported