Nvidia Mellanox ConnectX-5 User manual

Exported on Sep/19/2022 09:36 AM







NVIDIA ConnectX-5 Ethernet Adapter Cards
for OCP Spec 3.0 User Manual



2
Table of Contents
Introduction..................................................................................... 7
Features and Benefits ...............................................................................9
Operating Systems/Distributions ................................................................ 11
Connectivity ........................................................................................ 11
Interfaces ......................................................................................12
Ethernet SFP28 and QSFP28 Interfaces ......................................................... 12
Networking Port LEDS Interface ................................................................. 12
PCI Express Interface.............................................................................. 13
FRU EEPROM ........................................................................................ 13
Heat Sink Interface ................................................................................ 13
SMBus Interface .................................................................................... 14
Voltage Regulators ................................................................................. 14
CPLD Interface ..................................................................................... 14
Thermal Sensors.................................................................................... 14
Hardware Installation ........................................................................15
Safety Warnings .................................................................................... 15
Installation Procedure Overview................................................................. 16
System Requirements ............................................................................. 17
Hardware Requirements ...................................................................... 17
Airflow Requirements ......................................................................... 17
Software Requirements ....................................................................... 17
Safety Precautions ................................................................................. 18
Pre-Installation Checklist ......................................................................... 18
OCP 3.0 Bracket Replacement Instructions .................................................... 18
OCP 3.0 Adapter Card Installation Instructions................................................ 21
Cables and Modules................................................................................ 22
Identifying the Card in Your System............................................................. 22
Cards Extraction Instructions .................................................................... 24
Driver Installation.............................................................................25
Windows Driver Installation ...................................................................... 25
Software Requirements ....................................................................... 25
Downloading NVIDIA WinOF-2 Driver ........................................................ 25

3
Installing NVIDIA WinOF-2 Driver ............................................................ 26
Uninstalling NVIDIA WinOF-2 Driver ......................................................... 33
Extracting Files Without Running Installation ............................................. 34
Firmware Upgrade ............................................................................. 36
Linux Driver Installation .......................................................................... 36
Prerequisites.................................................................................... 36
Downloading NVIDIA OFED .................................................................... 37
Installing NVIDIA OFED ........................................................................ 37
Installing MLNX_OFED Using YUM............................................................ 43
Installing MLNX_OFED Using apt-get Tool .................................................. 45
Updating Firmware After Installation....................................................... 46
UEFI Secure Boot............................................................................... 48
Performance Tuning ........................................................................... 49
VMware Driver Installation ....................................................................... 49
Hardware and Software Requirements ..................................................... 49
Installing NVIDIA NATIVE ESXi Driver for VMware vSphere ............................... 50
Removing Earlier NVIDIA Drivers ............................................................. 50
Firmware Programming ....................................................................... 51
Updating Adapter Firmware .................................................................52
Troubleshooting ...............................................................................53
General Troubleshooting.......................................................................... 53
Linux Troubleshooting ............................................................................. 53
Windows Troubleshooting......................................................................... 54
Specifications..................................................................................55
MCX562A-ACAI / MCX562A-ACAB Specifications............................................... 55
MCX566A-CCAI Specifications .................................................................... 57
MCX565A-CCAB Specifications ................................................................... 59
MCX566A-CDAB / MCX566A-CDAI Specifications............................................... 61
MCX565M-CDAI / MCX565M-CDAB Specifications .............................................. 63
Board Mechanical Drawing and Dimensions.................................................... 65
Brackets Mechanical Drawing and Dimensions ................................................ 66
Finding MAC and Serial Number on the Adapter Card ...................................67
Document Revision History ..................................................................69


4
About This Manual
This is the User Guide for NVIDIA® Ethernet adapter cards based on the ConnectX®-5 integrated
circuit device for Open Compute Project Spec 3.0. These adapters' connectivity provide the highest
performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers
used in Enterprise Data Centers and High-Performance Computing environments.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-5 Ex and
ConnectX-5 Ethernet adapter cards for OCP Spec 3.0.
IC in Use OPN Marketing Description
ConnectX®-5 MCX562A-ACAI ConnectX®-5 EN network interface card for OCP 3.0, with host
management, 25GbE Dual-port SFP28, PCIe3.0 x16, Internal Lock bracket
MCX562A-ACAB ConnectX®-5 EN network interface card for OCP 3.0, with host
management, 25GbE Dual-port SFP28, PCIe3.0 x16, Thumbscrew (Pull
Tab) bracket
MCX566A-CCAI ConnectX®-5 EN network interface card for OCP 3.0, with host
management, 100GbE Dual-port QSFP28, PCIe3.0 x16, Internal Lock
bracket
MCX565A-CCAB ConnectX®-5 EN network interface card for OCP 3.0, with host
management, 100GbE Single-port QSFP28, PCIe3.0 x16, Thumbscrew
(Pull Tab) bracket
ConnectX®-5 Ex MCX566M-GDAI ConnectX®-5 Ex EN network interface card for OCP 3.0 with Multi-Host
or Socket Direct and host management, 50GbE Dual-port QSFP28, PCIe
4.0/3.0 x16, Internal Lock bracket
MCX565M-CDAI
ConnectX®-5 Ex EN network interface card for OCP 3.0, with Multi-Host
or Socket Direct and host management, 100GbE Single-port QSFP28,
PCIe4.0 x16, Internal Lock bracket
MCX565M-CDAB ConnectX®-5 Ex EN network interface card for OCP 3.0, with Multi-Host
or Socket Direct and host management, 100GbE Single-port QSFP28,
PCIe4.0 x16, Thumbscrew (Pull Tab) bracket
MCX566A-CDAI ConnectX®-5 Ex EN network interface card for OCP 3.0 with host
management, 100GbE Dual-port QSFP28, PCIe4.0 x16, Internal Lock
bracket
MCX566A-CDAB
ConnectX®-5 Ex EN network interface card for OCP 3.0, with host
management, 100GbE Dual-port QSFP28, PCIe4.0 x16, Thumbscrew (Pull
Tab) bracket
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic
familiarity with Ethernet network and architecture specifications.
Technical Support

5
•
•
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough
the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchased NVIDIA Global Support Services, please see your contract for details
regarding Technical Support.
Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek
assistance through their reseller.
Related Documentation
NVIDIAMLNX_OFED for Linux User Manual
and Release Notes
User Manual and Release Notesdescribing
MLNx_OFEDfeatures, performance, band diagnostic,
tools content and configuration. SeeNVIDIA MLNX_OFED
for Linux Documentation.
WinOF-2 for WindowsUser Manual and
Release Notes
User Manual describing WinOF-2 features, performance,
Ethernet diagnostic, tools content and configuration.
SeeWinOF-2 for Windows Documentation.
NVIDIA VMware for Ethernet User Manual and
Release Notes
User Manual describing the various components of the
NVIDIA ConnectX® NATIVE ESXi stack.SeeVMware® ESXi
Documentation.
NVIDIA Firmware Utility (mlxup) User Manual
and Release Notes
NVIDIA firmware update and query utility used to update
the firmware.SeeNVIDIA Firmware Utility (mlxup)
Documentation.
NVIDIA Firmware Tools (MFT) User Manual User Manual describing the set of MFT firmware
management tools for a single node. SeeMFT User
Manual.
Open Compute Project 3.0 Specifications https://www.opencompute.org/
IEEE Std 802.3 Specification IEEE Ethernet Specifications
PCI Express Specifications Industry Standard PCI Express Base and Card
Electromechanical Specifications. Refer toPCI-SIG
Specifications.
LinkX Interconnect Solutions LinkX Ethernet cables and transceivers are designed to
maximize the performance of High-Performance
Computing networks, requiring high-bandwidth, low-
latency connections between compute nodes and switch
nodes.NVIDIA offers one of the industry’s broadest
portfolio of 40GbE, 56GbE, 100GbE, 200GbE and400GbE
cables, includingDirect Attach Copper cables (DACs),
copper splitter cables, Active Optical Cables (AOCs) and
transceivers in a wide range of lengths from 0.5m to
10km. In addition to meeting Ethernetstandards, NVIDIA
tests every product in an end-to-end environment
ensuring a Bit Error Rate of less than 1E-15.Read more
atLinkX Cables and Transceivers.

6
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.

7
1.
2.
Introduction
This is the User Guide for Ethernet adapter cards based on the ConnectX®-5 and ConnectX®-5 Ex
integrated circuit device for OCP Spec 3.0. These adapters connectivity provide the highest
performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers
used in Enterprise Data Centers and High-Performance Computing environments.
Important Notes:
ConnectX-5 OCP 3.0 cards were tested for Shock & Vibe in accordance with NVIDIA
specifications and setup, as the OCPspec3.0 available at that time did not contain any S&V
definitions. A newer version of the OCP spec 3.0 has defined S&V specifications and NVIDIA is
in the midst of retesting these cards to comply with OCP spec 3.0.
In some of the OCP 3.0 cards, the insertion force that is required to install the card into a
particular PCI connector may exceed the maximum insertion force that is allowed by the
connector’s spec.NVIDIA is updating the OCP 3.0 cards to ensure the proper insertion
process.
The following tables provide the ordering part number, port speed, number of ports, and PCI Express
speed per ConnectX-5 and ConnectX-5 ExOPNs.
ConnectX-5 Ex Adapter Cards for OCP 3.0
Part Number MCX566M-
GDAI
MCX565M-
CDAI
MCX565M-
CDAB
MCX566A-
CDAI
MCX566A-
CDAB
Ethernet Data Rate
Up to 50Gb/s
Up to 100Gb/s
Up to 100Gb/s Up to 100Gb/s Up to 100Gb/s
Network Connector
Type
x2 QSFP28 ports
x1 QSFP28 port
x1 QSFP28 port x2 QSFP28
ports x2 QSFP28 ports
PCI Express
Connectors
PCIe Gen 3.0 / 4.0 x16 (a), (b)
SERDES @ 8.0GT/s / 16.0GT/s
Bracket
Internal-Lock
Bracket Internal-Lock
Bracket
Pull-tab
Bracket
Internal-Lock
Bracket Pull-tab bracket
Host Management
✔ ✔ ✔ ✔
✔
Multi-Host/ Socket
Direct
✔ ✔ ✔ NA NA
IC Part Number
MT28804A0-
FCCF-CEM
MT28804A0-
FCCF-CEM
MT28804A0-
FCCF-CEM MT28808A0-
FCCF-CE
MT28808A0-
FCCF-CE
Device ID 4121 for Physical Function (PF)
4122 for Virtual Function (VF)

8
Part Number MCX566M-
GDAI
MCX565M-
CDAI
MCX565M-
CDAB
MCX566A-
CDAI
MCX566A-
CDAB
Form Factor
Dimensions
Small Form Factor (SFF)
4.52in. x 2.99in. (115mm x 76mm)
RoHS RoHS Compliant
ConnectX-5 Adapter Cards for OCP 3.0
Part Number MCX562A-
ACAI
MCX562A-
ACAB
MCX566A-CCAI MCX565A-CCAB
Ethernet Data Rate Up to 25Gb/s
Up to 25Gb/s Up to 100Gb/s Up to 100Gb/s
Network Connector
Type
x2 SFP28 ports
x2 SFP28 ports x2 QSFP28 ports x1 QSFP28 ports
PCI Express Connectors PCIe Gen 3.0 x16
SERDES @ 8.0GT/s
Bracket Internal-Lock
Bracket
Pull-tab Bracket Internal-Lock Bracket
Pull-tab Bracket
Host Management ✔ ✔ ✔
✔
IC Part Number MT27808A0-
FCCF-AE
MT27808A0-FCCF-
AE
MT27808A0-FCCF-CE
MT27808A0-FCCF-CE
Device ID 4119 for Physical Function (PF)
4120 for Virtual Function (VF)
Form Factor Dimensions Small Form Factor (SFF)
4.52in. x 2.99in. (115mm x 76mm)
RoHS RoHS Compliant
a. PCIe 4.0 x16 bus can supply a maximum bandwidth of 256Gb/s (=16 *16GT/s, including overhead),
and can support 200Gb/s when both network ports of the card run at 100Gb/s.
b.TheGen4.0 cards have been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be
tested when servers with Gen 4.0 support become available.
For more detailed information seeSpecifications.

9
•
•
Features and Benefits
Feature Description
PCI Express
(PCIe)
Uses PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x8 or x16 edge connector. Gen 1.1
and 2.0 compatible.
Up to 100
Gigabit
Ethernet
NVIDIA adapters comply with the following IEEE 802.3 standards:
• 100GbE/ 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
Memory EEPROM - The EEPROM capacity is 32Kbit.FRU I2C address is (0x50) and is accessible
through the PCIe SMBus (Note: Address 0x58 is reserved.)
SPI - includes 128Mb SPI Flash device (by WINBOND-NUVOTON).
Overlay
Networks
In order to better scale their networks, data center operators often create overlay networks
that carry traffic from individual virtual machines over logical tunnels in encapsulated formats
such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet
from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5
effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines
that encapsulate and de-capsulate the overlay protocol.
RDMA and
RDMA
overConver
ged
Ethernet
(RoCE)
ConnectX-5, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged
Ethernet) technology, delivers low-latency and high-performance over Band and Ethernet
networks. Leveraging data center bridging (DCB) capabilities as well as ConnectX-5 advanced
congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services
over Layer 2 and Layer 3 networks.
NVIDIA
PeerDirect™
PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary
internal data copies between components on the PCIe bus (for example, from GPU to CPU), and
therefore significantly reduces application run time. ConnectX-5 advanced acceleration
technology enables higher cluster efficiency and scalability to tens of thousands of nodes.
This section describes hardware features and capabilities. Please refer to the relevant
driver and/or firmware release notes for feature availability.


10
Feature Description
CPU Offload Adapter functionality enabling reduced CPU overhead allowing more available CPU for
computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of
Service
(QoS)
Support for port-based Quality of Service enabling various application requirements for latency
and SLA.
Hardware-
based I/
OVirtualiza
tion
ConnectX-5 provides dedicated adapter resources and guaranteed isolation and protection for
virtual machines within the server.
Storage
Acceleratio
n
A consolidated compute and storage network achieves significant cost-performance advantages
over multi-fabric networks. Standard block and file access protocols can leverage RDMA for
high-performance storage access.
• NVMe over Fabric offloads for target machine
SR-IOV ConnectX-5 SR-IOV technology provides dedicated adapter resources and guaranteed isolation
and protection for virtual machines (VM) within the server.
NC-SI over
RMII The adapter supports a slave Network Controller Sideband Interface (NC-SI) that can be
connected to a BMC.
High-
Performanc
eAccelerati
ons
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing
Host
Managemen
t
Technology
NVIDIA’s host management technology for standard and multi-host platforms optimizes board
management and power, performance and firmware update management via NC-SI, MCTP over
SMBus and MCTP over PCIe, as well as PLDM for Monitor and Control DSP0248 and PLDM for
Firmware Update DSP0267.
Multi-Host
Technology NVIDIA Multi-Host™ technology, when enabled, allows multiple hosts to be connected into a
single adapter by separating the PCIe interface into multiple and independent interfaces. By
Using NVIDIA Multi Host™, ConnectX-5 lowers the total cost of ownership (TCO) in the data
center by reducing CAPEX (cables, NICs, and switch port expenses), and by reducing OPEX by
cutting down on switch port management and overall power usage. With NVIDIA Multi-Host™
technology powered by a shared buffer architecture, connection-tracking offloads, and RoCE
enhancements, ConnectX-5 offers an extremely flexible solution for today’s demanding data
center and cloud application.
NVIDIA
Socket
Direct™
NVIDIA Socket Direct technology brings improved performance to multi-socket servers by
enabling direct access from each CPU in a multi-socket server to the network through its
dedicated PCIe interface. With this type of configuration, each CPU connects directly to the
network; this enables the interconnect to bypass a QPI (UPI) and the other CPU, optimizing
performance and improving latency. CPU utilization improves as each CPU handles only its own
traffic, and not the traffic from the other CPU. NVIDIA’s OCP 3.0 cards include native support
for socket direct technology for multi-socket servers and can support up to 4 CPU sockets.
Wake-on-
LAN (WoL)
Supported

11
•
•
•
•
•
•
•
•
•
Feature Description
Reset-on-
Lan (RoL)
Supported
Operating Systems/Distributions
RHEL/CentOS
Windows
FreeBSD
VMware
OpenFabrics Enterprise Distribution (OFED)
OpenFabrics Windows Distribution (WinOF-2)
Connectivity
Interoperable with 1/10/25/40/50/100Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support

12
Interfaces
Ethernet SFP28 and QSFP28 Interfaces
The network ports of the ConnectX®-5 adapter card are compliant with the IEEE 802.3 Ethernet
standards listed inFeatures and Benefits. Ethernet traffic is transmitted through the SFP28/QSFP28
connectors on the adaptercard.
Networking Port LEDS Interface
There are two I/O LEDs, LED0 and LED1, per port to indicate speed and link status. LED0 is a bicolor
(Yellow and Green) LED and LED1 is a single color (Green) LED.

Link Indications
LED and State Description
1Hz blinking Yellow Beacon command for locating the adapter card.
4Hz blinking Yellow
Indicates an error with the link. The error can be one of the
following:
Error Type Description LED Behavior
I2C
I2C access to the
networking ports fails
Blinks until the error
is fixed
Over-current
Over-current condition
of the networking
ports
Blinks until the error
is fixed
The adapter card includes special circuits to protect from ESD shocks to the card/server
when plugging copper cables.


13
•
•
•
•
•
LED and State Description
LED0 -
Link Speed
A constant Green indicates a link withthe maximum
networking speed.
A constant Yellow indicates a link withless than the
maximum networking speed
If LED0 is off, then the link has not been established.
LED1 -
Activity
A blinking Green indicates a valid link with data
transfer.
If LED1 is off, then there is no activity
PCI Express Interface
The table below describes the supported PCIe interface in ConnectX-5 and ConnectX-5 Ex adapter
cards.
IC Model Supported PCIe Interface Features
ConnectX-5 PCIe Gen 3.0 (1.1 and 2.0 compatible)
through x16 edge connectors Link Rates: 2.5. 5.0 or 8.0GT/s.
Auto Negotiation to: x16, x8, x4, x2
or x1.
Support for MSI/MSI-X mechanisms.
ConnectX-5 Ex PCIe Gen 3.0/4.0 (1.1 and 2.0 compatible)
through x16 edge connectors Link Rates: 2.5. 5.0, 8.0 or 16.0GT/s.
Auto Negotiation to: x16, x8, x4, x2
or x1.
Support for MSI/MSI-X mechanisms.
FRU EEPROM
FRU EEPROM allows the baseboard to identify different types of Mezzanine cards. MEZZ FRU EEPROM
is accessible through MEZZ_SMCLK and MEZZ_SMDATA (Connector A18 and A19). MEZZ FRU EEPROM
address is 0xA2 and its capacity is 4Kb.
Heat Sink Interface
A heatsink is attached to the ConnectX-5 IC in order to dissipate the heat from the ConnectX- 5 IC. It
is attached either by using four spring-loaded push pins that insert into four mounting holes.
ConnectX-5 IC has a thermal shutdown safety mechanism which automatically shuts down the
ConnectX-5 card in cases of high-temperature event, improper thermal coupling or heatsink
removal.

14
SMBus Interface
ConnectX-6 technology maintains support for manageability through a BMC. ConnectX-6 OCP 3.0
adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a
standard NVIDIA OCP 3.0 adapter. For configuring the adapter for the specific manageability solution
in use by the server, please contact NVIDIA Support.
Voltage Regulators
The voltage regulator power is derived from the OCP 3.0 edge connector 12V and 3.3V supply pins.
These voltage supply pins feed onboard regulators that provide the necessary power to the various
components on the card.
CPLD Interface
The adapter card incorporates a CPLD device that controls the networking port LEDs and the scan
chain. It draws its power supply from 3.3V_EDGE.
Thermal Sensors
Unable to render include or excerpt-include. Could not retrieve page.

15
Hardware Installation
Installation and initialization of ConnectX-5 adapter cards for OCP Spec 3.0 require attention to the
mechanical attributes, power specification, and precautions for electronic equipment.
Safety Warnings
Note that not all warnings are relevant to all models.
General Installation Instructions
Read all installation instructions before connecting the equipment to the power source.
Jewelry Removal Warning
Before you install or remove equipment that is connected to power lines, remove jewelry such as
bracelets, necklaces, rings, watches, and so on. Metal objects heat up when connected to power
and ground and can meltdown, causing serious burns and/or welding the metal object to the
terminals.
Over-temperature
This equipment should not be operated in an area with an ambient temperature exceeding the
maximum recommended: 55°C (131°F).An airflow of 200LFM at this maximum ambient
temperature is required for HCA cards and NICs.To guarantee proper airflow, allow at least 8cm (3
inches) of clearance around the ventilation openings.
During Lightning - Electrical Hazard
During periods of lightning activity, do not work on the equipment or connect or disconnect cables.
Copper Cable Connecting/Disconnecting
Some copper cables are heavy and not flexible, as such, they should be carefully attached to or
detached from the connectors. Refer to the cable manufacturer for special warnings and
instructions.
Equipment Installation
This equipment should be installed, replaced, or serviced only by trained and qualified personnel.
Equipment Disposal
The disposal of this equipment should be in accordance to all national laws and regulations.
Local and National Electrical Codes
This equipment should be installed in compliance with local and national electrical codes.
Safety warnings are provided here in the English language. For safety warnings in other
languages, refer to the Adapter Installation Safety Instructions document available on
nvidia.com.


16
•
•
Hazardous Radiation Exposure
Caution – Use of controls or adjustment or performance of procedures other than
those specified herein may result in hazardous radiation exposure.For products with
optical ports.
CLASS 1 LASER PRODUCT and reference to the most recent laser standards:
IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20
Installation Procedure Overview
The installation procedure of ConnectX-5 adapter cards for OCP Spec 3.0 involves the following
steps:
Step Procedure Direct Link
1Check the system’s hardware and software requirements. Refer to System Requirements
2 Pay attention to the airflow consideration within the host
system
Refer to Airflow Requirements
3 Follow the safety precautions Refer to Safety Precautions
4 Follow the pre-installation checklist Refer to Pre-Installation Checklist
5 (Optional) Replace the assembled OCP 3.0 bracket with
the desired form factor bracket
Refer toOCP 3.0 Bracket Replacement
Instructions
6 Install ConnectX-5 adapter card in the system
according to the retention mechanism on the card
Refer to Thumbscrew (Pull-tab) Cards
Refer to Internal Lock Cards
7 Connect cables or modules to the card Refer to Cables and Modules
8 Identify ConnectX-5 adapter card in the system Refer to Identify the Card in Your
System

17
•
System Requirements
Hardware Requirements
A system with a PCI Express x16 slot for OCP spec 3.0 is required for installing the card.
Airflow Requirements
ConnectX-5 adapter cards are offered with two airflow patterns: from the heatsink to the network
ports, and vice versa, as shown below.
Please refer to the "Specifications" chapter for airflow numbers for each specific card model.
Heatsink-to-port AirflowDirection Port-to-heatsink Airflow Direction
Software Requirements
See Operating Systems/Distributions section under the Introduction section.
Unless otherwise specified,NVIDIA products are designed to work in an environmentally
controlled data center with low levels of gaseous and dust (particulate) contamination.
The operating environment should meet severity level G1 as per ISA 71.04 for gaseous
contamination and ISO 14644-1 class 8 for cleanliness level.

For proper operation and performance, please make sure to use a PCIe slot with a
corresponding bus width and that can supply sufficient power to your card. Refer to the
Specifications section of the manual for more power requirements.

All cards in the system should be planned with the same airflow direction.


18
•
1.
2.
3.
4.
1.
2.
Software Stacks -NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for
Windows, and VMware. See the Driver Installation section.
Safety Precautions
Remove any metallic objects from your hands and wrists.
Make sure to use only insulated tools.
Verify that the system is powered off and is unplugged.
It is strongly recommended to use an ESD strap or other antistatic devices.
Pre-Installation Checklist
Unpack the ConnectX-5 adapter card.
Unpack and remove the ConnectX-5 card. Check the parts for visible damage that may have
occurred during shipping. Please note that the cards must be placed on an antistatic surface.
Shut down your system if active:
Turn off the power to the system, and disconnect the power cord. Refer to the system
documentation for instructions. Before you install the ConnectX-5 card, make sure that the
system is disconnected from power.
OCP 3.0 Bracket Replacement Instructions
The OCP 3.0 adapter card is shipped assembled either with a thumbscrew (pull-tab) bracket, an
internal-lock bracket, or an ejector-latch bracket. If this form factor is suitable for your
requirements, you can skip the remainder of this section and move to Installation Instructions. If
you need to replace the assembled OCP 3.0 bracket with a different form-factor bracket, please
follow the instructions in this section.
•
•
•
•
The adapter is being installed in a system that operates with voltages that can be lethal.
Before opening the case of the system, observe the following precautions to avoid injury
and prevent damage to system components.
Remove any metallic objects from your hands and wrists.
Make sure to use only insulated tools.
Verify that the system is powered off and is unplugged.
It is strongly recommended to use an ESD strap or other antistatic devices.

Please note that if the card is removed hastily from the antistatic bag, the plastic
ziplock may harm the EMI fingers on the networking connector. Carefully remove the
card from the antistatic bag to avoid damaging the EMI fingers.

Due to risk of damaging the EMI gasket, it is not recommended to replace the bracket more
than three times.


19
•
•
•
1.
2.
3.
1.
To replace the bracket you will need the following parts:
The new bracket of the desired form factor.
The screws supplied with the new bracket kit.
The required torx tool type as specified in the instructions.
Removing the Existing Bracket
Using the torx tool type listed in the below table, remove the screws according to the
instructions per OCP 3.0 bracket type.
Internal Lock Bracket Pull-tab (Thumbscrew)
Bracket
Ejector-Latch Bracket
Gently separate the bracket from the OCP 3.0 card.
Internal Lock Bracket Pull-tab Bracket Ejector-Latch Bracket
Save the two screws.
Installing the New Bracket
Assemble the new bracket onto the card.
Be careful not to put stress on the LEDs on the adapter card.


20
2.
3.
Internal Lock Bracket Pull-tab Bracket Ejector-Latch Bracket
Ensure that the insulator's front edge is beneath the bracket, as shown in the below figure.
Screw on the OCP 3.0 bracket with the supplied screws that came with the new bracket kit.
Use the specified torx tool type and apply the specified torque on the screws per bracket
form factor.
Internal Lock Bracket Pull-tab Bracket Ejector-Latch Bracket
Note that one screw is flat-head 90° screw and
the other is socket-head screw. Screw on the
different types of screws as shown in the
below illustration.
Do not force the bracket onto the adapter card.
