Nvidia Mellanox ConnectX-6 Dx Series User manual

Exported on Oct/31/2023 10:25 AM
NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards for OCP
Spec 3.0 User Manual

2
Table of Contents
Introduction............................................................................................................................................ 9
Products Overview ............................................................................................................................................... 9
Features and Benefits ..........................................................................................................................................10
Operating Systems/Distributions .............................................................................................................................13
Connectivity .....................................................................................................................................................13
Interfaces ............................................................................................................................................. 14
InfiniBand Interface ............................................................................................................................................14
Ethernet QSFP56 Interfaces ...................................................................................................................................14
PCI Express Interface...........................................................................................................................................14
LED Interface ....................................................................................................................................................14
FRU EEPROM .....................................................................................................................................................15
Heatsink Interface ..............................................................................................................................................15
SMBus Interface .................................................................................................................................................16
Voltage Regulators ..............................................................................................................................................16
CPLD Interface ..................................................................................................................................................16
Hardware Installation ............................................................................................................................... 17
Safety Warnings .................................................................................................................................................17
Installation Procedure Overview..............................................................................................................................18
System Requirements ..........................................................................................................................................19

3
Hardware Requirements ................................................................................................................................... 19
Airflow Requirements ...................................................................................................................................... 19
Software Requirements .................................................................................................................................... 20
Safety Precautions ..............................................................................................................................................20
Pre-Installation Checklist ......................................................................................................................................21
OCP 3.0 Bracket Replacement Instructions .................................................................................................................21
OCP 3.0 Adapter Card Installation Instructions.............................................................................................................25
Cables and Modules.........................................................................................................................................27
Identifying the Card in Your System...................................................................................................................... 28
Adapter Cards Extraction Instructions.......................................................................................................................30
Safety Precautions .......................................................................................................................................... 30
Instructions .................................................................................................................................................. 30
Driver Installation.................................................................................................................................... 32
Windows Driver Installation ...................................................................................................................................32
Software Requirements .................................................................................................................................... 32
Downloading WinOF-2 Driver.............................................................................................................................. 33
Installing WinOF-2 Driver .................................................................................................................................. 33
Attended Installation................................................................................................................................... 34
Unattended Installation................................................................................................................................ 43
Firmware Upgrade .......................................................................................................................................... 45
Linux Driver Installation .......................................................................................................................................45

4
Prerequisites................................................................................................................................................. 45
Downloading MLNX_OFED.................................................................................................................................. 46
Installing MLNX_OFED ...................................................................................................................................... 46
Installation Script ....................................................................................................................................... 46
Installation Procedure.................................................................................................................................. 49
Installation Results ..................................................................................................................................... 52
Installation Logging..................................................................................................................................... 52
Driver Load Upon System Boot............................................................................................................................ 53
mlnxofedinstall Return Codes......................................................................................................................... 53
Installation Logging..................................................................................................................................... 54
Uninstalling MLNX_OFED ............................................................................................................................... 54
Additional Installation Procedures ....................................................................................................................... 55
Installing MLNX_OFED Using YUM..................................................................................................................... 55
Installing MLNX_OFED Using apt-get ................................................................................................................. 61
Performance Tuning ........................................................................................................................................ 65
VMware Driver Installation ....................................................................................................................................65
Hardware and Software Requirements ..................................................................................................................65
Installing NATIVE ESXi Driver for VMware vSphere ..................................................................................................... 66
Removing Earlier NVIDIA Drivers .......................................................................................................................... 66
Firmware Programming .................................................................................................................................... 67
Updating Adapter Firmware ........................................................................................................................ 68

5
Troubleshooting ...................................................................................................................................... 70
General Troubleshooting.......................................................................................................................................70
Linux Troubleshooting ..........................................................................................................................................71
Windows Troubleshooting......................................................................................................................................72
Specifications......................................................................................................................................... 73
MCX653435A-HDAI/MCX653435M-HDAI Specifications .....................................................................................................73
MCX653436A-HDAI/MCX653436A-HDAB Specifications ....................................................................................................74
MCX653435A-EDAI Specifications .............................................................................................................................76
MCX653435A-HDAE Specifications ............................................................................................................................78
Board Mechanical Drawing and Dimensions.................................................................................................................80
Brackets Mechanical Drawings and Dimensions ............................................................................................................81
Cards with Ejector Latch Bracket ........................................................................................................................ 81
Cards with Internal Lock Bracket......................................................................................................................... 81
Dual-port Cards with Internal Lock Bracket ............................................................................................................ 82
Monitoring............................................................................................................................................. 83
Thermal Sensors.................................................................................................................................................83
Finding the GUID/MAC and Serial Number on the Adapter Card.............................................................................. 84
Board Label Example ...........................................................................................................................................84
Document Revision History ......................................................................................................................... 86



6
About This Manual
This User Manual describes NVIDIA® ConnectX®-6 InfiniBand/Ethernet adapter cards for Open Compute Project (OCP), Spec 3.0. It provides details as to
the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 InfiniBand/Ethernet adapter cards for OCP Spec 3.0.
NVIDIA SKU Legacy OPN Marketing Description Lifecycle
900-9X657-0018-MI0 MCX653435M-HDAI ConnectX®-6 InfiniBand/Ethernet adapter card, 200Gb/s (HDR IB and
200GbE) for OCP 3.0, with host management, Single-port QSFP56,
PCIe4.0 x16, Internal Lock
Mass Production
900-9X657-0058-SI2 MCX653436A-HDAI ConnectX®-6 InfiniBand/Ethernet adapter card, 200Gb/s (HDR IB and
200GbE) for OCP 3.0, with host management, Dual-port QSFP56,
PCIe4.0 x16, Internal Lock
Mass Production
900-9X657-0058-SB0 MCX653436A-HDAB ConnectX®-6 InfiniBand/Ethernet adapter card,200Gb/s (HDR IB and
200GbE) for OCP 3.0, with host management, Dual-port
QSFP56,PCIe4.0 x16, Thumbscrew Bracket
Mass Production
EOL'ed (End of Life) Ordering Part Number
NVIDIA SKU Legacy OPN Marketing Description
900-9X657-0018-SI0 MCX653435A-HDAI ConnectX®-6 InfiniBand/Ethernet adapter card, 200Gb/s (HDR IB and 200GbE) for OCP 3.0,
with host management, Single-port QSFP56, PCIe4.0 x16, Internal Lock
900-9X657-0016-SI0 MCX653435A-EDAI ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s (HDR100, EDR IB and 100GbE) for
OCP 3.0, with host management, Single-port QSFP56, PCIe 3.0/4.0 x16, Internal Lock
900-9X657-0018-SE0 MCX653435A-HDAE ConnectX®-6 InfiniBand/Ethernet adapter card, 200Gb/s (HDR IB and 200GbE) for OCP 3.0,
with host management, Single-port QSFP56, PCIe4.0 x16, Ejector Latch
Intended Audience
This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with InfiniBand/Ethernet network and architecture
specifications.

7
•
•
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchased NVIDIA Global Support Services, please see your contract for details regarding Technical Support.
Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller.
Related Documentation
MLNX_OFED for Linux User Manual and Release Notes User Manual describing OFED features, performance, band diagnostic, tools content, and configuration.
SeeMLNX_OFED for Linux Documentation.
WinOF-2 for WindowsUser Manualand Release Notes User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools content, and configuration.
SeeWinOF-2 for Windows Documentation.
NVIDIA VMware for Ethernet User Manual User Manual and release notes describing the various components of the NVIDIA ConnectX® NATIVE ESXi stack.
SeeVMware® ESXi Drivers Documentation.
NVIDIA Firmware Utility (mlxup) User Manual and Release Notes NVIDIA firmware update and query utility used to update the firmware. Refer toFirmware Utility (mlxup)
Documentation.
NVIDIA Firmware Tools (MFT) User Manual User Manual describing the set of MFT firmware management tools for a single node. SeeMFT User Manual.
InfiniBand Architecture Specification Release 1.2.1, Vol 2 - Release 1.3 InfiniBand Specifications
IEEE Std 802.3 Specification IEEE Ethernet Specifications
PCI Express Specifications Industry Standard PCI Express Base and Card Electromechanical Specifications. Refer toPCI-SIG Specifications.
LinkX Interconnect Solutions LinkX InfiniBand cables and transceivers are designed to maximize the performance of High-Performance
Computing networks, requiring high-bandwidth, low-latency connections between compute nodes and switch
nodes.NVIDIA offers one of the industry’s broadest portfolio ofQDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/HDR100
(100Gb/s), HDR (200Gb/s)and NDR (400Gb/s) cables, includingDirect Attach Copper cables (DACs), copper splitter
cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths from 0.5m to 10km.In addition to
meeting IBTA standards, NVIDIAtests every product in an end-to-end environment ensuring a Bit Error Rate of less
than 1E-15.Read more atLinkX Cables and Transceivers.
Open Compute Project 3.0 Specifications https://www.opencompute.org/

8
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega Bytes. The use of Mb or Mbits (small b) indicates size in
mega bits. In this document PCIe is used to mean PCI Express.

9
Introduction
This is the User Guide for Ethernet adapter cards based on the ConnectX®-6 InfiniBand/Ethernet integrated circuit device for OCP Spec 3.0. These
adapters connectivity provide the highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in
Enterprise Data Centers and High-Performance Computing environments.
The following provides the ordering part number, port speed, number of ports, and PCI Express speed.
Important Note:
ConnectX-6 OCP 3.0 cards were tested for Shock & Vibe in accordance with NVIDIA specifications and setups as defined in document XXX, as the OCP
spec3.0 available at that time did not contain any S&V definitions. A newer version of the OCP spec 3.0 has defined S&V specifications and NVIDIA is in the
midst of retesting these cards to comply with OCP spec 3.0.
Products Overview
Model Cards for OCP Spec 3.0
Part Number MCX653435A-HDAI MCX653436A-HDAI MCX653435A-HDAE(a) MCX653436A-HDAB MCX653435A-EDAI
Host Management √ √ √ √ √
Network Connector Type Single-port QSFP56 Dual-port QSFP56 Single-port QSFP56 Dual-port QSFP56 Single-port QSFP56
Data Rate
InfiniBand SDR/DDR/QDR/FDR/EDR/HDR100/HDR SDR/DDR/QDR/FDR/EDR/HDR100
Ethernet 10/25/40/50/100/200 Gb/s 10/25/40/50/100 Gb/s
PCI Express Connectors PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Retention Mechanism Internal Lock Ejector Latch Pull Tab(Thumbscrew) Internal Lock

10
Model Cards for OCP Spec 3.0
RoHS RoHS Compliant
IC Part Number MT28908A0-XCCF-HVM MT28908A0-XCCF-HVM MT28908A0-XCCF-HVM MT28908A0-XCCF-HVM MT28908A0-XCCF-HVM
Device ID 4123 for Physical Function (PF)
4124 for Virtual Function (VF)
a.MCX653435A-HDAE - The Gen 4.0 card has been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be tested when servers with Gen 4.0
support become available.
For more detailed information seeSpecifications.
Features and Benefits
Feature Description
PCI Express (PCIe) Uses PCIe Gen 3.0 (8GT/s) or Gen 4.0 (16GT/s) through x16 edge connector.
200Gb/s InfiniBand/EthernetAdapter ConnectX-6 offers the highest throughput InfiniBand/Ethernet adapter, supporting HDR 200Gb/s InfiniBand and 200Gb/s Ethernet and
enabling any standard networking, clustering, or storage to operate seamlessly over any converged network leveraging a consolidated
software stack.
InfiniBand Architecture Specification v1.3
compliant
ConnectX-6 delivers low latency, high bandwidth, and computing efficiency for performance-driven server and storage clustering
applications. ConnectX-6 is InfiniBand Architecture Specification v1.3 compliant.
InfiniBand HDR A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of 53.125Gb/s with a 64b/66b encoding, resulting in an effective
bandwidth of 200Gb/s.
This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature
availability.


11
•
•
Feature Description
InfiniBand HDR100 A standard InfiniBand data rate, where each lane of a 2X port runs a bit rate of 53.125Gb/s with a 64b/66b encoding, resulting in an effective
bandwidth of 100Gb/s.
Up to 200 Gigabit Ethernet NVIDIA adapters comply with the following IEEE 802.3 standards:
• 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
InfiniBand EDR A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of 25.78125Gb/s with a 64b/66b encoding, resulting in an
effective bandwidth of 100Gb/s.
Memory Components EEPROM - The EEPROM capacity is 32Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus.(Note: Address 0x58 is
reserved)
SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G device by Macronix)
Overlay Networks In order to better scale their networks, data center operators often create overlay networks that carry traffic from individual virtual
machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP
packet from the hardware offloading engines, placing higher loads on the host CPU. ConnectX-6 effectively addresses this by providing
advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-capsulate the overlay protocol.
RDMA and RDMA overConverged Ethernet
(RoCE)
ConnectX-6, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency
and high-performance over Band and Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as ConnectX-6 advanced
congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and Layer 3 networks.

12
Feature Description
NVIDIA PeerDirect™ PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on
the PCIe bus (for example, from GPU to CPU), and therefore significantly reduces application run time. ConnectX-6 advanced acceleration
technology enables higher cluster efficiency and scalability to tens of thousands of nodes.
CPU Offload Adapter functionality enables reduced CPU overhead leaving more CPU resources available for computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of Service (QoS) Support for port-based Quality of Service enabling various application requirements for latency and SLA.
Hardware-based I/OVirtualization ConnectX-6 provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server.
Storage Acceleration A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block
and file access protocols can leverage RDMA for high-performance storage access.
• NVMe over Fabric offloads for the target machine
SR-IOV ConnectX-6 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within
the server.
NC-SI The adapter supports a Network Controller Sideband Interface (NC-SI), MCTP over SMBus and MCTP over PCIe - Baseboard Management
Controller interface.
High-PerformanceAccelerations • Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing
Wake-on-LAN (WoL) The adapter supported Wake-on-LAN (WoL), a computer networkingstandard that allows an adapter to be turned on orawakenedby a
network message.
In STBY mode, only port0 is available.
Reset-on-LAN (RoL) Supported

13
•
•
•
•
•
•
•
•
•
Operating Systems/Distributions
RHEL/CentOS
Windows
FreeBSD
VMware
OpenFabrics Enterprise Distribution (OFED)
OpenFabrics Windows Distribution (WinOF-2)
Connectivity
Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support

14
Interfaces
InfiniBand Interface
The network ports of the ConnectX®-6 adapter cards are compliant with the InfiniBand Architecture Specification, Release 1.3. InfiniBand traffic is
transmitted through the cards' QSFP56 connectors.
Ethernet QSFP56 Interfaces
The network ports of the ConnectX®-6 adapter card are compliant with the IEEE 802.3 Ethernet standards listed inFeatures and Benefits. Ethernet traffic
is transmitted through the QSFP56 connectors on the adaptercard.
PCI Express Interface
The table below describes the supported PCIe interface in ConnectX-6 adapter cards.
Supported PCIe Interface Features
PCIe Gen 3.0/4.0 (1.1 and 3.0 compatible) through x16 edge connectors Link Rates: 2.5. 5.0, 8.0GT/s or 16GT/s.
Auto Negotiation to: x16, x8, x4, x2 or x1.
Support for MSI/MSI-X mechanisms.
LED Interface
There are two I/O LEDs, LED0 and LED1, per port to indicate speed and link status. LED0 is bicolor (yellow and green) LED and LED1 is a single color
(green) LED.
The adapter card includes special circuits to protect from ESD shocks to the card/server when plugging copper cables.


15
•
•
•
•
•
Link Indications
LED and State Description
1Hz blinking Yellow Beacon command for locating the adapter card
4Hz blinking Yellow Indicates an error with the link. The error can be one of the following:
Error Type Description LED Behavior
I2C I2C access to the networking ports
fails
Blinks until the error is fixed
Over-current Over-current condition of the
networking ports
Blinks until the error is fixed
LED0 -
Link Speed
A constant Green indicates a link withthe maximum networking speed.
A constant Yellow indicates a link withless than the maximum networking speed
If LED0 is off, then the link has not been established.
LED1 -
Activity
A blinking Green indicates a valid link with data transfer.
If LED1 is off, then there is no activity
FRU EEPROM
FRU EEPROM allows the baseboard to identify different types of Mezzanine cards. FRU EEPROM is accessible through SMCLK and SMDATA. FRU EEPROM
address is defined according to SLOT_ID0 and SLOT_ID1 and its capacity is 4Kb.
Heatsink Interface
A heatsink is attached to the ConnectX-6 IC in order to dissipate the heat from the ConnectX-6 IC. It is attached either by using four spring-loaded push
pins that insert into four mounting holes.

16
ConnectX-6 IC has a thermal shutdown safety mechanism that automatically shuts down the ConnectX-6 card in case of a high-temperature event,
improper thermal coupling or heatsink removal.
SMBus Interface
ConnectX-6 technology maintains support for manageability through a BMC. ConnectX-6 OCP 3.0 adapter can be connected to a BMC using MCTP over
SMBus or MCTP over PCIe protocols as if it is a standard NVIDIA OCP 3.0 adapter. For configuring the adapter for the specific manageability solution in use
by the server, please contact NVIDIA Support.
Voltage Regulators
The voltage regulator power is derived from the OCP 3.0 edge connector 12V and 3.3V supply pins. These voltage supply pins feed onboard regulators that
provide the necessary power to the various components on the card.
CPLDInterface
The adapter card incorporates a CPLD device that controls the networking port LEDs and the scan chain. It draws its power supply from 3.3V_EDGE.

17
Hardware Installation
Installation and initialization of ConnectX-6 adapter cards for OCP Spec 3.0 require attention to the mechanical attributes, power specification, and
precautions for electronic equipment.
Safety Warnings
Note that not all warnings are relevant to all models.
General Installation Instructions
Read all installation instructions before connecting the equipment to the power source.
Jewelry Removal Warning
Before you install or remove equipment that is connected to power lines, remove jewelry such as bracelets, necklaces, rings, watches, and so on.
Metal objects heat up when connected to power and ground and can meltdown, causing serious burns and/or welding the metal object to the
terminals.
Over-temperature
This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: 55°C (131°F).An airflow
of 200LFM at this maximum ambient temperature is required for HCA cards and NICs.To guarantee proper airflow, allow at least 8cm (3 inches) of
clearance around the ventilation openings.
During Lightning - Electrical Hazard
During periods of lightning activity, do not work on the equipment or connect or disconnect cables.
Copper Cable Connecting/Disconnecting
Some copper cables are heavy and not flexible, as such, they should be carefully attached to or detached from the connectors. Refer to the cable
manufacturer for special warnings and instructions.
Safety warnings are provided here in the English language. For safety warnings in other languages, refer to the Adapter Installation Safety
Instructions.


18
•
•
Equipment Installation
This equipment should be installed, replaced, or serviced only by trained and qualified personnel.
Equipment Disposal
The disposal of this equipment should be in accordance to all national laws and regulations.
Local and National Electrical Codes
This equipment should be installed in compliance with local and national electrical codes.
Hazardous Radiation Exposure
Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous
radiation exposure.For products with optical ports.
CLASS 1 LASER PRODUCT and reference to the most recent laser standards:
IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20
Installation Procedure Overview
The installation procedure of ConnectX-6 adapter cards for OCP Spec 3.0 involves the following steps:
Step Procedure Direct Link
1Check the system’s hardware and software requirements. Refer to System Requirements
2 Pay attention to the airflow consideration within the host system Refer to Airflow Requirements
3 Follow the safety precautions Refer to Safety Precautions
4 Follow the pre-installation checklist Refer to Pre-Installation Checklist

19
Step Procedure Direct Link
5 (Optional) Replace the assembled OCP 3.0 bracket with the desired form factor
bracket
Refer toOCP 3.0 Bracket Replacement Instructions
6 Install ConnectX-6 adapter card for OCP spec 3.0 in the system Refer to Installation Instructions
7 Connect cables or modules to the card Refer to Cables and Modules
8 Identify ConnectX-6 adapter card in the system Refer to Identify the Card in Your System
System Requirements
Hardware Requirements
A system with a PCI Express x16 slot for OCP spec 3.0 is required for installing the card.
Airflow Requirements
ConnectX-6 adapter cards are offered with two airflow patterns: from the heatsink to the network ports, and vice versa, as shown below.
Unless otherwise specified, NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous and dust
(particulate) contamination.
The operating environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for cleanliness level.

For proper operation and performance, please make sure to use a PCIe slot with a corresponding bus width and that can supply sufficient power to
your card. Refer to the Specifications section of the manual for more power requirements.


20
•
•
Please refer to the "Specifications" chapter for airflow numbers for each specific card model.
Hot Aisle Cooling
Heatsink-to-port AirflowDirection
Cold Aisle Cooling
Port-to-heatsink Airflow Direction
Software Requirements
See Operating Systems/Distributions section under the Introduction section.
Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for Windows, and VMware. See the Driver Installation section.
Safety Precautions
All cards in the system should be planned with the same airflow direction.

The adapter is being installed in a system that operates with voltages that can be lethal. Before opening the case of the system, observe the
following precautions to avoid injury and prevent damage to system components.
