Nvidia Mellanox ConnectX-4 Lx User manual

Exported on Dec/05/2023 11:54 AM







NVIDIA ConnectX-4 Lx Ethernet Adapter
Cards for OCP Spec 2.0 User Manual
ConnectX-4 Lx Ethernet Adapter Cards for OCP Spec 2.0


2
Table of Contents
Introduction..................................................................................... 8
Product Overview ....................................................................................8
Features and Benefits ...............................................................................9
Operating Systems/Distributions ................................................................ 11
Connectivity ........................................................................................ 12
Interfaces ......................................................................................13
Ethernet SFP28 and QSFP28 Interfaces ......................................................... 13
PCI Express Interface.............................................................................. 13
FRU EEPROM ........................................................................................ 13
LED Interface ....................................................................................... 13
Heatsink Interface ................................................................................. 14
SMBus Interface .................................................................................... 14
Voltage Regulators ................................................................................. 14
Hardware Installation ........................................................................15
Safety Warnings .................................................................................... 15
Installation Procedure Overview................................................................. 16
System Requirements ............................................................................. 16
Hardware Requirements ...................................................................... 16
Airflow Requirements ......................................................................... 16
Software Requirements ....................................................................... 17
Safety Precautions ................................................................................. 17
Unpacking the Package ........................................................................... 17
Installation Instructions........................................................................... 18
Cables and Modules................................................................................ 19
Identifying the Card in Your System............................................................. 20
Uninstalling the Card.............................................................................. 21
Safety Precautions ............................................................................. 17
Card Removal................................................................................... 21
Driver Installation.............................................................................24
Linux Driver Installation .......................................................................... 24
Prerequisites.................................................................................... 24
Downloading NVIDIA OFED .................................................................... 24

3
Installing MLNX_OFED ......................................................................... 25
Driver Load Upon System Boot............................................................... 30
Additional Installation Procedures .......................................................... 31
Performance Tuning ........................................................................... 37
Windows Driver Installation ...................................................................... 37
Software Requirements ....................................................................... 38
Downloading WinOF-2 Driver................................................................. 38
Installing WinOF-2 Driver ..................................................................... 38
Firmware Upgrade ............................................................................. 46
VMware Driver Installation ....................................................................... 46
Hardware and Software Requirements ..................................................... 46
Installing NATIVE ESXi Driver for VMware vSphere ........................................ 46
Removing Earlier NVIDIA Drivers ............................................................. 47
Firmware Programming ....................................................................... 47
Updating Adapter Firmware .................................................................48
Troubleshooting ...............................................................................49
General Troubleshooting.......................................................................... 49
Linux Troubleshooting ............................................................................. 50
Windows Troubleshooting......................................................................... 50
Specifications..................................................................................51
MCX4421A-XCQN Specifications.................................................................. 51
MCX4411A-ACAN/MCX4411A-ACUN Specifications ............................................ 52
MCX4421A-ACAN/MCX4421A-ACUN Specifications ............................................ 53
MCX4411A-ACQN Specifications.................................................................. 54
MCX4421A-ACQN Specifications.................................................................. 55
MCX4431A-GCAN/MCX4431A-GCUN Specifications............................................ 56
MCX4431M-GCAN Specifications ................................................................. 57
Board Mechanical Drawing and Dimensions.................................................... 58
Finding the MAC on the Adapter Card .....................................................59
MCX4421A-XCQN Board Label (Example) ....................................................... 59
MCX4411A-ACAN Board Label (Example) ....................................................... 59
MCX4421A-ACAN Board Label (Example) ....................................................... 60
MCX4411A-ACQN Board Label (Example) ....................................................... 60
MCX4421A-ACQN Board Label (Example) ....................................................... 60

4
MCX4411A-ACUN Board Label (Example) ....................................................... 61
MCX4421A-ACUN Board Label (Example) ....................................................... 61
MCX4431A-GCAN Board Label (Example) ....................................................... 62
MCX4431M-GCAN Board Label (Example)....................................................... 62
MCX4431A-GCUN Board Label (Example)....................................................... 63
Document Revision History ..................................................................64


5
About This Manual
This User Manual describesNVIDIA® ConnectX®-4 Lx Ethernet adapter cards for Open Compute
Project, spec 2.0. It provides details as to the interfaces of the board, specifications, required
software and firmware for operating the board, and relevant documentation.
EOL'd (End of Life) Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for ConnectX-4 Lx Ethernet adapter cards
for OCP Spec 2.0.
NVIDIA SKU Legacy OPN Marketing Description OCP Spec and Type
900-9X414-0052-SN0 MCX4421A-XCQN ConnectX®-4 Lx EN network interface
card for OCP with Host Management,
10GbE dual-port SFP28, PCIe3.0 x8, no
bracket
OCP Spec 2.0 Type 1
Stacking Height
900-9X414-0013-SN1 MCX4411A-ACAN ConnectX®-4 Lx EN network interface
card for OCP, without host management,
25GbE single-port SFP28, PCIe3.0 x8, no
bracket
900-9X414-0053-SN1 MCX4421A-ACAN ConnectX®-4 Lx EN network interface
card for OCP, without host management,
25GbE dual-port SFP28, PCIe3.0 x8, no
bracket
900-9X414-0013-SN3 MCX4411A-ACQN ConnectX®-4 Lx EN network interface
card for OCP with Host Management,
25GbE single-port SFP28, PCIe3.0 x8, no
bracket
900-9X414-0053-SN3 MCX4421A-ACQN ConnectX®-4 Lx EN network interface
card for OCP with Host Management,
25GbE dual-port SFP28, PCIe3.0 x8, no
bracket
900-9X414-0013-SN0 MCX4411A-ACUN ConnectX®-4 Lx EN network interface
card for OCP, without host management,
25GbE single-port SFP28, PCIe3.0 x8,
UEFI Enabled, no bracket
900-9X414-0053-SN0 MCX4421A-ACUN ConnectX®-4 Lx EN network interface
card for OCP, without host management,
25GbE dual-port SFP28, PCIe3.0 x8, UEFI
Enabled, no bracket
900-9X473-0015-SN1 MCX4431A-GCAN ConnectX®-4 Lx EN network interface
card for OCP, with Host Management,
50GbE single-port QSFP28, PCIe3.0 x8,
no bracket
900-9X473-0015-SN0 MCX4431A-GCUN ConnectX®-4 Lx EN network interface
card for OCP2.0, Type 1, with Host
Management, 50GbE single-port QSFP28,
PCIe3.0 x8, UEFI Enabled, no bracket
900-9X473-0015-MN0 MCX4431M-GCAN ConnectX®-4 Lx EN network interface
card for OCP with Multi-Host and Host
Management, 50GbE single-port QSFP28,
PCIe3.0 x8, no bracket
OCP Spec 2.0 Type 2
Stacking Height
Intended Audience

6
•
•
This manual is intended for the installer and user of these cards.The manual assumes basic
familiarity with Ethernet network and architecture specifications.
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough
the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchased NVIDIA Global Support Services, please see your contract for details
regarding Technical Support.
Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek
assistance through their reseller.
Related Documentation
NVIDIA MLNX_OFED for
Linux User Manual and
Release Notes
User Manual and Release Notes describing MLNx_OFED features, performance,
band diagnostic, tools content and configuration. SeeNVIDIA MLNX_OFED for
Linux Documentation.
WinOF-2 for
WindowsUser Manual
and Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools
content and configuration. SeeWinOF-2 for Windows Documentation.
NVIDIA VMware for
Ethernet User Manual
and Release Notes
User Manual describing the various components of the NVIDIA ConnectX® NATIVE
ESXi stack. SeeVMware® ESXi Documentation.
NVIDIA Firmware
Update
NVIDIA firmware update and query utility used to update the firmware.
SeeNVIDIA Firmware Utility (mlxup) Documentation.
NVIDIA Firmware Tools
(MFT) User Manual
User Manual describing the set of MFT firmware management tools for a single
node. SeeMFT User Manual.
IEEE Std 802.3
Specification
IEEE Ethernet Specifications
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications.
Open Compute Project
Specification 3.0
OCP Spec 3.0
LinkX Interconnect
Solutions
LinkX Ethernet cables and transceivers are designed to maximize the performance
of High-Performance Computing networks, requiring high-bandwidth, low-latency
connections between compute nodes and switch nodes.NVIDIA offers one of the
industry’s broadest portfolio of 40GbE, 56GbE, 100GbE, 200GbE and400GbE
cables, includingDirect Attach Copper cables (DACs), copper splitter cables,
Active Optical Cables (AOCs) and transceivers in a wide range of lengths from
0.5m to 10km. In addition to meeting Ethernetstandards, NVIDIA tests every
product in an end-to-end environment ensuring a Bit Error Rate of less than
1E-15.Read more atLinkX Cables and Transceivers.
Document Conventions

7
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.

8
Introduction
Product Overview
This is the User Guide for Ethernet adapter cards based on the ConnectX®-4 Lx integrated circuit
device. These adapters connectivity provide the highest performing low latency and most flexible
interconnect solution for PCI Express Gen 3.0 servers used in Enterprise Data Centers and High-
Performance Computing environments.
The following provides the ordering part number, port speed, number of ports, and PCI Express
speed. Each adapter is shipped with two bracket heights - short and tall (assembled).
ConnectX-4 Lx 10Gb/s Ethernet Cards for OCP Spec 2.0
ConnectX-4 Lx 10GbE for OCP Spec 2.0
Part Number MCX4421A-XCQN
OCP Spec 2.0 Stacking Height Complying with OCP Type1 stacking height
Network Connector Type Dual-port SFP28
Ethernet Data Rate 1/10 Gb/s
PCI Express Connectors PCIe Gen 3.0 SERDES @ 8.0GT/s
Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile
Host Management Supported
RoHS RoHS Compliant
IC Part Number MT27712A0-FDCF-XE
Device ID 4117 for Physical Function (PF)
4118 for Virtual Function (VF)
ConnectX-4 Lx 25Gb/s Ethernet Cards for OCP Spec 2.0
ConnectX-4 Lx Ethernet Adapter Cards for OCP Spec 2.0
Part Number MCX4411A-
ACQN
MCX4411A-
ACAN
MCX4411A-
ACUN
MCX4421A-
ACUN
MCX4421A-
ACQN
MCX4421A-
ACAN
Ethernet Data
Rate
1/10/25 Gb/s 1/10/25 Gb/s 1/10/25 Gb/s
OCP Spec 2.0
Stacking Height
Complying with OCP Type 1 stacking height
Network
Connector Type
Single-port SFP28 Dual-port SFP28
Ethernet Data
Rate
1/10/25 Gb/s
PCI Express
Connectors
PCIe Gen 3.0 SERDES @ 8.0GT/s
Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile
Host
Management
√---√-
UEFI - - √ √ - -

9
ConnectX-4 Lx Ethernet Adapter Cards for OCP Spec 2.0
RoHS RoHS Compliant
IC Part Number MT27712A0-FDCF-AE
Device ID 4117 for Physical Function (PF)
4118 for Virtual Function (VF)
ConnectX-4 Lx 40/50 Gb/s Ethernet Adapter Cards
Model ConnectX-4 Lx 40/50 Gb/s Ethernet Adapter Cards
Part Number MCX4131A-BCAT MCX4131A-GCAT MCX4431A-GCUN
Ethernet Data Rate 1/10/25/40 Gb/s 1/10/25/40/50 Gb/s 1/10/25/40/50 Gb/s
OCP Spec 2.0 Stacking
Height
Complying with OCP Type 1 stacking height
Network Connector Type Single-port SFP28
PCI Express Connectors PCIe Gen 3.0 x8 SerDes @ 8.0GT/s
Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile
RoHS RoHS Compliant
IC Part Number MT27711A0-FDCF-BE MT27711A0-FDCF-GE MT27711A0-FDCF-GE
Host Management - - √
UEFI - - √
Device ID 4117 for Physical Function (PF)
4118 for Virtual Function (VF)
For more detailed information see.Specifications v3.1.
Features and Benefits
Feature Description
PCI Express (PCIe) Uses PCIe Gen 3.0 (8GT/s) through an x8 edge connector. Gen 1.1
and 2.0 compatible.
This section describes hardware features and capabilities. Please refer to the relevant
driver and/or firmware release notes for feature availability.


10
•
•
Feature Description
Up to 50 Gigabit Ethernet NVIDIA adapters comply with the following IEEE 802.3 standards:
• 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet,
supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
Memory PCI Express - stores and accesses Ethernet fabric connection
information and packet data.
SPI - includes one 16MB SPI Flash device (W25Q128FVSIG
device by WINBOND-NUVOTON).
Overlay Networks In order to better scale their networks, data center operators often
create overlay networks that carry traffic from individual virtual
machines over logical tunnels in encapsulated formats such as NVGRE
and VXLAN. While this solves network scalability issues, it hides the
TCP packet from the hardware offloading engines, placing higher
loads on the host CPU. ConnectX-4 Lx effectively addresses this by
providing advanced NVGRE and VXLAN hardware offloading engines
that encapsulate and de-capsulate the overlay protocol.
RDMA and RDMA overConverged
Ethernet (RoCE)
ConnectX-4 Lx, utilizing IBTA RDMA (Remote Data Memory Access)
and RoCE (RDMA over Converged Ethernet) technology, delivers low-
latency and high-performance over Band and Ethernet networks.
Leveraging data center bridging (DCB) capabilities as well as
ConnectX-4 Lx advanced congestion control hardware mechanisms,
RoCE provides efficient low-latency RDMA services over Layer 2 and
Layer 3 networks.
NVIDIA PeerDirect™ PeerDirect™ communication provides high efficiency RDMA access by
eliminating unnecessary internal data copies between components
on the PCIe bus (for example, from GPU to CPU), and therefore
significantly reduces application run time. ConnectX-4 Lx advanced
acceleration technology enables higher cluster efficiency and
scalability to tens of thousands of nodes.
CPU Offload Adapter functionality enables reduced CPU overhead leaving more
CPU resources available for computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of Service (QoS) Support for port-based Quality of Service enabling various
application requirements for latency and SLA.
Hardware-based I/OVirtualization ConnectX-4 Lx provides dedicated adapter resources and guaranteed
isolation and protection for virtual machines within the server.

11
•
•
•
•
•
•
Feature Description
Storage Acceleration A consolidated compute and storage network achieves significant
cost-performance advantages over multi-fabric networks. Standard
block and file access protocols can leverage RDMA for high-
performance storage access.
• NVMe over Fabric offloads for target machine
SR-IOV ConnectX-4 Lx SR-IOV technology provides dedicated adapter
resources and guaranteed isolation and protection for virtual
machines (VM) within the server.
NC-SI The adapter supports a Network Controller Sideband Interface (NC-
SI), MCTP over SMBus and MCTP over PCIe - Baseboard Management
Controller interface.
High-PerformanceAccelerations • Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing
UEFI UEFI is a standard firmware interface designed to replace BIOS.
NVIDIA UEFI Network driver allows boot over network via PXE
(Preboot eXecution Environment). This network driver allows remote
boot over InfiniBand or Ethernet, or Boot over iSCSI (Bo-iSCSI) in UEFI
mode, and also supports the SecureBoot standard. The UEFI Network
driver allows IT managers the flexibility to deploy servers with a
single adapter card into InfiniBand or Ethernet networks while also
enabling booting from LAN or remote storage targets. In addition to
boot capabilities, NVIDIA UEFI Network driver provides firmware
management and diagnostic protocols compliant with the UEFI
specification.
For further information, refer to the NVIDIA PreBoot Drivers User
Manual.
Supported in MCX4111A-ACUT, MCX4121A-ACUT and MCX4431A-
GCUN.
Secure Firmware A “Secure firmware update” is the ability of a device to verify digital
signatures of new firmware binaries, in order to assure that only
officially approved versions can be installed from the host, the
network or a Board Management Controller (BMC). The firmware of
devices with “secure firmware update” functionality (secure FW),
restricts access to specific commands and registers that can be used
to modify the firmware binary image on the flash, as well as
commands that can jeopardize security in general.
For further information, refer to the MFT User Manual.
Supported in MCX4121A-ACST.
Operating Systems/Distributions
RHEL/CentOS
Windows
FreeBSD
VMware
OpenFabrics Enterprise Distribution (OFED)
OpenFabrics Windows Distribution (WinOF-2)

12
•
•
•
Connectivity
Interoperable with 1/10/25/40/50 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support

13
Interfaces
Ethernet SFP28 and QSFP28 Interfaces
The network ports of the ConnectX®-4 Lx adapter card for OCP Spec 2.0 are compliant with the IEEE
802.3 Ethernet standards listed in Features and Benefits. Ethernet traffic is transmitted through the
SFP28/QSFP28 connectors on the adaptercard.
PCI Express Interface
The ConnectX®-4 adapter card supports PCI Express Gen 3.0 (1.1 and 2.0 compatible)through an x8
or x16 edge connector. The device can be either a master initiating the PCI Express busoperations,
or asubordinateresponding to PCI bus operations.The following lists PCIe interface features:
Supported PCIe Interface Features
PCIe Gen 3.0 (1.1 and 2.0 compatible) through x8/x16
edge connectors
Link Rates: 2.5. 5.0 or 8.0GT/s.
Auto Negotiation to: x16, x8, x4, x2 or x1.
Support for MSI/MSI-X mechanisms.
FRU EEPROM
FRU EEPROM allows the baseboard to identify different types of Mezzanine cards. MEZZ FRU EEPROM
is accessible through MEZZ_SMCLK and MEZZ_SMDATA (Connector A18 and A19). MEZZ FRU EEPROM
address is 0xA2 and its capacity is 128Kb.
LED Interface
There are two, LED0 and LED1, and speed. LED0 is bi-color (yellow and green) LED and LED1 is a
single color (green) LED. I/O LEDs per port to indicate link status.
Link Indications
LED and State Description
1Hz blinking Yellow Beacon command for locating the adapter card
The adapter card includes special circuits to protect from ESD shocks to the card/server
when plugging copper cables.

Applies to MCX4431M-GCAN


14
•
•
•
•
•
LED and State Description
4Hz blinking Yellow Indicates an error with the link. The error can be one of the
following:
Error Type Description LED Behavior
I2C I2C access to the
networking ports fails
Blinks until the error
is fixed
Over-current Over-current condition
of the networking ports
Blinks until the error
is fixed
LED0 -
Link Speed
A constant Green indicates a link withthe maximum
networking speed.
A constant Yellow indicates a link withless than the
maximum networking speed
If LED0 is off, then the link has not been established.
LED1 -
Activity
A blinking Green indicates a valid link with data transfer.
If LED1 is off, then there is no activity
Heatsink Interface
A heatsink is attached to the ConnectX-4 Lx IC in order to dissipate the heat from the ConnectX-4 Lx
IC. It is attached either by using four spring-loaded push pins that insert into four mounting holes or
by screws.
ConnectX-4 Lx IC has a thermal shutdown safety mechanism that automatically shuts down the
ConnectX-4 Lx card in cases of a high-temperature event, improper thermal coupling, or heatsink
removal.
SMBus Interface
ConnectX-4 Lx technology maintains support for manageability through a BMC. ConnectX-4 Lx PCIe
stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as
if it is a standard NVIDIA PCIe stand-up adapter. For configuring the adapter for the specific
manageability solution in use by the server, please contact NVIDIA Support.
Voltage Regulators
The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These
voltage supply pins feed onboard regulators that provide the necessary power to the various
components on the card.

15
•
•
Hardware Installation
Installation and initialization of ConnectX-4 Lx adapter cards require attention to the mechanical
attributes, power specification, and precautions for electronic equipment.
Safety Warnings
Please observe all safety warnings to avoidinjury and prevent damage to system components. Note
that not all warnings are relevant to all models.
Note that not all warnings are relevant to all models.
General Installation Instructions
Read all installation instructions before connecting the equipment to the power source.
Jewelry Removal Warning
Before you install or remove equipment that is connected to power lines, remove jewelry such as
bracelets, necklaces, rings, watches, and so on. Metal objects heat up when connected to power and
ground and can meltdown, causing serious burns and/or welding the metal object to the terminals.
Over-temperature
This equipment should not be operated in an area with an ambient temperature exceeding the
maximum recommended: 55°C (131°F).An airflow of 200LFM at this maximum ambient
temperature is required for HCA cards and NICs.To guarantee proper airflow, allow at least 8cm (3
inches) of clearance around the ventilation openings.
During Lightning - Electrical Hazard
During periods of lightning activity, do not work on the equipment or connect or disconnect cables.
Copper Cable Connecting/Disconnecting
Some copper cables are heavy and not flexible, as such, they should be carefully attached to or
detached from the connectors. Refer to the cable manufacturer for special warnings and
instructions.
Equipment Installation
This equipment should be installed, replaced, or serviced only by trained and qualified personnel.
Equipment Disposal
The disposal of this equipment should be in accordance to all national laws and regulations.
Local and National Electrical Codes
This equipment should be installed in compliance with local and national electrical codes.
Hazardous Radiation Exposure
Caution – Use of controls or adjustment or performance of procedures other than
those specified herein may result in hazardous radiation exposure.For products with
optical ports.
CLASS 1 LASER PRODUCT and reference to the most recent laser standards:
IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20
Safety warnings are provided here in the English language. For safety warnings in other
languages, refer to the Adapter Installation Safety Instructions.


16
Installation Procedure Overview
The installation procedure of ConnectX-4 Lx adapter cards involves the following steps:
Step Procedure Direct Link
1Check the system’s hardware and software
requirements.
System Requirements
2 Pay attention to the airflow consideration within
the host system
Airflow Requirements
3 Follow the safety precautions Safety Precautions
4 Unpack the package Unpack the package
5 Follow the pre-installation checklist Pre-Installation Checklist
7 Install the ConnectX-4 Lx for OCP Spec 2.0 card in
the system
Installation Instructions
8 Connect cables or modules to the card Cables and Modules
9 Identify ConnectX-4 Lx in the system Identifying Your Card
System Requirements
Hardware Requirements
A system with a PCI Express x8 FCI B2B connector is required for installing the card.
Airflow Requirements
ConnectX-4 Lx adapter cards are offered with two airflow patterns: from the heatsink to the
network ports, and vice versa, as shown below.
Please refer to theSpecificationssection for airflow numbers for each specific card model.
Unless otherwise specified, NVIDIA products are designed to work in an environmentally
controlled data center with low levels of gaseous and dust (particulate) contamination.
The operation environment should meet severity level G1 as per ISA 71.04 for gaseous
contamination and ISO 14644-1 class 8 for cleanliness level.

For proper operation and performance, please make sure to use a PCIe slot with a
corresponding bus width and that can supply sufficient power to your card. Refer to the
Specifications section of the manual for more power requirements.

All cards in the system should be planned with the same airflow direction.


17
•
•
•
•
•
•
•
Software Requirements
SeeOperating Systems/Distributionssection under the Introduction section.
Software Stacks - NVIDIA OpenFabric software package MLNX_OFED for Linux, WinOF-2 for
Windows, and VMware. See theDriver Installationsection.
Safety Precautions
The adapter is being installed in a system that operates with voltages that can be lethal. Before
opening the case of the system, observe the following precautions to avoid injury and prevent
damage to system components.
Remove any metallic objects from your hands and wrists.
Make sure to use only insulated tools.
Verify that the system is powered off and is unplugged.
It is strongly recommended to use an ESD strap or other antistatic devices.
Shut down your system if active;Turn off the power to the system, and disconnect the power
cord. Refer to the system documentation for instructions. Before you install the ConnectX-4
Lx card, make sure that the system is disconnected from power.
Unpacking the Package
Check the adapter for visible damage that may have occurred during shipping. Please note that the
cards must be placed on an antistatic surface.
Please note that if the card is removed hastily from the antistatic bag, the plastic ziplock
may harm the EMI fingers on the networking connector. Carefully remove the card from the
antistatic bag to avoid damaging the EMI fingers.


18
1.
2.
3.
Installation Instructions
This section provides detailed instructions on how to install your adapter card in a system.
Before inserting the card in a system, make sure that the system is off and the power cord is
not connected to the server. Please follow proper electrical grounding procedures.
Locate an available PCI Express slot for the adapter card. Make sure the adapter clips or
screws are open.
Place the adapter card on the clips without applying any pressure.
Please note that the following figures are for illustration purposes only.


19
4.
5.
6.
1.
2.
a.
b.
c.
d.
3.
Applying even pressure on four corners of the card (as shown in the below picture), insert the
adapter card into the PCI Express slot until firmly seated.
Secure the adapter with the adapter clips or screws.
Close the system case.
Cables and Modules
Cable Installation
All cables can be inserted or removed with the unit powered on.
To insert a cable, press the connector into the port receptacle until the connector is firmly
seated.
Support the weight of the cable before connecting the cable to the adapter card. Do
this by using a cable holder or tying the cable to the rack.
Determine the correct orientation of the connector to the card before inserting the
connector. Do not try and insert the connector upside down. This may damage the
adapter card.
Insert the connector into the adapter card. Be careful to insert the connector straight
into the cage. Do not apply any torque, up or down, to the connector cage in the
adapter card.
Make sure that the connector locks in place.
After inserting a cable into a port, the Green LED indicator will light when the physical
connection is established (that is, when the unit is powered on and a cable is plugged into
To uninstall the adapter card, see Uninstalling the Card.

When installing cables make sure that the latches engage.

Always install and remove cables by pushing or pulling the cable and
connector in a straight line with the card.


20
4.
5.
6.
1.
2.
3.
4.
the port with the other end of the connector plugged into a functioning port).SeeLED
Interfaceunder the Interfaces section.
After plugging in a cable, lock the connector using the latching mechanism particular to the
cable vendor. When data is being transferred the Green LED will blink. SeeLED
Interfaceunder the Interfaces section.
Care should be taken as not to impede the air exhaust flow through the ventilation holes. Use
cable lengths which allow for routing horizontally around to the side of the chassis before
bending upward or downward in the rack.
To remove a cable, disengage the locks and slowly pull the connector away from the port
receptacle.LEDindicator will turn off when the cable is unseated.
Identifying the Card in Your System
On Linux
Get the device location on the PCI bus by runninglspciand locating lines with the string“Mellanox
Technologies”:
lspci |grep -i Mellanox
Network controller: Mellanox Technologies MT27710 Family [ConnectX-4Lx]
On Windows
Open Device Manager on the server. ClickStart=>Run, and then enterdevmgmt.msc.
ExpandSystem Devices and locate your NVIDIAConnectX-4 Lx adapter card.
Rightclickthe mouse on your adapter's row and selectPropertiesto display the adapter card
properties window.
Click theDetailstab and selectHardware Ids(Windows 2012/R2/2016) from
thePropertypull-down menu.
PCI Device (Example)