Nvidia Mellanox ConnectX-4 Lx User manual

NVIDIA Mellanox ConnectX-4 Lx Ethernet
Adapter Cards User Manual
Exported on Dec/24/2020 12:18 PM
https://docs.mellanox.com/x/mYaO

2
Table of Contents
Introduction ................................................................................................................. 6
Product Overview................................................................................................................6
Features and Benefits .........................................................................................................7
Operating Systems/Distributions .........................................................................................9
Connectivity ......................................................................................................................10
Interfaces ................................................................................................................... 11
Ethernet SFP28 andQSFP28 Interfaces..............................................................................11
PCI Express Interface .......................................................................................................11
LED Interface ....................................................................................................................11
Heatsink Interface .............................................................................................................11
SMBus Interface ...............................................................................................................12
Voltage Regulators............................................................................................................12
Hardware Installation ................................................................................................ 13
Safety Warnings................................................................................................................13
Installation Procedure Overview........................................................................................13
System Requirements.......................................................................................................14
Hardware Requirements...............................................................................................14
Airflow Requirements ..................................................................................................14
Software Requirements................................................................................................14
Unpacking the Product......................................................................................................14
Bracket Replacement Instructions.....................................................................................15
Installation Instructions ....................................................................................................16
Cables and Modules .........................................................................................................17
Identifying the Card inYour System .....................................................................................18
Uninstalling the Card.........................................................................................................19
Driver Installation ...................................................................................................... 21
Linux Driver Installation ...................................................................................................21
Prerequisites ...............................................................................................................21
Downloading Mellanox OFED.......................................................................................21
Installing Mellanox OFED .............................................................................................22

3
Installing MLNX_OFED Using YUM..............................................................................27
Installing MLNX_OFED Using apt-get Tool...................................................................29
Updating Firmware AfterInstallation ............................................................................31
UEFI Secure Boot ........................................................................................................32
Performance Tuning.....................................................................................................33
Windows Driver Installation ...............................................................................................34
Software Requirements................................................................................................34
Downloading Mellanox WinOF-2 Driver........................................................................34
Installing Mellanox WinOF-2 Driver ..............................................................................35
Uninstalling Mellanox WinOF-2 Driver..........................................................................41
Extracting Files Without Running Installation................................................................42
Firmware Upgrade........................................................................................................44
VMware Driver Installation ................................................................................................44
Hardware and Software Requirements.........................................................................44
Installing Mellanox NATIVE ESXi Driver for VMware vSphere......................................45
Removing Earlier Mellanox Drivers ..............................................................................45
Firmware Programming ..............................................................................................46
Updating Adapter Firmware ....................................................................................... 47
Troubleshooting ......................................................................................................... 48
General Troubleshooting...................................................................................................48
Linux Troubleshooting.......................................................................................................48
Windows Troubleshooting .................................................................................................49
Specifications ............................................................................................................. 50
MCX4121A-XCAT Specifications .........................................................................................50
MCX4121A-ACAT Specifications ......................................................................................51
Airflow Specifications ........................................................................................................53
Board Mechanical Drawing andDimensions.........................................................................54
Mechanical Drawing of Dual-port 10/25 Gb/s Adapter Card .........................................54
Bracket Mechanical Drawing.............................................................................................55
Dual-port Tall Bracket .................................................................................................56
Finding the MAC on the Adapter Card ....................................................................... 57
MCX4121A-ACAT Board Label(Example) ...........................................................................58
Document Revision History ........................................................................................ 59

4
About This Manual
This User Manual describes NVIDIA® Mellanox® ConnectX®-4 Lx Ethernet adapter cards. It provides
details as to the interfaces of the board, specifications, required software and firmware for operating
the board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-4 Lx Ethernet
adapter cards.
OPN Marketing Description
MCX4121A-XCAT ConnectX®-4 Lx EN network interface card, 10GbE dual-port
SFP28, PCIe3.0 x8, tall bracket
MCX4121A-ACAT ConnectX®-4 Lx EN network interface card, 25GbE dual-port
SFP28, PCIe3.0 x8, tall bracket
Intended Audience
This manual is intended for the installer and user of these cards. The manual assumes basic familiarity
with Ethernet network and architecture specifications.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact us through
the following methods:
• URL: http://www.mellanox.com > Support
• E-mail: [e-mail]
• Tel: [phone]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract for details
regarding Technical Support.

5
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seek assistance through their reseller.
Related Documentation
MellanoxOFEDfor
Linux User Manual
andRelease Notes
User Manual describing OFED features, performance, band diagnostic, tools content
and configuration. See Mellanox OFED for Linux Documentation.
WinOF-2 for
Windows User
Manual and
Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools
content and configuration. See WinOF-2 for Windows Documentation.
Mellanox VMware
for Ethernet User
Manual and
Release Notes
User Manual describing the various components of the Mellanox ConnectX® NATIVE
ESXi stack. See http://www.mellanox.com Products > Software > Ethernet Drivers >
VMware Driver
Mellanox
Firmware Utility
(mlxup) User
Manual and
Release Notes
Mellanox firmware update and query utility used to update the firmware. See http://
www.mellanox.com > Products > Software > Firmware Tools >mlxup Firmware Utility
Mellanox
Firmware Tools
(MFT)UserManual
User Manual describing the set of MFT firmware management tools for a single node.
See MFT User Manual.
IEEE Std 802.3
Specification
IEEE Ethernet specification at http://standards.ieee.org/
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications at
https://pcisig.com/specifications
Mellanox LinkX
Interconnect
Solutions
Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the
performance of High-Performance Computing networks, requiring high-bandwidth,
low-latency connections between computenodes andswitchnodes. Mellanox offers
one of industry’s broadest portfolio of QDR/FDR10(40Gb/s), FDR (56Gb/s), EDR/
HDR100 (100Gb/s) and HDR (200Gb/s) cables, including Direct Attach Copper cables
(DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a wide
range of lengths from 0.5m to 10km. In addition to meeting IBTA standards, Mellanox
tests every product in an end-to-end environment ensuring a Bit Error Rate of less
than 1E-15. Read more at https://www.mellanox.com/products/interconnect/
infiniband-overview.php
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.

6
Introduction
Product Overview
This is the User Guide for Ethernet adapter cards based on the ConnectX®-4 Lx integrated circuit
device. These adapters connectivity provide the highest performing low latency and most flexible
interconnect solution for PCI Express Gen 3.0 servers used in Enterprise Data Centers and High-
Performance Computing environments.
The following provides the ordering part number, port speed, number of ports, and PCI Express speed.
Each adapter is shipped with two bracket heights - short and tall (assembled).
ConnectX-4 Lx 10Gb/s Ethernet Adapter Cards
Model ConnectX-4 Lx 10Gb/s Ethernet Adapter Cards
Part Number MCX4121A-XCAT
Ethernet Data Rate 1/10 Gb/s
Network Connector Type Dual-port SFP28
PCI Express Connectors PCIe Gen 3.0 SERDES @ 8.0GT/s
Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile
UEFI -
Host Management -
RoHS RoHS Compliant
IC Part Number MT27712A0-FDCF-XE
Device ID 4117 for Physical Function (PF)
4118 for Virtual Function (VF)
ConnectX-4 Lx 25Gb/s Ethernet Adapter Cards
Model ConnectX-4 Lx 25Gb/s Ethernet Adapter Cards
Part Number
MCX4121A- ACAT
Network
Connector
Type
Dual-port SFP28
Ethernet Data
Rate
1/10/25 Gb/s
PCI Express
Connectors
PCIe Gen 3.0 SERDES @ 8.0GT/s
Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile
UEFI -

7

This section describes hardware features and capabilities. Please refer to the relevant driver
and/or firmware release notes for feature availability.
Model ConnectX-4 Lx 25Gb/s Ethernet Adapter Cards
Secure
Firmware
-
Host
Management
-
RoHS RoHS Compliant
IC Part
Number
MT27712A0-FDCF-AE
Device ID 4117 for Physical Function (PF)
4118 for Virtual Function (VF)
For more detailed information see Specifications.
Features and Benefits
Feature Description
PCI Express (PCIe) Uses PCIe Gen 3.0 (8GT/s) through an x8 edge connector. Gen 1.1 and
2.0 compatible.

8
Feature Description
Up to 25 Gigabit Ethernet Mellanox adapters comply with the following IEEE 802.3 standards:
• 25GbE / 10GbE / 1GbE
• IEEE 802.3by, Ethernet Consortium25 Gigabit Ethernet,
supporting all FEC modes
• IEEE 802.3by 25 GigabitEthernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• IEEE 802.3ad, 802.1AX LinkAggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb(PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
Memory
• PCIExpress-storesandaccessesEthernetfabricconnection
information and packet data.
• SPI-includesone 16MB SPI Flash device (W25Q128FVSIG
device by WINBOND-NUVOTON).
Overlay Networks In order to better scale their networks, data center operators often
create overlay networks that carry traffic from individual virtual
machines over logical tunnels in encapsulated formats such as
NVGRE and VXLAN. While this solves network scalability issues, it
hides the TCP packet from the hardware offloading engines, placing
higher loads on the host CPU. ConnectX-4 Lx effectively addresses
this by providing advanced NVGRE and VXLAN hardware offloading
engines that encapsulate and de-capsulate the overlay protocol.
RDMAand RDMA overConverged
Ethernet (RoCE)
ConnectX-4 Lx, utilizing IBTA RDMA (Remote Data Memory Access)
and RoCE (RDMA over Converged Ethernet) technology, delivers low-
latency and high-performance over Band and Ethernet networks.
Leveraging data center bridging (DCB) capabilities, as well as
ConnectX-4 Lx, advanced congestion control hardware mechanisms,
RoCE provides efficient low-latency RDMA services over Layer 2 and
Layer 3 networks.
Mellanox PeerDirect™ PeerDirect™ communication provides high-efficiency RDMA access
by eliminating unnecessary internal data copies between components
on the PCIe bus (for example, from GPU to CPU), and therefore
significantly reduces application run time. ConnectX-4 Lx advanced
acceleration technology enables higher cluster efficiency and
scalability to tens of thousands of nodes.
CPU Offload Adapter functionality enabling reduced CPU overhead allowing more
available CPU for computation tasks.
Quality of Service (QoS)
Supportforport-basedQualityofServiceenablingvarious application
requirements for latency andSLA.
Hardware-based I/O Virtualization ConnectX-4 Lx provides dedicated adapter resources and guaranteed
isolation and protection for virtual machines within the server.

9
Feature Description
Storage Acceleration A consolidated compute and storage network achieves significant
cost-performance advantages over multi-fabric networks. Standard
block and file access protocols can leverage RDMA for high-
performance storage access.
• NVMe over Fabric offloads for the targetmachine
SR-IOV ConnectX-4 Lx SR-IOV technology provides dedicated adapter
resources and guaranteed isolation and protection for virtual
machines (VM) within the server.
NC-SI The adapter supports a Network Controller Sideband Interface (NC-
SI), MCTP over SMBus and MCTP over PCIe - Baseboard Management
Controller interface.
High-Performance Accelerations
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on ReliableTransport
• Burst Buffer Offloads for Background Checkpointing
Operating Systems/Distributions
• RHEL/CentOS
• Windows
• FreeBSD
• VMware
• OpenFabrics Enterprise Distribution (OFED)
• OpenFabrics Windows Distribution (WinOF-2)

10
Connectivity
• Interoperable with 1/10/25 Gb/s Ethernet switches
• Passive copper cable with ESD protection
• Powered connectors for optical and active cablesupport

11

The adapter card includes special circuits to protect from ESD shocks to the card/server when
plugging copper cables.
Interfaces
Ethernet SFP28 andQSFP28 Interfaces
The network ports of the ConnectX®-4 Lx adapter card are compliant with the IEEE 802.3 Ethernet
standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28/QSFP28
connectors on the adapter card.
PCI Express Interface
The table below describes the supported PCIe interface in ConnectX-4 Lx adapter cards.
Supported PCIe Interface Features
PCIe Gen 3.0 (1.1 and 2.0 compatible)
through x8 edge connectors
Link Rates: 2.5. 5.0 or 8.0GT/s.
Auto Negotiation to: x8, x4, x2 or x1.
Support for MSI/MSI-X mechanisms.
LED Interface
There is one bi-color I/O LED per port to indicate link status located on the adapter card.
Link Indications (Ethernet Mode)
LED Color and State Description
Off A link has not been established
Blinking Amber(a) 4 Hz blinking amber indicates a problem with the link
Solid Green Indicates a valid link with no active traffic
Blinking Green Indicates a valid logical link with active traffic
Note a. 1Hz Blinking amber occurs due to running a beacon command for locating the adapter card.
Heatsink Interface
A heatsink is attached to the ConnectX-4 Lx IC in order to dissipate the heat from the ConnectX-4 Lx IC.
It is attached either by using four spring-loaded push pins that insert into four mounting holes, or by
screws.

12
ConnectX-4 Lx IC has a thermal shutdown safety mechanism which automatically shuts down the
ConnectX-4 Lx card in cases of high-temperature event, improper thermal coupling or heatsink
removal.
SMBus Interface
ConnectX-4 Lx technology maintains support for manageability through a BMC. ConnectX-4 Lx PCIe
stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as
if it is a standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific
manageability solution in use by the server, please contact Mellanox Support.
Voltage Regulators
The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These
voltage supply pins feed on-board regulators that provide the necessary power to the various
components on the card.

13

Safety warnings are provided here in the English language. For safety warnings in other
languages, refer to the
Adapter Installation Safety Instructions document available on
Mellanox's website.
Hardware Installation
Installation and initialization of ConnectX-4 Lx adapter cards require attention to the mechanical
attributes, power specification, and precautions for electronic equipment.
Safety Warnings
Please observe all safety warnings to avoid injury and prevent damage to system components. Note
that not all warnings are relevant to allmodels,
Unable to render include or excerpt-include. Could not retrieve page.
Installation Procedure Overview
The installation procedure of ConnectX-4 Lx adapter cards involves the following steps:
Step Procedure Direct Link
1 Check the system’s hardware and software
requirements.
Refer to System Requirements
2 Pay attention to the airflow consideration within the
host system.
Refer to Airflow Requirements
3
Unpack the product.
Refer to Unpacking the Product
4
(Optional) Replace the full-height mountingbracket
with the supplied shortbracket.
Refer to Bracket Replacement
Instructions
5 Install the ConnectX-4 Lx adapter card in the
system.
Refer to Installation Instructions
6 Connect cables or modules to the card. Refer to Cables and Modules
7 Identify the ConnectX-4 Lx adapter card in the
system.
Refer to Identifying the Card in Your
System

14

Unless otherwise specified, Mellanox products are designed to work in an environmentally
controlled data center with low levels of gaseous and dust (particulate) contamination.
The
operation environment should meet severity level G1 as per ISA 71.04 for gaseous
contamination and ISO 14644-1 class 8 for cleanliness level.

All cards in the system should be planned with the same airflow direction.

The adapter is being installed in a system that operates with voltages that can be lethal.
Before
opening the case of the system, observe the following precautions to avoid injury
and
prevent damage to system
components.
• Remove any metallic objects from your hands and wrists.
• Make sure to use only insulated tools.
• Verify that the system is powered off and is unplugged.
• It is strongly recommended to use an ESD strap or other antistatic devices.
System Requirements
Hardware Requirements
A system with a PCI Express x8 edge connector is required for installing the card.
Airflow Requirements
ConnectX-4 Lx adapter cards are offered with two airflow patterns: from the heatsink to the network
ports, and vice versa, as shown below.
Please refer to the Specifications chapter for airflow numbers for each specific card model.
Software Requirements
• See Operating Systems/Distributions section under the Introduction section.
• Software Stacks - Mellanox OpenFabric software package MLNX_OFED for Linux, WinOF-2 for
Windows, and VMware. See the Driver Installation
section.
Unpacking the Product

15

Due to risk of damaging the EMI gasket, it is not recommended to replace the bracket more
than three times.
1. Unpack the adapter package and place them on an antistatic surface, and verify you have
received the following items:
a. ConnectX-4 Lx Adapter Card, with a tall bracket assembled on the card.
b. Short bracket (accessory)
2. Check the parts for visible damage that may have occurred during shipping.
3. Shut down your system if active:
Turn off the power to the system, and disconnect the power cord. Refer to the system
documentation for instructions. Before you install the ConnectX-4 Lx card, make sure that the
system is disconnected from power.
Bracket Replacement Instructions
The card is usually shipped with an assembled high-profile bracket. If this form factor is suitable for
your requirements, you can skip the remainder of this section and move to Installation Instructions. If
you need to replace the high-profile bracket with the short bracket that is included in the shipping box,
please follow the instructions in this section.
To replace the bracket you will need the following parts:
• The new brackets of the proper height
• The 2 screws saved from the removal of the bracket
Removing the Existing Bracket
1. Using a torque driver, remove the two screws holding the bracket in place.

Please note that if the card is removed hastily from the antistatic bag, the plastic
ziplock may harm the EMI fingers on the networking connector. Carefully remove the
card from the antistatic bag to avoid damaging the EMI fingers.

16

Please note that the following figures are for illustration purposes only.
2. Separate the bracket from the ConnectX-4 Lxcard.
3. Save the two screws.
Installing the New Bracket
1. Place the bracket onto the card until the screw holes line up.
2. Screw on the bracket using the screws saved from the bracket removal procedure above.
Installation Instructions
This section provides detailed instructions on how to install your adapter card in a system.
1. Before installing the card, make sure that the system is off and the power cord is notconnected
to the server. Please follow proper electrical grounding procedures.
2. Open the system case.
3. Place the adapter in an available PCI Express slot.

Use a torque driver to apply up to 2 lbs-in torque on the screws.

Do not force the bracket onto the adapter card.

Be careful not to put stress on the LEDs on the adapter card.

17

To uninstall the adapter card, see Uninstalling the Card.
4. Applying even pressure at both corners of the card, insert the adapter card into the PCI Express
slot until firmly seated.
5. When the adapter is properly seated, the port connectors are aligned with the slot opening, and
the adapter faceplate is visible against the system chassis.
6. Secure the adapter with the adapter clip orscrew.
Cables and Modules
To obtain the list of supported Mellanox cables for your adapter, please refer to the Cables Reference
Table at http://www.mellanox.com/products/interconnect/cables-configurator.php.
Cable Installation
1. All cables can be inserted or removed with the unit powered on.
2. To insert a cable, press the connector into the port receptacle until the connector is firmly
seated.

18
lspci |grep -i Mellanox
Network controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
a. Support the weight of the cable before connecting the cable to the adapter card. Do this
by using a cable holder or tying the cable to the rack.
b. Determine the correct orientation of the connector to the card before inserting the
connector. Do not try and insert the connector upside down. This may damage the
adapter card.
c. Insert the connector into the adapter card. Be careful to insert the connector straight into
the cage. Do not apply any torque, up or down, to the connector cage in the adapter card.
d. Make sure that the connector locks in place.
3. After inserting a cable into a port, the Green LED indicator will light when the physical
connection is established (that is, when the unit is powered on and a cable is plugged into the
port with the other end of the connector plugged into a functioning port). See LED Operations
under the Interfaces section.
4. After plugging in a cable, lock the connector using the latching mechanism particular to the
cable vendor. When data is being transferred, the Green LED will blink. See LED
Operations under the Interfaces section.
5. Care should be taken as not to impede the air exhaust flow through the ventilation holes. Use
cable lengths which allow for routing horizontally around to the side of the chassis before
bending upward or downward in the rack.
6. To remove a cable, disengage the locks and slowly pull the connector away from the port
receptacle. The LED indicator will turn off when the cable is unseated.
Identifying the Card inYour System
On Linux
Get the device location on the PCI bus by running lspci and locating lines with the string “Mellanox
Technologies”:
On Windows
1. Open Device Manager on the server. Click Start => Run, and then enter devmgmt.msc.
2. Expand System Devices and locate your Mellanox ConnectX-4 Lx adapter card.
3. Right click the mouse on your adapter's row and select Properties to display the adapter card
properties window.
4. Click the Details tab and select Hardware Ids (Windows 2012/R2/2016) from the Propertypull-
down menu.

When installing cables make sure that the latches engage.

Always install and remove cables by pushing or pulling the cable and connector
in a straight line with the card.

19

If the PCI device does not have a Mellanox adapter ID, return to Step 2 to check
another device.

The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID
repository at http://pci-ids.ucw.cz/read/PC/15b3.
PCI Device (Example)
5. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the
display example above, notice the sub-string “PCI\VEN_15B3&DEV_1003”: VEN is equal to
0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1018 (for
ConnectX-4 Lx) – this is a valid Mellanox Technologies PCI Device ID.
Uninstalling the Card
Safety Precautions
The adapter is installed in a system that operates with voltages that can be lethal. Before uninstalling
the adapter card, please observe the following precautions to avoid injury and prevent damage to
system components.
1. Remove any metallic objects from your hands and wrists.
2. It is strongly recommended to use an ESD strap or other antistatic devices.
3. Turn off the system and disconnect the power cord from the server.
Card Removal
Please note that the following images are for illustration purposes only.

20
1. Verify that the system is powered off and unplugged.
2. Wait 30 seconds.
3. To remove the card, disengage the retention mechanisms on the bracket (clips or screws).
4. Holding the adapter card from its center, gently pull the adapter card out of the PCI Express
slot.
5. When the port connectors reach the top of the chassis window, gently pull the adapter card in
parallel to the motherboard.

Please note that the following images are for illustration purposes only.