Nvidia Mellanox ConnectX-5 User manual

NVIDIA Mellanox ConnectX-5 Ethernet Adapter
Cards User Manual
100Gb/s Ethernet Adapter Cards; Intelligent RDMA (RoCE) enabled NICs. Supports 1, 10, 25, 50, and
100 Gigabit Ethernet speeds.
Exported on Dec/23/2020 12:46 PM
https://docs.mellanox.com/x/SQFp

2
Table of Contents
Introduction ..........................................................................................................................8
Product Overview.........................................................................................................................8
Operating Systems/Distributions ................................................................................................11
Connectivity ...............................................................................................................................11
Interfaces ............................................................................................................................13
Ethernet SFP28 and QSFP28Interfaces.......................................................................................13
PCI Express Interface ................................................................................................................13
LED Interface .............................................................................................................................13
Heat Sink Interface.....................................................................................................................14
SMBus Interface ........................................................................................................................14
Voltage Regulators.....................................................................................................................14
Thermal Sensors........................................................................................................................15
Hardware Installation .........................................................................................................16
Safety Warnings.........................................................................................................................16
Installation Procedure Overview.................................................................................................17
System Requirements................................................................................................................17
Hardware Requirements........................................................................................................17
Airflow Requirements ...........................................................................................................18
Software Requirements.........................................................................................................18
Unpacking the Product...............................................................................................................19
Bracket Replacement Instructions..............................................................................................20
Installation Instructions .............................................................................................................20
Cables and Modules ..................................................................................................................22
Identifying the Card in YourSystem..............................................................................................22
Uninstalling the Card..................................................................................................................23
Driver Installation ...............................................................................................................25
Linux Driver Installation ............................................................................................................25
Prerequisites ........................................................................................................................25
Downloading Mellanox OFED................................................................................................25
Installing Mellanox OFED ......................................................................................................26

3
Installing MLNX_OFED Using YUM.......................................................................................31
Installing MLNX_OFED Using apt-get Tool............................................................................33
Updating Firmware AfterInstallation .....................................................................................35
UEFI Secure Boot .................................................................................................................36
Performance Tuning..............................................................................................................37
Windows Driver Installation ........................................................................................................38
Software Requirements.........................................................................................................38
Downloading Mellanox WinOF-2 Driver.................................................................................38
Installing Mellanox WinOF-2 Driver .......................................................................................39
Uninstalling Mellanox WinOF-2 Driver...................................................................................45
Extracting Files Without Running Installation.........................................................................46
Firmware Upgrade.................................................................................................................48
VMware Driver Installation .........................................................................................................48
Hardware and Software Requirements..................................................................................48
Installing Mellanox NATIVE ESXi Driver for VMware vSphere...............................................49
Removing Earlier Mellanox Drivers .......................................................................................49
Firmware Programming .......................................................................................................50
Updating Adapter Firmware ................................................................................................51
Troubleshooting ..................................................................................................................52
General Troubleshooting............................................................................................................52
Linux Troubleshooting................................................................................................................52
Windows Troubleshooting ..........................................................................................................53
Specifications ......................................................................................................................54
Airflow Specifications .................................................................................................................57
Board Mechanical Drawing andDimensions..................................................................................58
Mechanical Drawing of a Dual-Port x8 Card ..........................................................................58
Bracket Mechanical Drawing......................................................................................................59
Single-port Tall Bracket .......................................................................................................59
Dual-port Tall Bracket ..........................................................................................................59
Finding the MAC on the Adapter Card ................................................................................61
MCX512A- ACAT Board Label(Example) .....................................................................................61
MCX515A-CCAT Board Label (Example) .......................................................................................62
MCX516A-CCAT Board Label (Example) .......................................................................................62

4
Document Revision History .................................................................................................63

5
About This Manual
This User Manual describes NVIDIA® Mellanox® ConnectX®-5 and ConnectX®-5 Ex Ethernet Single
and Dual SFP28 and QSFP28 port PCI Express x8/x16 adapter cards. It provides details as to the
interfaces of the board, specifications, required software and firmware for operating the board, and
relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-5 Ethernet
adapter cards.
IC in Use OPN Marketing Description
ConnectX®-5 MCX512A-ACAT ConnectX®-5 EN network interface card,
25GbE dual-port SFP28, PCIe Gen 3.0 x8, tall
bracket
MCX515A-CCAT ConnectX®-5 EN network interface card,
100GbE single-port QSFP28, PCIe Gen 3.0
x16, tall bracket
MCX516A-CCAT ConnectX®-5 EN network interface card,
100GbE dual-port QSFP28, PCIe Gen 3.0
x16, tall bracket

6
Intended Audience
This manual is intended for the installer and user of these cards. The manual assumes basic familiarity
with Ethernet network and architecture specifications.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact us through
the following methods:
• URL: http://www.mellanox.com > Support
• E-mail: [e-mail]
• Tel: [phone]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract for details
regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seek assistance through their reseller.
Related Documentation
Mellanox OFED for
Linux User Manual
and Release Notes
User Manual describing OFED features, performance, band diagnostic, tools
content and configuration. See Mellanox OFED for Linux Documentation.
WinOF-2 for
Windows User Manual
and Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools
content and configuration. See WinOF-2 for Windows Documentation.
Mellanox VMware for
Ethernet User Manual
and Release Notes
User Manual describing the various components of the Mellanox ConnectX®
NATIVE ESXi stack. See http://www.mellanox.com Products > Software > Ethernet
Drivers > VMware Driver
Mellanox Firmware
Utility (mlxup) User
Manual and Release
Notes
Mellanox firmware update and query utility used to update the firmware. See http:/
/www.mellanox.com > Products > Software > Firmware Tools >mlxup Firmware
Utility
Mellanox Firmware
Tools (MFT) User
Manual
User Manual describing the set of MFT firmware management tools for a single
node. See MFT User Manual.
IEEE Std 802.3
Specification
IEEE Ethernet specification at http://standards.ieee.org/
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications at
https://pcisig.com/specifications

7
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in MegaBytes.
The use of Mb or Mbits (small b) indicates size in MegaBits. In this document, PCIe is used to mean PCI
Express.
Revision History
A list of the changes made to this document are provided in Document Revision History.
Mellanox LinkX Ethernet cables and transceivers are designed to maximize the
performance of High-Performance Computing networks, requiring high-
bandwidth, low-latency connections between compute nodes and switch nodes.
Mellanox offers one of the industry’s broadest portfolios of 40Gb/s, 56Gb/s and
100Gb/s cables, including Direct Attach Copper cables (DACs), copper splitter
cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths
from 0.5m to 10km. In addition to meeting Ethernet standards, Mellanox tests
every product in an end-to-end environment ensuring a Bit Error Rate of less than
1E-15. Read more at LinkX® Ethernet Cables and Transceivers.
Mellanox LinkX
Interconnect
Solutions

8
Introduction
Product Overview
This is the User Guide for Ethernet adapter cards based on the ConnectX®-5 and ConnectX®-5 Ex
integrated circuit device. These adapters connectivity provide the highest performing low latency and
most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in Enterprise Data Centers
and High-Performance Computing environments.
The following provides the ordering part number, port speed, number of ports, and PCI Express speed.
Each adapter comes with two bracket heights - short and tall.
ConnectX-5 Ethernet 25Gb/s Adapter Cards
Model ConnectX-5 Ethernet Adapter Cards
Part Number MCX512A-ACAT
Ethernet Data Rate 1/10/25 Gb/s
Network Connector
Type
Dual-port SFP28
PCI Express
Connectors
PCIe Gen 3.0 x8 SerDes @ 8.0GT/s
Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile
Host Management -

9
Model ConnectX-5 Ethernet Adapter Cards
UEFI Enabled (x86/
Arm)
-
RoHS RoHS Compliant
IC Part Number MT27808A0-FCCF-EV
ConnectX-5 Ethernet 100Gb/s Adapter Cards
Model ConnectX-5 Ethernet Adapter Cards
Part Number MCX515A-CCAT MCX516A-CCAT
Ethernet Data Rate 10/25/40/50/100 Gb/s
Network Connector
Type
Single-port QSFP28 Dual-port QSFP28
PCI Express
Connectors
PCIe Gen 3.0 x16
SerDes @ 8.0GT/s
Dimensions
2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile
Host Management - -
UEFI Enabled (x86/
Arm)
- -
RoHS RoHS Compliant
IC Part Number MT27808A0-FCCF-EV
For more detailed information see Specifications.

10

This section describes hardware features and capabilities. Please refer to the relevant driver
and/or firmware release notes for feature availability.
Features and Benefits
Feature Description
PCI Express (PCIe) Uses PCIe Gen 3.0 (8GT/s) through an x8 or x16 edge connector. Gen
1.1 and 2.0 compatible.
Up to 25 Gigabit Ethernet Mellanox adapters comply with the following IEEE 802.3 standards:
• 100GbE/ 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet,
supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 GigabitEthernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX LinkAggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb(PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
Memory
• PCIExpress - stores and accesses Ethernet fabric
connection information and packet data.
• SPI Quad - includes 128Mbit SPI Quad Flash device
(W25Q128FVSIG device by ST Microelectronics).
• FRU EEPROM - Stores the parameters and personality of the
card. The EEPROM capacity is 128Kbit. FRU I2C address is
(0x50) and is accessible through the PCIe SMBus. (Available
in QSFP28 OPNs)
Overlay Networks In order to better scale their networks, data center operators often
create overlay networks that carry traffic from individual virtual
machines over logical tunnels in encapsulated formats such as
NVGRE and VXLAN. While this solves network scalability issues, it
hides the TCP packet from the hardware offloading engines, placing
higher loads on the host CPU. ConnectX-5 effectively addresses this
by providing advanced NVGRE and VXLAN hardware offloading
engines that encapsulate and de-capsulate the overlay protocol.
RDMAand RDMA over Converged
Ethernet (RoCE)
ConnectX-5, utilizing IBTA RDMA (Remote Data Memory Access) and
RoCE (RDMA over Converged Ethernet) technology, delivers low-
latency and high-performance over Band and Ethernet networks.
Leveraging data center bridging (DCB) capabilities as well as
ConnectX-5 advanced congestion control hardware mechanisms,
RoCE provides efficient low-latency RDMA services over Layer 2 and
Layer 3 networks.

11
Feature Description
Mellanox PeerDirect™ PeerDirect™ communication provides high-efficiency RDMA access
by eliminating unnecessary internal data copies between
components on the PCIe bus (for example, from GPU to CPU), and
therefore significantly reduces application run time. ConnectX-5
advanced acceleration technology enables higher cluster efficiency
and scalability to tens of thousands of nodes.
CPU Offload Adapter functionality enabling reduced CPU overhead allowing more
available CPU for computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of Service (QoS)
Supportforport-basedQualityofServiceenablingvarious
application requirements for latency and SLA.
Hardware-based I/O Virtualization ConnectX-5 provides dedicated adapter resources and guaranteed
isolation and protection for virtual machines within the server.
Storage Acceleration A consolidated compute and storage network achieves significant
cost-performance advantages over multi-fabric networks. Standard
block and file access protocols can leverage RDMA for high-
performance storage access.
• NVMe over Fabric offloads for the targetmachine
• Erasure Coding
• T10-DIF Signature Handover
SR-IOV ConnectX-5 SR-IOV technology provides dedicated adapter
resources and guaranteed isolation and protection for virtual
machines (VM) within the server.
NC-SI The adapter supports a Network Controller Sideband Interface (NC-
SI), MCTP over SMBus and MCTP over PCIe - Baseboard
Management Controller interface.
High-Performance Accelerations
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on ReliableTransport
• Burst Buffer Offloads for Background Checkpointing
Operating Systems/Distributions
• RHEL/CentOS
• Windows
• FreeBSD
• VMware
• OpenFabrics Enterprise Distribution (OFED)
• OpenFabrics Windows Distribution (WinOF-2)
Connectivity
• Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches

12
• Passive copper cable with ESD protection
• Powered connectors for optical and active cablesupport

13

The adapter card includes special circuits to protect from ESD shocks to the card/server when
plugging copper cables.
Interfaces
Ethernet SFP28 and QSFP28Interfaces
The network ports of the ConnectX®-5 adapter card are compliant with the IEEE 802.3 Ethernet
standards listed in Features and Benefits. Ethernet traffic is transmitted through the SFP28/QSFP28
connectors on the adapter card.
PCI Express Interface
The table below describes the supported PCIe interface in ConnectX-5 and ConnectX-5 Ex adapter
cards.
IC Model Supported PCIe Interface Features
ConnectX-5 PCIe Gen 3.0 (1.1 and 2.0 compatible)
through x8 or x16 edgeconnectors
Link Rates: 2.5. 5.0 or 8.0GT/s.
Auto Negotiation to: x16, x8, x4, x2 or
x1.
Support for MSI/MSI-X mechanisms.
LED Interface
There is one bi-color I/O LED per port to indicate link status located on the adapter card.
Link Indications
State Bi-color LED (Yellow/Green) - Link speed
Beacon command for locating the adapter card
1Hz blinking Yellow

14
State Bi-color LED (Yellow/Green) - Link speed
Error 4Hz blinking Yellow Indicates an error with the link. The
error can be one of the following:
Link Activity
• A constant Green indicates a link with the maximum
networking speed.
• A constant Yellow indicates a link with less than the
maximum networking speed.
Link Up
• A constant Green indicates a link with the maximum
networking speed.
• A constant Yellow indicates a link with less than the
maximum networking speed.
Heat Sink Interface
A heatsink is attached to the ConnectX-5 IC in order to dissipate the heat from the ConnectX- 5 IC. It is
attached either by using four spring-loaded push pins that insert into four mounting holes or by
screws.
ConnectX-5 IC has a thermal shutdown safety mechanism which automatically shuts down the
ConnectX-5 card in cases of a high-temperature event, improper thermal coupling or heatsink
removal.
SMBus Interface
ConnectX-5 technology maintains support for manageability through a BMC. ConnectX-5 PCIe stand-up
adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a
standard Mellanox PCIe stand-up adapter. For configuring the adapter for the specific manageability
solution in use by the server, please contact Mellanox Support.
Voltage Regulators
The voltage regulator power is derived from the PCI Express edge connector 12V supply pins. These
voltage supply pins feed onboard regulators that provide the necessary power to the various
components on the card.
Error Type Description LED Behavior
I2C I2C access to the
networking ports
fails
Blinks until error
is fixed
Over-current
Over-current
condition of the
networking ports
Blinks until error
is fixed

15
Thermal Sensors
The adapter card incorporates the ConnectX IC which operates in the range of temperatures between
0C and 105C.
There are three thermal threshold definitions which impact the overall system operation state:
• Warning – 105°C: On managed systems only: When the device crosses the 100°C threshold, a
Warning Threshold message will be issued by the management SW, indicating to system
administration that the card has crossed the Warning threshold. Note that this temperature
threshold does not require nor lead to any action by hardware (such as adapter card shutdown).
• Critical – 120°C: When the device crosses this temperature, the firmware will automatically shut
down the device.
• Emergency – 130°C: In case the firmware fails to shut down the device upon crossing the
Critical threshold, the device will auto-shutdown upon crossing the Emergency (130°C)
threshold.
The card's thermal sensors can be read through the system’s SMBus. The user can read these
thermal sensors and adapt the system airflow in accordance with the readouts and the needs of the
above-mentioned IC thermal requirements.

16

Safety warnings are provided here in the English language. For safety warnings in other
languages, refer to the
Adapter Installation Safety Instructions document available on
mellanox.com.
Hardware Installation
Installation and initialization of ConnectX-5 adapter cards require attention to the mechanical
attributes, power specification, and precautions for electronic equipment.
Safety Warnings
Please observe all safety warnings to avoid injury and prevent damage to system components. Note
that not all warnings are relevant to allmodels.
Note that not all warnings are relevant to all models.
General Installation Instructions
Read all installation instructions before connecting the equipment to the power source.
Jewelry Removal Warning
Before you install or remove equipment that is connected to power lines, remove
jewelry such as bracelets, necklaces, rings, watches and so on. Metal objects heat up
when connected to power and ground and can meltdown, causing serious burns and/or
welding the metal object to the terminals.
Over-temperature
This equipment should not be operated in an area with an ambient temperature
exceeding the maximum recommended: 55°C (131°F). An airflow of 200LFM at this
maximum ambient temperature is required for HCA cards and NICs. To guarantee
proper airflow, allow at least 8cm (3 inches) of clearance around the ventilation
openings.
During Lightning - Electrical Hazard
During periods of lightning activity, do not work on the equipment or connect or dis
connect cables.
Copper Cable Connecting/Disconnecting
Some copper cables are heavy and not flexible, as such they should be carefully
attached to or detached from the connectors. Refer to the cable manufacturer for spe
cial warnings and instructions.
Equipment Installation
This equipment should be installed, replaced, or serviced only by trained and qualified
personnel.
Equipment Disposal
Disposal of this equipment should be in accordance to all national laws and regula
tions.

17

Unless otherwise specified, Mellanox products are designed to work in an environmentally
controlled data center with low levels of gaseous and dust (particulate) contamination.
The
operation environment should meet severity level G1 as per ISA 71.04 for gaseous
contamination and ISO 14644-1 class 8 for cleanliness level.
Local and National Electrical Codes
This equipment should be installed in compliance with local and national electrical
codes.
Hazardous Radiation Exposure
• Caution – Use of controls or adjustment or performance of procedures
other than those specified herein may result in hazardous radiation
exposure.For products with optical ports.
• CLASS 1 LASER PRODUCT and reference to the most recent laser
standards:
IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+
A2:20
Installation Procedure Overview
The installation procedure of ConnectX-5 adapter cards involves the following steps:
Step Procedure
1 Check the system’s hardware and software requirements.
2 Pay attention to the airflow consideration within the host system.
3
Unpack the product.
4 (Optional) Replace the full-height mounting bracket with the supplied short bracket.
5 Install the adapter card in the system.
6 Connect cables or modules to the card.
7 Identify ConnectX-5 adapter card in the system.
System Requirements
Hardware Requirements
A system with a PCI Express slot of corresponding bus width is required for installing the card.

For proper operation and performance, please make sure to use a PCIe slot with a
corresponding
bus width and that can supply sufficient power to your card. Refer to
the
Specifications section of the manual for more power requirements.

18

All cards in the system should be planned with the same airflow direction.
Airflow Requirements
ConnectX-5 adapter cards are offered with two airflow patterns: from the heatsink to the network
ports, and vice versa, as shown below.
Please refer to the Specifications section for airflow numbers for each specific card model.
Airflow from the heatsink to the network ports:
Airflowfromthenetworkports tothe heatsink:
Software Requirements
• See Operating Systems/Distributions section under the Introduction section.
• Software Stacks - Mellanox OpenFabric software package MLNX_OFED for Linux, WinOF-2 for
Windows, and VMware. See the Driver Installation
section.

19

The adapter is being installed in a system that operates with voltages that can be lethal.
Before
opening the case of the system, observe the following precautions to avoid injury
and
prevent damage to system
components.
• Remove any metallic objects from your hands and wrists.
• Make sure to use only insulated tools.
• Verify that the system is powered off and is unplugged.
• It is strongly recommended to use an ESD strap or other antistatic devices.
Unpacking the Product
1. Unpack the adapter package and place them on an antistatic surface, and verify you have
received the following items:
a. ConnectX-5 Adapter Card, with a tall bracket assembled on the card.
b. Short bracket (accessory)
2. Check the parts for visible damage that may have occurred during shipping.
3. Shut down your system if active:
Turn off the power to the system, and disconnect the power cord. Refer to the system
documentation for instructions. Before you install the ConnectX-5 card, make sure that the
system is disconnected from power.

Please note that if the card is removed hastily from the antistatic bag, the plastic
ziplock may harm the EMI fingers on the networking connector. Carefully remove the
card from the antistatic bag to avoid damaging the EMI fingers.

20

Due to risk of damaging the EMI gasket, it is not recommended to replace the bracket more
than three times.

Please note that the following figures are for illustration purposes only.
Bracket Replacement Instructions
The card is usually shipped with an assembled high-profile bracket. If this form factor is suitable for
your requirements, you can skip the remainder of this section and move to Installation Instructions. If
you need to replace the high-profile bracket with the short bracket that is included in the shipping box,
please follow the instructions in this section.
To replace the bracket you will need the following parts:
• The new brackets of the proper height
• The 2 screws saved from the removal of the bracket
Removing the Existing Bracket
1. Using a torque driver, remove the two screws holding the bracket in place.
2. Separate the bracket from the ConnectX-5 card.
3. Save the two screws.
Installing the New Bracket
1. Place the bracket onto the card until the screw holes line up.
2. Screw on the bracket using the screws saved from the bracket removal procedure above.
Installation Instructions
This section provides detailed instructions on how to install your adapter card in a system.
1. Before installing the card, make sure that the system is off and the power cord is notconnected
to the server. Please follow proper electrical grounding procedures.
2. Open the system case.

Use a torque driver to apply up to 2 lbs-in torque on the screws.

Do not force the bracket onto the adapter card.

Be careful not to put stress on the LEDs on the adapter card.