Nvidia Mellanox User manual

Exported onOct/23/2020 12:05 PM
https://docs.mellanox.com/x/TwvTAQ
NVIDIA Mellanox ConnectX-6 InfiniBand/VPI
Adapter Cards for OCP Spec 3.0 User Manual

2
Table of Contents
Table of Contents .......................................................................................................2
Introduction................................................................................................................8
Products Overview................................................................................................................... 8
Features and Benefits............................................................................................................. 9
Operating Systems/Distributions ......................................................................................... 11
Connectivity ........................................................................................................................... 11
Interfaces .................................................................................................................12
InfiniBand Interface............................................................................................................... 12
Ethernet QSFP56 Interfaces ................................................................................................. 12
PCI Express Interface ........................................................................................................... 12
LED Interface......................................................................................................................... 12
SMBus Interface.................................................................................................................... 13
Voltage Regulators................................................................................................................ 13
CPLDInterface ...................................................................................................................... 14
Hardware Installation ..............................................................................................15
Safety Warnings..................................................................................................................... 15
Installation Procedure Overview........................................................................................... 15
System Requirements........................................................................................................... 15
Hardware Requirements ................................................................................................. 16
Airflow Requirements ...................................................................................................... 16
Software Requirements ................................................................................................... 16
Safety Precautions ................................................................................................................ 17
Pre-Installation Checklist..................................................................................................... 17
OCP 3.0 Bracket Replacement Instructions ........................................................................ 17
OCP 3.0 Adapter Card Installation Instructions................................................................... 17
Cards with Thumbscrew (Pull-tab) Brackets ................................................................ 18
Cards with Ejector Latch............................................................................................ 19
Cards with Internal Lock.............................................................................................20
Cables and Modules......................................................................................................... 20
Identifying the Card in Your System ................................................................................ 21
Adapter Cards Extraction Instructions................................................................................. 22
Safety Precautions ........................................................................................................... 17

3
Cards Extraction Instructions.......................................................................................... 23
Cards with Thumbscrew (Pull-tab) BracketExtraction Instructions........................23
Cards with Internal-Lock Bracket Extraction Instructions .......................................23
Cards with Ejector Latch Extraction Instructions......................................................23
Driver Installation ....................................................................................................25
Windows Driver Installation.................................................................................................. 25
Hardware and Software Requirements........................................................................... 25
Downloading Mellanox WinOF-2 Driver .......................................................................... 25
Installing Mellanox WinOF-2 Driver ................................................................................ 26
Attended Installation...................................................................................................26
Unattended Installation...............................................................................................30
Installation Results .......................................................................................................... 31
Extracting Files Without Running Installation ................................................................ 31
Uninstalling Mellanox WinOF-2 Driver............................................................................ 33
Attended Uninstallation ..............................................................................................33
Unattended Uninstallation..........................................................................................33
Firmware Upgrade ........................................................................................................... 33
Deploying the Driver on a Nano Server........................................................................... 33
Offline Installation.......................................................................................................33
Online Update.............................................................................................................. 34
Linux Driver Installation........................................................................................................ 35
Prerequisites .................................................................................................................... 35
Downloading Mellanox OFED .......................................................................................... 35
Installing Mellanox OFED ................................................................................................ 36
Installation Script ........................................................................................................ 36
Installation Procedure ................................................................................................ 37
Installation Results .....................................................................................................39
Installation Logs..........................................................................................................39
openibd Script..............................................................................................................39
Driver Load Upon System Boot ..................................................................................40
mlnxofedinstall Return Codes....................................................................................40
Uninstalling MLNX_OFED...........................................................................................41
Installing MLNX_OFED Using YUM ................................................................................. 41
Setting up MLNX_OFED YUM Repository...................................................................41

4
Installing MLNX_OFED Using the YUM Tool ..............................................................42
Uninstalling MLNX_OFED Using the YUM Tool .........................................................43
Installing MLNX_OFED Using apt-get Tool..................................................................... 43
Setting up MLNX_OFED apt-get Repository ..............................................................43
Installing MLNX_OFED Using the apt-get Tool..........................................................43
Uninstalling MLNX_OFED Using the apt-get Tool.....................................................44
Updating Firmware After Installation.............................................................................. 44
Updating the Device Online.........................................................................................44
Updating the Device Manually ....................................................................................45
Updating the Device Firmware Automatically upon System Boot ............................45
UEFI Secure Boot............................................................................................................. 46
Enrolling Mellanox's x.509 Public Key on Your Systems...........................................46
Removing Signature from kernel Modules ................................................................46
Performance Tuning ........................................................................................................ 47
VMware Driver Installation ................................................................................................... 47
Hardware and Software Requirements........................................................................... 47
Installing Mellanox NATIVE ESXi Driver for VMware vSphere........................................ 48
Removing Earlier Mellanox Drivers................................................................................. 48
Firmware Programming .................................................................................................. 49
Updating Adapter Firmware ....................................................................................50
Troubleshooting .......................................................................................................51
GeneralTroubleshooting ...................................................................................................... 51
LinuxTroubleshooting .......................................................................................................... 51
WindowsTroubleshooting..................................................................................................... 52
Specifications ...........................................................................................................53
MCX653435A-HDAISpecifications........................................................................................ 53
MCX653436A-HDAISpecifications........................................................................................ 54
MCX653435A-EDAISpecifications ........................................................................................ 56
MCX653435A-HDAE Specifications....................................................................................... 57
Board Mechanical Drawing and Dimensions....................................................................... 58
Brackets Mechanical Drawings and Dimensions............................................................... 59
Cards with Ejector Latch Bracket .................................................................................. 59
Cards with Internal Lock Bracket ................................................................................... 60
Finding the GUID/MAC and Serial Number on the Adapter Card.............................61

5
MCX653435A-HDAIBoard Labels (Example) - TBD ............................................................ 61
MCX653436A-HDAIBoard Labels (Example)- TBD ............................................................ 61
MCX653435A-EDAIBoard Labels (Example)- TBD............................................................. 61
MCX653435A-HDAEBoard Labels (Example)- TBD ........................................................... 61
Document Revision History ......................................................................................62

6
•
•
•
About This Manual
This User Manual describes NVIDIA® Mellanox® ConnectX®-6 VPI adapter cards for Open Compute
Project (OCP), Spec 3.0. It provides details as to the interfaces of the board, specifications, required
software and firmware for operating the board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 VPI adapter
cards for OCP Spec 3.0.
OPN Marketing Description
MCX653435A-
HDAI
ConnectX®-6 VPI adapter card, 200Gb/s (HDR IB and 200GbE) for OCP 3.0, with host
management, Single-port QSFP56, PCIe4.0 x16, Internal Lock
MCX653436A-
HDAI
ConnectX®-6 VPI adapter card, 200Gb/s (HDR IB and 200GbE) for OCP 3.0, with host
management, Dual-port QSFP56, PCIe4.0 x16, Internal Lock
MCX653435A-
EDAI
ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and 100GbE) for OCP 3.0, with host
management, Single-port QSFP56, PCIe 3.0/4.0 x16, Internal Lock
MCX653435A-
HDAE
ConnectX®-6 VPI adapter card, 200Gb/s (HDR IB and 200GbE) for OCP 3.0, with host
management, Single-port QSFP56, PCIe4.0 x16, Ejector Latch
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic familiarity
with InfiniBand/VPI network and architecture specifications.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact usthrough
the following methods:
URL:http://www.mellanox.com> Support
E-mail:[email protected]
Tel: [phone]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails
regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seekassistance through their reseller.
Related Documentation
Mellanox OFED for Linux
User Manual and Release
Notes
User Manual describing OFED features, performance, band diagnostic, tools
content and configuration. SeeMellanox OFED for Linux Documentation.
WinOF-2 for WindowsUser
Manual and Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic,
tools content and configuration. SeeWinOF-2 for Windows Documentation.
Mellanox VMware for
Ethernet User Manual and
Release Notes
User Manual describing the various components of the Mellanox ConnectX®
NATIVE ESXi stack. See http://www.mellanox.comProducts > Software >
Ethernet Drivers > VMware Driver

7
Mellanox Firmware Utility
(mlxup) User Manual and
Release Notes
Mellanox firmware update and query utility used to update the firmware. Seeh
ttp://www.mellanox.com> Products > Software > Firmware Tools >mlxup
Firmware Utility
Mellanox Firmware Tools
(MFT) User Manual
User Manual describing the set of MFT firmware management tools for a
single node. SeeMFT User Manual.
IEEE Std 802.3
Specification
IEEE Ethernet specification at http://standards.ieee.org/
PCI Express 3.0/4.0
Specifications
Industry Standard PCI Express Base and Card Electromechanical
Specifications at https://pcisig.com/specifications
Open Compute Project 3.0
Specification
https://www.opencompute.org/
Mellanox LinkX
Interconnect Solutions
Mellanox LinkX InfiniBand cables and transceivers are designed to maximize
the performance of High-Performance Computing networks, requiring high-
bandwidth, low-latency connections between compute nodes and switch
nodes. Mellanox offers one of industry’s broadest portfolio of QDR/FDR10
(40Gb/s), FDR (56Gb/s), EDR/HDR100 (100Gb/s) and HDR (200Gb/s) cables,
including Direct Attach Copper cables (DACs), copper splitter cables, Active
Optical Cables (AOCs) and transceivers in a wide range of lengths from 0.5m
to 10km. In addition to meeting IBTA standards,Mellanox tests every product
in an end-to-end environment ensuring a Bit Error Rate of less than 1E-15.Re
ad more at https://www.mellanox.com/products/interconnect/infiniband-
overview.php
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.

8
Introduction
This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the ConnectX®-6
VPI integrated circuit device for OCP Spec 3.0. These adapters connectivity provide the highest
performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers
used in Enterprise Data Centers and High-Performance Computing environments.
The following provides the ordering part number, port speed, number of ports, and PCI Express speed.
Important Note:
ConnectX-6 OCP 3.0 cards were tested for Shock & Vibe in accordance with Mellanox specifications
and setups as defined in document XXX, as the OCP spec3.0 available at that time did not contain any
S&V definitions. A newer version of the OCP spec 3.0 has defined S&V specifications and Mellanox is in
the midst of retesting these cards to comply with OCP spec 3.0.
Products Overview
Model ConnectX-6 VPI Cards for OCP Spec 3.0
Part Number MCX653435A-HDAI MCX653436A-HDAI MCX653435A-HDAE(a) MCX653435A-EDAI
Host
Management
√ √ √ √
Network
Connector
Type
Single-port QSFP56 Dual-port QSFP56 Single-port QSFP56 Single-port QSFP56
Data
Rate
InfiniB
and
SDR/DDR/QDR/FDR/EDR/HDR100/HDR SDR/DDR/QDR/FDR/EDR/
HDR100
Ethern
et
10/25/40/50/100/200 Gb/s 10/25/40/50/100 Gb/s
PCI Express
Connectors
PCIe Gen 3.0 / 4.0 x16
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions Small Form Factor (SFF) OCP 3.0
4.52 in. x 2.99 in (115.00mm x 76.00mm)
Retention
Mechanism
Internal Lock Ejector Latch Internal Lock
RoHS RoHS Compliant
IC Part
Number
MT28908A0-XCCF-
HVM
Device ID 4123 for Physical Function (PF)
4124 for Virtual Function (VF)
a.MCX653435A-HDAE - The Gen 4.0 card has been tested and certified with PCIe 3.0 servers. PCIe 4.0
interface will be tested when servers with Gen 4.0 support become available.
For more detailed information seeSpecifications.

9
•
•
Features and Benefits
Feature Description
PCI Express (PCIe) Uses PCIe Gen 3.0 (8GT/s) or Gen 4.0 (16GT/s) through x16 edge connector.
200Gb/s Virtual Protocol
Interconnect (VPI) Adapter
ConnectX-6 offers the highest throughput VPI adapter, supporting HDR 200Gb/s
InfiniBand and 200Gb/s Ethernet and enabling any standard networking,
clustering, or storage to operate seamlessly over any converged network
leveraging a consolidated software stack.
InfiniBand Architecture
Specification v1.3
compliant
ConnectX-6 delivers low latency, high bandwidth, and computing efficiency for
performance-driven server and storage clustering applications. ConnectX-6 is
InfiniBand Architecture Specification v1.3 compliant.
InfiniBand HDR A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of
53.125Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of 200Gb/
s.
InfiniBand HDR100 A standard InfiniBand data rate, where each lane of a 2X port runs a bit rate of
53.125Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of 100Gb/
s.
Up to 200 Gigabit Ethernet Mellanox adapters comply with the following IEEE 802.3 standards:
• 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC
modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
InfiniBand EDR A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of
25.78125Gb/s with a 64b/66b encoding, resulting in an effective bandwidth of
100Gb/s.
Memory Components EEPROM - The EEPROM capacity is 32Kbit
SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G
device by Macronix)
This section describes hardware features and capabilities. Please refer to the relevant driver
and/or firmware release notes for feature availability.


10
Feature Description
Overlay Networks In order to better scale their networks, data center operators often create overlay
networks that carry traffic from individual virtual machines over logical tunnels in
encapsulated formats such as NVGRE and VXLAN. While this solves network
scalability issues, it hides the TCP packet from the hardware offloading engines,
placing higher loads on the host CPU. ConnectX-6 effectively addresses this by
providing advanced NVGRE and VXLAN hardware offloading engines that
encapsulate and de-capsulate the overlay protocol.
RDMA and RDMA
overConverged Ethernet
(RoCE)
ConnectX-6, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA
over Converged Ethernet) technology, delivers low-latency and high-performance
over Band and Ethernet networks. Leveraging data center bridging (DCB)
capabilities as well as ConnectX-6 advanced congestion control hardware
mechanisms, RoCE provides efficient low-latency RDMA services over Layer 2 and
Layer 3 networks.
Mellanox PeerDirect™ PeerDirect™ communication provides high-efficiency RDMA access by eliminating
unnecessary internal data copies between components on the PCIe bus (for
example, from GPU to CPU), and therefore significantly reduces application run
time. ConnectX-6 advanced acceleration technology enables higher cluster
efficiency and scalability to tens of thousands of nodes.
CPU Offload Adapter functionality enabling reduced CPU overhead allowing more available
CPU for computation tasks.
Open vSwitch (OVS) offload using ASAP2(TM)
• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation
Quality of Service (QoS) Support for port-based Quality of Service enabling various application
requirements for latency and SLA.
Hardware-based I/
OVirtualization
ConnectX-6 provides dedicated adapter resources and guaranteed isolation and
protection for virtual machines within the server.
Storage Acceleration A consolidated compute and storage network achieves significant cost-
performance advantages over multi-fabric networks. Standard block and file
access protocols can leverage RDMA for high-performance storage access.
• NVMe over Fabric offloads for the target machine
• Erasure Coding
• T10-DIF Signature Handover
SR-IOV ConnectX-6 SR-IOV technology provides dedicated adapter resources and
guaranteed isolation and protection for virtual machines (VM) within the server.
NC-SI The adapter supports a Network Controller Sideband Interface (NC-SI), MCTP over
SMBus and MCTP over PCIe - Baseboard Management Controller interface.
High-
PerformanceAcceleration
s
• Tag Matching and Rendezvous Offloads
• Adaptive Routing on Reliable Transport
• Burst Buffer Offloads for Background Checkpointing
Wake-on-LAN (WoL) The adapter supported Wake-on-LAN (WoL), a computer networkingstandard that
allows an adapter to be turned on orawakenedby a network message.
TBD: In STBY mode, only port0 is available.
Reset-on-LAN (RoL) Supported