Nvidia MCX545B-CCUN User manual

Exported onDec/16/2020 07:33 PM
https://docs.mellanox.com/x/V4eO
NVIDIA Mellanox ConnectX-5 Ethernet Adapter
Cards for OCP Spec 2.0 User Manual
25 and 100 Gb/s Ethernet Adapter Cards; Intelligent RDMA (RoCE) enabled NICs. Supporting OCP
Specification 2.0 type 1 and type 2.

2
Table of Contents
Introduction................................................................................................................8
OCP Spec 2.0 Stacking Heights .............................................................................................. 9
OCP Spec 2.0 Type 1 Stacking Height - Single-port Card ............................................... 9
OCP Spec 2.0 Type 1 Stacking Height - Dual-port Card................................................. 10
OCP Spec 2.0 Type 2 Stacking Height - Single-port Card .............................................. 10
OCP Spec 2.0 Type 2 Stacking Height - Dual-port Card................................................. 10
Features and Benefits........................................................................................................... 11
Operating Systems/Distributions ......................................................................................... 12
Connectivity ........................................................................................................................... 13
Interfaces .................................................................................................................14
Ethernet SFP28 and QSFP28 Interfaces .............................................................................. 14
PCI Express Interface ........................................................................................................... 14
LED Interface......................................................................................................................... 14
SMBus Interface.................................................................................................................... 15
Voltage Regulators................................................................................................................ 15
Thermal Sensors................................................................................................................... 15
Hardware Installation ..............................................................................................16
Safety Warnings..................................................................................................................... 16
Installation Procedure Overview........................................................................................... 16
System Requirements........................................................................................................... 16
Hardware Requirements ................................................................................................. 17
Airflow Requirements .................................................................................................17
Software Requirements ................................................................................................... 17
Unpacking the Product ......................................................................................................... 17
Adapter Cards Installation Instructions............................................................................... 18
Cables and Modules.............................................................................................................. 19
Identifying the Card in Your System ..................................................................................... 20
Adapter Cards Extraction Instructions................................................................................. 21
Safety Precautions ........................................................................................................... 21
Card Extraction................................................................................................................. 22
Driver Installation ....................................................................................................24
Windows Driver Installation.................................................................................................. 24

3
Software Requirements ................................................................................................... 24
Downloading Mellanox WinOF-2 Driver .......................................................................... 24
Installing Mellanox WinOF-2 Driver ............................................................................... 25
Attended Installation...................................................................................................25
Unattended Installation...............................................................................................30
Installation Results .....................................................................................................31
Uninstalling Mellanox WinOF-2 Driver............................................................................ 31
Attended Uninstallation ..............................................................................................31
Unattended Uninstallation..........................................................................................31
Extracting Files Without Running Installation ................................................................ 32
Firmware Upgrade ........................................................................................................... 34
Linux Driver Installation........................................................................................................ 34
Prerequisites .................................................................................................................... 34
Downloading Mellanox OFED .......................................................................................... 35
Installing Mellanox OFED ................................................................................................ 35
Installation Script ........................................................................................................ 35
Installation Procedure ................................................................................................ 37
Installation Results .....................................................................................................39
Installation Logs..........................................................................................................39
openibd Script..............................................................................................................39
Driver Load Upon System Boot ..................................................................................40
mlnxofedinstall Return Codes.................................................................................... 40
Uninstalling MLNX_OFED...........................................................................................40
Installing MLNX_OFED Using YUM ................................................................................. 40
Setting up MLNX_OFED YUM Repository...................................................................41
Installing MLNX_OFED Using the YUM Tool ..............................................................41
Uninstalling MLNX_OFED Using the YUM Tool .........................................................43
Installing MLNX_OFED Using apt-get Tool..................................................................... 43
Setting up MLNX_OFED apt-get Repository ..............................................................43
Installing MLNX_OFED Using the apt-get Tool..........................................................43
Uninstalling MLNX_OFED Using the apt-get Tool.....................................................44
Updating Firmware After Installation.............................................................................. 44
Updating the Device Online.........................................................................................44
Updating the Device Manually ....................................................................................44

4
Updating the Device Firmware Automatically upon System Boot ............................45
UEFI Secure Boot............................................................................................................. 45
Enrolling Mellanox's x.509 Public Key on Your Systems...........................................45
Removing Signature from kernel Modules ................................................................ 46
Performance Tuning ........................................................................................................ 47
VMware Driver Installation ................................................................................................... 47
Hardware and Software Requirements........................................................................... 47
Installing Mellanox NATIVE ESXi Driver for VMware vSphere........................................ 47
Removing Earlier Mellanox Drivers................................................................................. 48
Firmware Programming .................................................................................................. 48
Updating Adapter Firmware ....................................................................................49
Troubleshooting .......................................................................................................50
GeneralTroubleshooting ...................................................................................................... 50
LinuxTroubleshooting .......................................................................................................... 50
WindowsTroubleshooting..................................................................................................... 51
Specifications ...........................................................................................................52
MCX542A-ACANSpecifications ............................................................................................ 52
MCX542B-AC[A/U]NSpecifications...................................................................................... 53
MCX545B-GCUNSpecifications............................................................................................ 54
MCX545B-CCUNSpecifications............................................................................................ 55
MCX545A-CCAN andMCX545A-CCUN Specifications......................................................... 56
MCX546A-BCANSpecifications ............................................................................................ 57
MCX546A-CDANSpecifications ............................................................................................ 58
Board Mechanical Drawing and Dimensions....................................................................... 59
Finding the MAC and Serial Number on the Adapter Card ......................................63
Document Revision History ......................................................................................64

5
About This Manual
This User Manual describes NVIDIA® Mellanox® ConnectX®-5 and ConnectX®-5 Ex Ethernet adapter
cards for Open Compute Project (OCP), Spec 2.0. It provides details as to the interfaces of the board,
specifications, required software and firmware for operating the board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-5 Ex and
ConnectX-5 Ethernet adapter cards for OCP Spec 2.0.
IC in Use OPN Marketing Description
ConnectX®-5
MCX542A-ACAN ConnectX®-5 EN network interface card for
OCP2.0, Type 1, with host management,
25GbE dual-port SFP28, PCIe3.0 x16,
Halogen-free, no bracket
MCX542B-ACAN ConnectX®-5 EN network interface card for
OCP2.0,Type 1, with host management,
25GbE dual-port SFP28, PCIe3.0 x8, Halogen-
free, no bracket
MCX542B-ACUN ConnectX®-5 EN network interface card for
OCP2.0, Type 1, with host management,
25GbE dual-port SFP28, PCIe3.0 x8, UEFI
Enabled (x86/ARM), Halogen free, no bracket
MCX545B-GCUN ConnectX®-5 EN network interface card for
OCP2.0, Type 1, with hostmanagement,
50GbE,single-port QSFP28,PCIe3.0x16,UEFI
Enabled, no bracket
MCX545B-CCUN ConnectX®-5 EN network interface card for
OCP2.0, Type 1, with host management,
100GbE, single-port QSFP28, PCIe3.0 x16,
UEFI Enabled, no bracket
MCX545A-CCAN ConnectX®-5 EN network interface card for
OCP2.0, Type 2, with host management,
100GbE, single-port QSFP28, PCIe3.0 x16, no
bracket
MCX545A-CCUN ConnectX®-5 EN network interface card for
OCP2.0, Type 2, with host management,
100GbE, single-port QSFP28, PCIe3.0 x16,
UEFI Enabled, no bracket
ConnectX®-5 Ex
MCX546A-BCAN ConnectX®-5 Ex EN network interface card
for OCP2.0, Type 2, 40GbE dual-port QSFP28,
PCIe3.0 x16, no bracket
MCX546A-CDAN ConnectX®-5 Ex network interface card for
OCP2.0, Type 2, with host management,
100GbE dual-port QSFP28, PCIe4.0 x16, no
bracket
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic familiarity
with Ethernet network and architecture specifications.
Technical Support

6
•
•
•
Customers who purchased Mellanox products directly from Mellanox are invited to contact usthrough
the following methods:
URL:http://www.mellanox.com> Support
E-mail:[email protected]
Tel: [phone]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails
regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seekassistance through their reseller.
Related Documentation
Mellanox OFED for
Linux User Manual and
Release Notes
User Manual describing OFED features, performance, band diagnostic, tools
content and configuration. SeeMellanox OFED for Linux Documentation.
WinOF-2 for
WindowsUser Manual
and Release Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic,
tools content and configuration. SeeWinOF-2 for Windows Documentation.
Mellanox VMware for
Ethernet User Manual
and Release Notes
User Manual describing the various components of the Mellanox ConnectX®
NATIVE ESXi stack. See http://www.mellanox.comProducts > Software > Ethernet
Drivers > VMware Driver
Mellanox Firmware
Utility (mlxup) User
Manual and Release
Notes
Mellanox firmware update and query utility used to update the firmware. Seehttp:
//www.mellanox.com> Products > Software > Firmware Tools >mlxup Firmware
Utility
Mellanox Firmware
Tools (MFT) User
Manual
User Manual describing the set of MFT firmware management tools for a single
node. SeeMFT User Manual.
IEEE Std 802.3
Specification
IEEE Ethernet specification at http://standards.ieee.org/
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications at
https://pcisig.com/specifications
Mellanox LinkX
Interconnect Solutions
Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the
performance of High-Performance Computing networks, requiring high-
bandwidth, low-latency connections between compute nodes and switch nodes.
Mellanox offers one of industry’s broadest portfolio of QDR/FDR10 (40Gb/s), FDR
(56Gb/s), EDR/HDR100 (100Gb/s) and HDR (200Gb/s) cables, including Direct
Attach Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs)
and transceivers in a wide range of lengths from 0.5m to 10km. In addition to
meeting IBTA standards,Mellanox tests every product in an end-to-end
environment ensuring a Bit Error Rate of less than 1E-15.Read more at https://
www.mellanox.com/products/interconnect/infiniband-overview.php
Open Compute Project
2.0 Specification
https://www.opencompute.org/
Document Conventions

7
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to
mean PCI Express.

8
Introduction
This is the User Guide for Ethernet adapter cards based on the ConnectX®-5 integrated circuit device
for OCP Spec 2.0. These adapters connectivity provide the highest performing low latency and most
flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in Enterprise Data Centers and
High-Performance Computing environments.
The following provides the ordering part number, port speed, number of ports, and PCI Express speed.
ConnectX-5 Ex Ethernet Adapter Cards
Model ConnectX-5 Ex Cards for OCP Spec 2.0
Part Number MCX546A-BCAN MCX546A-CDAN
Ethernet Data Rate 10/25/40 Gb/s 10/25/40/50/100 Gb/s
Host Management - √
Network Connector Type Dual-port QSFP28 Dual-port QSFP28
OCP Spec 2.0 Type Type 2 Stacking Height
PCI Express Connectors PCIe Gen 3.0 x16 (a), (b)
SERDES @ 8.0GT/s
PCIe Gen 3.0 / 4.0 x16 (a), (b)
SERDES @ 8.0GT/s / 16.0GT/s
Dimensions 3.07 in. x 4.33 in (78.00mm x 110.05mm)
RoHS RoHS Compliant
IC Part Number MT28808A0-FCCF-EV
Device ID 4121 for Physical Function (PF)
4122 for Virtual Function (VF)
a. PCIe 4.0 x16 bus can supply a maximum bandwidth of 256Gb/s (=16 *16GT/s, including overhead),
and can support 200Gb/s when both network ports of the card run at 100Gb/s.
b.TheGen4.0 cards have been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be
tested when servers with Gen 4.0 support become available.
ConnectX-5 Ethernet Adapter Cards
Model ConnectX-5 Cards for OCP Spec 2.0
Part
Number
MCX542A-
ACAN
MCX542B-
ACAN(a)
MCX542B-
ACUN
MCX545B-
GCUN
MCX545A-
CCAN
MCX545A-
CCUN
MCX545B-
CCUN
Network
Connecto
r Type
Dual-port
SFP28
Dual-port
SFP28
Dual-port
SFP28
Single-port
QSFP28
Single-port
QSFP28
Single-port
QSFP28
Single-port
QSFP28
OCP Spec
2.0 Type
Type 1
Stacking
Height
Type 1
Stacking
Height
Type 1
Stacking
Height
Type 1
Stacking
Height
Type 2
Stacking
Height
Type 2
Stacking
Height
Type 1
Stacking
Height

9
Model ConnectX-5 Cards for OCP Spec 2.0
Ethernet
Data Rate
10/25 Gb/s 10/25/40/50
Gb/s
10/25/40/50/
100 Gb/s
10/25/40/50/
100 Gb/s
10/25/40/50/
100 Gb/s
PCI
Express
Connecto
rs
PCIe Gen 3.0
x16
SERDES @
8.0GT/s
PCIe Gen 3.0
x8
SERDES @
8.0GT/s
PCIe Gen 3.0
x8
SERDES @
8.0GT/s
PCIe Gen 3.0
x16
SERDES @
8.0GT/s
PCIe Gen 3.0
x16
SERDES @
8.0GT/s
PCIe Gen 3.0
x16
SERDES @
8.0GT/s
PCIe Gen 3.0
x16
SERDES @
8.0GT/s
Halogen
Free
√√√- - --
Host
Managem
ent
√√√√√√√
UEFI - - √ √ -√ √
Dimensio
ns
3.07 in. x
4.33 in
(78.00mm x
110.05mm)
2.67 in. x
4.33 in
(68.00mm x
110.05mm)
2.67 in. x
4.33 in
(68.00mm x
110.05mm)
2.67 in. x
4.33 in
(68.00mm x
110.05mm)
3.07 in. x
4.33 in
(78.00mm x
110.05mm)
3.07 in. x
4.33 in
(78.00mm x
110.05mm)
3.07 in. x
4.33 in
(78.00mm x
110.05mm)
RoHS RoHS Compliant
IC Part
Number
MT28808A0-FCCF-EV
Device ID 4119 for Physical Function (PF)
4120 for Virtual Function (VF)
a.Mellanox recommends populating MCX542B-ACAN in a standard PCIe x8 OCP connector which
exposes PCIe lanes in a straight manner.
 In case the OCP slot exposes PCIe lanes in a reversed manner, MCX542B-ACAN supports automatic
lane reversal withFW image from April 2019 release and above.
For more detailed information seeSpecifications.
OCP Spec 2.0 Stacking Heights
OCP Spec 2.0 Type 1 Stacking Height - Single-port Card
The single port 100Gb/s Ethernet adapter card complies with OCP Spec 2.0 Type 1 with 8mm stacking
height.
This section applies to MCX545B-CCUN andMCX545B-GCUN.


10
OCP Spec 2.0 Type 1 Stacking Height - Dual-port Card
The dual port 10/25Gb/s Ethernet adapter card comply with OCP Spec 2.0 Type 1 with 8mm stacking
height.
OCP Spec 2.0 Type 2 Stacking Height - Single-port Card
The single-port 100Gb/s adapter card follows OCP Spec 2.0 Type 2 with 12mm stacking height.
OCP Spec 2.0 Type 2 Stacking Height - Dual-port Card
The dual-port 100Gb/s adapter card follows OCP Spec 2.0 Type 2 with 12mm stacking height.
This section applies to MCX542A-ACAN and MCX542B-ACAN.

This section applies to MCX545A-CCAN and MCX545A-CCUN.

This section applies to MCX546A-CDAN and MCX546A-BCAN.
