Nvidia Mellanox ConnectX-6 MCX651105A-EDAT User manual

Exported on Nov/09/2023 12:30 PM
https://docs.nvidia.com/networking/x/zYWmBQ







NVIDIA ConnectX-6 InfiniBand/Ethernet Adapter Cards User
Manual



2
Table of Contents
Introduction........................................................................................................................................... 11
Product Overview ...............................................................................................................................................11
ConnectX-6 PCIe x8 Card .................................................................................................................................. 12
ConnectX-6 PCIe x16 Card ................................................................................................................................. 14
ConnectX-6 DE PCIe x16 Card ............................................................................................................................. 15
ConnectX-6 for Liquid-Cooled Intel® Server System D50TNP Platforms............................................................................ 16
ConnectX-6 Socket Direct™ Cards ........................................................................................................................ 17
ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe x16) ......................................................................................... 17
ConnectX-6 Single-slot Socket Direct Cards (2x PCIe x8 in a row) .............................................................................. 19
Package Contents ...............................................................................................................................................20
ConnectX-6 PCIe x8/x16 Adapter Cards ................................................................................................................. 20
ConnectX-6 PCIe x16 Adapter Card for liquid-cooled Intel® Server System D50TNP Platforms ................................................ 21
ConnectX-6 Socket Direct Cards (2x PCIe x16) ......................................................................................................... 21
Features and Benefits ..........................................................................................................................................22
Operating Systems/Distributions .............................................................................................................................25
Connectivity .....................................................................................................................................................25
Manageability....................................................................................................................................................26
Interfaces ............................................................................................................................................. 27
InfiniBand Interface ............................................................................................................................................27

3
Ethernet Interfaces.............................................................................................................................................27
PCI Express Interface...........................................................................................................................................27
LED Interface ....................................................................................................................................................28
Heatsink Interface ..............................................................................................................................................30
SMBus Interface .................................................................................................................................................30
Voltage Regulators ..............................................................................................................................................30
Hardware Installation ............................................................................................................................... 31
Safety Warnings .................................................................................................................................................31
Installation Procedure Overview..............................................................................................................................32
System Requirements ..........................................................................................................................................33
Hardware Requirements ................................................................................................................................... 33
Airflow Requirements ..........................................................................................................................................34
Software Requirements .................................................................................................................................... 35
Safety Precautions ..............................................................................................................................................35
Pre-Installation Checklist ......................................................................................................................................36
Bracket Replacement Instructions ...........................................................................................................................36
Installation Instructions........................................................................................................................................37
Cables and Modules.........................................................................................................................................38
Identifying the Card in Your System..........................................................................................................................39
PCIe x8/16 Cards Installation Instructions ..................................................................................................................42
Installing the Card .......................................................................................................................................... 42

4
Uninstalling the Card.......................................................................................................................................44
Socket Direct (2x PCIe x16) Cards Installation Instructions ..............................................................................................46
Installing the Card .......................................................................................................................................... 47
Uninstalling the Card.......................................................................................................................................57
Cards for Intel Liquid-Cooled Platforms Installation Instructions .......................................................................................59
Installing the Card .......................................................................................................................................... 59
Driver Installation.................................................................................................................................... 65
Linux Driver Installation .......................................................................................................................................65
Prerequisites................................................................................................................................................. 65
Downloading NVIDIA OFED ................................................................................................................................. 66
Installing MLNX_OFED ...................................................................................................................................... 67
Installation Script ....................................................................................................................................... 67
Installation Procedure.................................................................................................................................. 70
Installation Results ..................................................................................................................................... 73
Installation Logging..................................................................................................................................... 73
Driver Load Upon System Boot............................................................................................................................ 74
mlnxofedinstall Return Codes......................................................................................................................... 74
Installation Logging..................................................................................................................................... 75
Uninstalling MLNX_OFED ............................................................................................................................... 75
Additional Installation Procedures ....................................................................................................................... 76
Installing MLNX_OFED Using YUM..................................................................................................................... 76

5
Installing MLNX_OFED Using apt-get ................................................................................................................. 82
Performance Tuning ........................................................................................................................................ 86
Windows Driver Installation ...................................................................................................................................86
Software Requirements .................................................................................................................................... 87
Downloading WinOF-2 Driver.............................................................................................................................. 87
Installing WinOF-2 Driver .................................................................................................................................. 88
Attended Installation................................................................................................................................... 88
Unattended Installation................................................................................................................................ 98
Firmware Upgrade ........................................................................................................................................ 100
VMware Driver Installation .................................................................................................................................. 100
Software Requirements .................................................................................................................................. 100
Installing NATIVE ESXi Driver for VMware vSphere ................................................................................................... 101
Removing Earlier NVIDIA Drivers ........................................................................................................................ 101
Firmware Programming .................................................................................................................................. 102
Troubleshooting .....................................................................................................................................103
General Troubleshooting..................................................................................................................................... 103
Linux Troubleshooting ........................................................................................................................................ 104
Windows Troubleshooting.................................................................................................................................... 105
Updating Adapter Firmware .......................................................................................................................106
Monitoring............................................................................................................................................108
Specifications........................................................................................................................................109

6
MCX651105A-EDAT Specifications........................................................................................................................... 109
MCX653105A-HDAT Specifications .......................................................................................................................... 111
MCX653106A-HDAT Specifications .......................................................................................................................... 113
MCX653105A-HDAL Specifications .......................................................................................................................... 115
MCX653106A-HDAL Specifications .......................................................................................................................... 117
MCX653105A-ECAT Specifications........................................................................................................................... 119
MCX653106A-ECAT Specifications........................................................................................................................... 121
MCX654105A-HCAT Specifications .......................................................................................................................... 123
MCX654106A-HCAT Specifications .......................................................................................................................... 125
MCX654106A-ECAT Specifications........................................................................................................................... 128
MCX653105A-EFAT Specifications ........................................................................................................................... 130
MCX653106A-EFAT Specifications ........................................................................................................................... 132
MCX683105AN-HDAT Specifications ........................................................................................................................ 134
Adapter Card and Bracket Mechanical Drawings and Dimensions ..................................................................................... 136
Adapter Cards ............................................................................................................................................. 136
Brackets Dimensions...................................................................................................................................... 138
PCI Express Pinouts Description for Single-Slot Socket Direct Card .........................................................................140
Finding the GUID/MAC on the Adapter Card ....................................................................................................141
Document Revision History ........................................................................................................................143



7
About This Manual
This User Manual describes NVIDIA® ConnectX®-6 InfiniBand/Ethernet adapter cards. It provides details as to the interfaces of the board, specifications,
required software and firmware for operating the board, and relevant documentation.
Ordering Part Numbers
The table below provides the ordering part numbers (OPN) for the available ConnectX-6 InfiniBand/Ethernet adapter cards.
NVIDIA SKU Legacy OPN Marketing Description Lifecycle
900-9X603-0016-DT0 MCX653105A-EFAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE), single-port QSFP56,
PCIe3.0/4.0 Socket Direct 2x8 in a row, tall bracket
Mass Production
900-9X603-0056-DT0 MCX653106A-EFAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IBand100GbE), dual-port QSFP56, PCIe3.0/4.0
Socket Direct 2x8 in a row, tall bracket
Mass Production
900-9X628-0016-ST0 MCX651105A-EDAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE, single-port QSFP56, PCIe4.0 x8,
tall bracket
Mass Production
900-9X6AF-0016-ST1 MCX653105A-ECAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE), single-port QSFP56,
PCIe3.0/4.0 x16, tall bracket
Mass Production
900-9X6AF-0056-MT1 MCX653106A-ECAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR IB and 100GbE), dual-port QSFP56, PCIe3.0/4.0
x16, tall bracket
Mass Production
900-9X6AF-0018-MT2 MCX653105A-HDAT ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, single-port QSFP56, PCIe3.0/4.0 x16,
tall bracket
Mass Production
900-9X6AF-0058-ST1 MCX653106A-HDAT ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, dual-port QSFP56, PCIe3.0/4.0 x16,
tall bracket
Mass Production
900-9X6B4-0058-DT0 MCX654106A-HCAT ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, dual-port QSFP56, Socket Direct 2x
PCIe3.0/4.0x16, tall bracket
Mass Production

8
NVIDIA SKU Legacy OPN Marketing Description Lifecycle
900-9X6AF-0018-SS0 MCX653105A-HDAL ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, single-port QSFP56, PCIe3.0/4.0 x16,
cold plate for liquid-cooledIntel® Server System D50TNP
platforms, tall bracket, ROHS R6
Mass Production
900-9X6AF-0058-SS0 MCX653106A-HDAL ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, dual-port QSFP56, PCIe3.0/4.0 x16,
cold plate for liquid-cooledIntel® Server System D50TNP
platforms, tall bracket, ROHS R6
Mass Production
900-9X0BC-001H-ST1 MCX683105AN-HDAT ConnectX®-6 DE adapter card, HDR IB (200Gb/s), single-port
QSFP, PCIe4.0 x16,no crypto, tall bracket
Mass Production
EOL'ed (End of Life) Ordering Part Numbers
NVIDIA SKU Legacy OPN Marketing Description Lifecycle
900-9X6B4-0056-DT0 MCX654106A-ECAT ConnectX®-6 InfiniBand/Ethernet adapter card, 100Gb/s
(HDR100, EDR InfiniBand and 100GbE), dual-port QSFP56,
Socket Direct 2x PCIe 3.0/4.0 x16, tall bracket
End of Life
900-9X6B4-0018-DT2 MCX654105A-HCAT ConnectX®-6 InfiniBand/Ethernet adapter card, HDR IB
(200Gb/s) and 200GbE, single-port QSFP56, Socket Direct 2x
PCIe3.0/4.0x16, tall bracket
End of Life
Intended Audience
This manual is intended for the installer and user of these cards.
The manual assumes basic familiarity with InfiniBand and Ethernet network and architecture specifications.
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough the following methods:

9
•
•
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchased NVIDIA M-1 Global Support Services, please see your contract fordetails regarding Technical Support.
Customers who purchased NVIDIA products through a NVIDIA approved reseller should first seekassistance through their reseller.
Related Documentation
MLNX_OFED for Linux User Manual and Release Notes User Manual describing OFED features, performance, band diagnostic, tools content, and configuration.
SeeMLNX_OFED for Linux Documentation.
WinOF-2 for WindowsUser Manualand Release Notes User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools content, and configuration.
SeeWinOF-2 for Windows Documentation.
NVIDIA VMware for Ethernet User Manual User Manual and release notes describing the various components of the NVIDIA ConnectX® NATIVE ESXi stack.
SeeVMware® ESXi Drivers Documentation.
NVIDIA Firmware Utility (mlxup) User Manual and Release Notes NVIDIA firmware update and query utility used to update the firmware. Refer toFirmware Utility (mlxup)
Documentation.
NVIDIA Firmware Tools (MFT) User Manual User Manual describing the set of MFT firmware management tools for a single node. SeeMFT User Manual.
InfiniBand Architecture Specification Release 1.2.1, Vol 2 - Release 1.3 InfiniBand Specifications
IEEE Std 802.3 Specification IEEE Ethernet Specifications
PCI Express Specifications Industry Standard PCI Express Base and Card Electromechanical Specifications. Refer toPCI-SIG Specifications.
LinkX Interconnect Solutions LinkX InfiniBand cables and transceivers are designed to maximize the performance of High-Performance
Computing networks, requiring high-bandwidth, low-latency connections between compute nodes and switch
nodes.NVIDIA offers one of the industry’s broadest portfolio ofQDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/HDR100
(100Gb/s), HDR (200Gb/s)and NDR (400Gb/s) cables, includingDirect Attach Copper cables (DACs), copper splitter
cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths from 0.5m to 10km.In addition to
meeting IBTA standards, NVIDIAtests every product in an end-to-end environment ensuring a Bit Error Rate of less
than 1E-15.Read more atLinkX Cables and Transceivers.
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in MegaBytes. The use of Mb or Mbits (small b) indicates the size in
MegaBits. In this document, PCIe is used to mean PCI Express.
Revision History

10
A list of the changes made to this document are provided inDocument Revision History.

11
Introduction
Product Overview
This is the user guide for InfiniBand/Ethernet adapter cards based on the ConnectX-6 integrated circuit device. ConnectX-6 connectivity provides the
highest performing low latency and most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in enterprise datacenters and high-
performance computing environments.
ConnectX-6 Virtual Protocol Interconnect® adapter cards provide up to two ports of 200Gb/s for InfiniBand and Ethernet connectivity, sub-600ns latency
and 200 million messages per second, enabling the highest performance and most flexible solution for the most demanding High-Performance Computing
(HPC), storage, and datacenter applications.
ConnectX-6 is a groundbreaking addition to the NVIDIA ConnectX series of industry-leading adapter cards. In addition to all the existing innovative features
of past ConnectX versions, ConnectX-6 offers a number of enhancements that further improve the performance and scalability of datacenter
applications.In addition, specific PCIe stand-up cards are available with a cold plate for insertion into liquid-cooled Intel® Server System D50TNP
platforms.
ConnectX-6 is available in two form factors: low-profile stand-up PCIe and Open Compute Project (OCP) Spec 3.0 cards with QSFP connectors. Single-port,
HDR, stand-up PCIe adapters are available based on either ConnectX-6 or ConnectX-6 DE (ConnectX-6 Dx enhanced for HPC applications).
Configuration OPN Marketing Description
ConnectX-6 PCIe x8 Card MCX651105A-EDAT ConnectX-6 InfiniBand/Ethernet adapter card, 100Gb/s (HDR100, EDR IB and 100GbE, single-port QSFP56, PCIe4.0 x8,
tall bracket
ConnectX-6 PCIe x16 Card MCX653105A-
HDAT
ConnectX-6 InfiniBand/Ethernet adapter card, HDR IB (200Gb/s) and 200GbE, single-port QSFP56, PCIe4.0 x16, tall
bracket
MCX653106A-
HDAT
ConnectX-6 InfiniBand/Ethernet adapter card, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56, PCIe3.0/4.0 x16, tall
bracket
MCX653105A-ECAT ConnectX-6 InfiniBand/Ethernet adapter card, 100Gb/s (HDR100, EDR IB and 100GbE), single-port QSFP56,
PCIe3.0/4.0 x16, tall bracket
Make sure to use a PCIe slot that is capable of supplying the required power and airflow to the ConnectX-6 as stated in Specifications.


12
Configuration OPN Marketing Description
MCX653106A-ECAT ConnectX-6 InfiniBand/Ethernet adapter card, 100Gb/s (HDR100, EDR IB and 100GbE), dual-port QSFP56, PCIe3.0/4.0
x16, tall bracket
ConnectX-6 DE PCIe x16 Card MCX683105AN-
HDAT
ConnectX-6 DE InfiniBand adapter card, HDR, single-port QSFP, PCIe 3.0/4.0 x16, No Crypto, Tall Bracket
ConnectX-6 PCIe x16 Cards for liquid-cooled Intel®
Server System D50TNP platforms
MCX653105A-
HDAL
ConnectX-6 InfiniBand/Ethernet adapter card, HDR IB (200Gb/s) and 200GbE, single-port QSFP56, PCIe4.0 x16, cold
plate for liquid-cooled Intel® Server System D50TNP platforms, tall bracket, ROHS R6
MCX653106A-
HDAL
ConnectX-6 InfiniBand/Ethernet adapter card, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56, PCIe4.0 x16, cold
plate for liquid-cooled Intel® Server System D50TNP platforms, tall bracket, ROHS R6
ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe
x16)
MCX654105A-
HCAT
ConnectX-6 InfiniBand/Ethernet adapter card kit, HDR IB (200Gb/s) and 200GbE, single-port QSFP56, Socket Direct 2x
PCIe3.0 x16, tall brackets
MCX654106A-
HCAT
ConnectX-6 InfiniBand/Ethernet adapter card, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56, Socket Direct 2x
PCIe3.0/4.0x16, tall bracket
MCX654106A-ECAT ConnectX-6 InfiniBand/Ethernet adapter card, 100Gb/s (HDR100, EDR InfiniBand and 100GbE), dual-port QSFP56,
Socket Direct 2x PCIe3.0/4.0 x16, tall bracket
ConnectX-6 Single-slot Socket Direct Cards (2x PCIe
x8 in a row)
MCX653105A-EFAT ConnectX-6 InfiniBand/Ethernet adapter card, 100Gb/s (HDR100, EDR IB and 100GbE), single-port QSFP56,
PCIe3.0/4.0 Socket Direct 2x8 in a row, tall bracket
MCX653106A-EFAT ConnectX-6 InfiniBand/Ethernet adapter card, 100Gb/s (HDR100, EDR IBand100GbE), dual-port QSFP56, PCIe3.0/4.0
Socket Direct 2x8 in a row, tall bracket
ConnectX-6 PCIe x8 Card
ConnectX-6 with a single PCIe x8 slot can support a bandwidth of up to 100Gb/s in a PCIe Gen 4.0 slot.

13
Part Number MCX651105A-EDAT
Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm
Data Transmission Rate Ethernet: 10/25/40/50/100 Gb/s
InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100
Network Connector Type Single-port QSFP56
PCIe x8 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s
RoHS RoHS Compliant
Adapter IC Part Number MT28908A0-XCCF-HVM

14
ConnectX-6 PCIe x16 Card
ConnectX-6 with a single PCIe x16 slot can support a bandwidth of up to 100Gb/s in a PCIe Gen 3.0 slot, or up to 200Gb/s in a PCIe Gen 4.0 slot. This form-
factor is available also forIntel® Server System D50TNP Platforms where an Intel liquid-cooled cold plate is used for adapter cooling mechanism.
Part Number MCX653105A-ECAT MCX653106A-ECAT MCX653105A-HDAT MCX653106A-HDAT
Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm
Data Transmission Rate Ethernet: 10/25/40/50/100 Gb/s
InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100
Ethernet: 10/25/40/50/100/200 Gb/s
InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100, HDR
Network Connector Type Single-port QSFP56 Dual-port QSFP56 Single-port QSFP56 Dual-port QSFP56

15
Part Number MCX653105A-ECAT MCX653106A-ECAT MCX653105A-HDAT MCX653106A-HDAT
PCIe x16 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s
RoHS RoHS Compliant
Adapter IC Part Number MT28908A0-XCCF-HVM
ConnectX-6 DE PCIe x16 Card
ConnectX-6 DE (ConnectX-6 Dx enhanced for HPC applications) with a single PCIe x16 slot can support a bandwidth of up to 100Gb/s in a PCIe Gen 3.0 slot,
or up to 200Gb/s in a PCIe Gen 4.0 slot.
Part Number MCX683105AN-HDAT
Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm
Data Transmission Rate
InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100, HDR
Network Connector Type Single-port QSFP56
PCIe x16 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s
RoHS RoHS Compliant
Adapter IC Part Number MT28924A0-NCCF-VE

16
ConnectX-6 for Liquid-Cooled Intel® Server System D50TNP Platforms
The below cards are available with a cold plate for insertion into liquid-cooled Intel® Server System D50TNP platforms.
Part Number MCX653105A-HDAL MCX653106A-HDAL
Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm
Data Transmission Rate Ethernet: 10/25/40/50/100/200 Gb/s
InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100, HDR
Network Connector Type Single-port QSFP56 Dual-port QSFP56
PCIe x16 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s
RoHS RoHS Compliant

17
Part Number MCX653105A-HDAL MCX653106A-HDAL
Adapter IC Part Number MT28908A0-XCCF-HVM
ConnectX-6 Socket Direct™ Cards
The Socket Direct technology offers improved performance to dual-socket servers by enabling direct access from each CPU in a dual-socket server
to the network through its dedicated PCIe interface.
Please note thatConnectX-6 Socket Direct cards do not support Multi-Host functionality (i.e. connectivity to two independent CPUs). For
ConnectX-6 Socket Direct card with Multi-Host functionality, please contact NVIDIA.
ConnectX-6 Socket Direct cards are available in two configurations: Dual-slot Configuration (2x PCIe x16) and Single-slot Configuration (2x PCIe x8).
ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe x16)
In order to obtain 200Gb/s speed,NVIDIA offers ConnectX-6 Socket Direct that enable 200Gb/s connectivity also for servers with PCIe Gen 3.0 capability.
The adapter’s 32-lane PCIe bus is split into two 16-lane buses, with one bus accessible through a PCIe x16 edge connector and the other bus through an x16
Auxiliary PCIe Connection card. The two cards should be installed into two PCIe x16 slots and connected using two Cabline SA-II Plus harnesses, as shown in
the below figure.

18
Part Number MCX654105A-HCAT MCX654106A-HCAT MCX654106A-ECAT
Form Factor/Dimensions Adapter Card: PCIe Half Height, Half Length / 167.65mm x 68.90mm
Auxiliary PCIe Connection Card: 5.09 in. x 2.32 in. (129.30mm x 59.00mm)
Two 35cm Cabline CA-II Plus harnesses
Data Transmission Rate Ethernet: 10/25/40/50/100/200 Gb/s
InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100, HDR
Ethernet: 10/25/40/50/100 Gb/s
InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100
Network Connector Type Single-port QSFP56 Dual-port QSFP56
PCIe x16 through Edge Connector PCIe Gen 3.0 / 4.0SERDES@ 8.0GT/s / 16.0GT/s
PCIe x16 through Auxiliary Card PCIe Gen 3.0SERDES@ 8.0GT/s

19
Part Number MCX654105A-HCAT MCX654106A-HCAT MCX654106A-ECAT
RoHS RoHS Compliant
Adapter IC Part Number MT28908A0-XCCF-HVM
ConnectX-6 Single-slot Socket Direct Cards (2x PCIe x8 in a row)
The PCIe x16 interface comprises two PCIe x8 in a row, such that each of the PCIe x8 lanes can be connected to a dedicated CPU in a dual-socket server. In
such a configuration, Socket Direct brings lower latency and lower CPU utilization as the direct connection from each CPU to the network means the
interconnect can bypass a QPI (UPI) and the other CPU, optimizing performance and improving latency. CPU utilization is improved as each CPU handles
only its own traffic and not traffic from the other CPU.
A system with a custom PCI Express x16 slot that includes special signals is required for installing the card. Please refer to PCI Express Pinouts Description
for Single-Slot Socket Direct Cardfor pinout definitions.

20
Part Number MCX653105A-EFAT MCX653106A-EFAT
Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm
Data Transmission Rate Ethernet: 10/25/40/50/100 Gb/s
InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100
Network Connector Type Single-port QSFP56 Dual-port QSFP56
PCIe x16 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s Socket Direct 2x8 in a row
RoHS RoHS Compliant
Adapter IC Part Number MT28908A0-XCCF-HVM
Package Contents
ConnectX-6 PCIe x8/x16 Adapter Cards
Category Qty Item
Cards 1 ConnectX-6 adapter card
Accessories 1 Adapter card short bracket
Applies to MCX651105A-EDAT, MCX653105A-ECAT, MCX653106A-ECAT, MCX653105A-HDAT, MCX653106A-HDAT, MCX653105A-EFAT, MCX653106A-EFAT,
and MCX683105AN-HDAT.
