Nvidia ConnectX-7 User manual

Exported on Feb/16/2023 05:54 AM







NVIDIA ConnectX-7 Adapter Cards User
Manual



2
Table of Contents
About This Manual ............................................................................. 6
Intended Audience............................................................................. 7
Ordering Part Numbers........................................................................ 8
ConnectX-7 PCIe x16 Stand-up Adapter Cards...................................................8
ConnectX-7 for Telecommunication Applications ........................................... 9
ConnectX-7 Socket Direct Ready Cards for Dual-Slot Servers ............................... 10
Technical Support .............................................................................11
Related Documentation ......................................................................12
Document Conventions.......................................................................13
Revision History ...............................................................................14
Introduction....................................................................................15
Product Overview .................................................................................. 15
PCIe x16 Stand-up Adapter Cards............................................................ 15
Socket Direct Ready Cards.................................................................... 15
System Requirements ............................................................................. 16
Package Contents .................................................................................. 16
Features and Benefits ............................................................................. 17
Supported Interfaces .........................................................................21
ConnectX-7 Layout and Interface Information ................................................ 21
Interfaces Detailed Description.................................................................. 22
ConnectX-7 IC .................................................................................. 22
PCI Express Interface.......................................................................... 22
Networking Interfaces......................................................................... 22
Networking Ports LEDs Specifications....................................................... 22
Cabline CA-II Plus Connectors................................................................ 23
SMBus Interface ................................................................................ 24
Voltage Regulators ............................................................................. 24
Hardware Installation ........................................................................25
Safety Warnings .................................................................................... 25
Installation Procedure Overview................................................................. 26
System Requirements ............................................................................. 26
Hardware Requirements ...................................................................... 26

3
Airflow Requirements ......................................................................... 27
Software Requirements ....................................................................... 27
Safety Precautions ................................................................................. 27
Pre-Installation Checklist ......................................................................... 27
Bracket Replacement Instructions .............................................................. 28
Installation Instructions........................................................................... 28
Cables and Modules............................................................................ 29
Identifying the Card in Your System............................................................. 29
ConnectX-7 PCIe x16 Installation Instructions................................................. 31
Installing the Card ............................................................................. 31
Uninstalling the Card.......................................................................... 32
ConnectX-7 Socket Direct (2x PCIe x16) Installation Instructions .......................... 34
Installing the Cards ............................................................................ 35
Uninstalling the Cards......................................................................... 40
Driver Installation.............................................................................43
Linux Driver Installation .......................................................................... 43
Prerequisites.................................................................................... 43
Downloading MLNX_OFED..................................................................... 43
Installing MLNX_OFED ......................................................................... 44
Driver Load Upon System Boot............................................................... 49
Additional Installation Procedures .......................................................... 51
Performance Tuning ........................................................................... 57
Windows Driver Installation ...................................................................... 57
Software Requirements ....................................................................... 58
Downloading WinOF-2 Driver................................................................. 58
Installing WinOF-2 Driver ..................................................................... 58
Firmware Upgrade ............................................................................. 66
VMware Driver Installation ....................................................................... 66
Hardware and Software Requirements ..................................................... 66
Installing NATIVE ESXi Driver for VMware vSphere ........................................ 66
Removing Earlier NVIDIA Drivers ............................................................. 67
Firmware Programming ....................................................................... 67
Updating Adapter Firmware .................................................................68
Setting High-Speed-Port Link Type .........................................................69

4
mlxconfig............................................................................................ 69
UEFI .................................................................................................. 69
Troubleshooting ...............................................................................70
General Troubleshooting .......................................................................... 70
Linux Troubleshooting ............................................................................. 70
Windows Troubleshooting......................................................................... 71
Specifications..................................................................................72
MCX75310AAC-NEAT / MCX75310AAS-NEAT Specifications................................... 72
MCX75310AAS-HEAT Specifications .............................................................. 73
MCX755106AC-HEAT / MCX755106AS-HEAT Specifications ................................... 74
MCX715105AS-WEAT Specifications.............................................................. 75
MCX75510AAS-HEAT Specifications .............................................................. 76
MCX75510AAS-NEAT Specifications .............................................................. 77
MCX713106AC-CEAT and MCX713106AS-CEAT Specifications ................................ 79
MCX713106AC-VEAT and MCX713106AS-VEAT Specifications ................................ 80
MCX713104AC-ADAT and MCX713104AS-ADAT Specifications ................................ 81
MCX713114TC-GEAT and MCX713114GC-GEAT Specifications ............................... 81
Cards Mechanical Drawings and Dimensions ................................................... 82
Brackets Mechanical Drawings and Dimensions ............................................... 83
Monitoring......................................................................................85
Thermal Sensors.................................................................................... 85
Adapter Card Heatsink ............................................................................ 85
Finding the GUID/MAC on the Adapter Card ..............................................86
PCIe Auxiliary Card Kit .......................................................................87
PCIe Auxiliary Card Package Contents .......................................................... 88
Channel Insertion Loss ............................................................................ 88
Cabline CA-II Plus Harness Pinouts .............................................................. 88
Component Side................................................................................ 88
Print Side........................................................................................ 97
PCIe Auxiliary Kit Technical Specifications....................................................106
Technical Specifications ..................................................................... 106
PCIe Auxiliary Card Mechanical Drawings and Dimensions.............................. 107
Bracket Mechanical Drawings and Dimensions ........................................... 107
Cabline CA-II Plus Harnesses Mechanical Drawing ....................................... 108

5
Document Revision History ................................................................ 109


6
About This Manual
This User Manual describes NVIDIA® ConnectX®-7 InfiniBand and Ethernet adapter cards. It provides
details as to the interfaces of the board, specifications, required software and firmware for
operating the board, and relevant documentation.

7
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic
familiarity with InfiniBand and Ethernet network and architecture specifications.

8
Ordering Part Numbers
The table below lists the ordering part numbers (OPNs) for the available ConnectX-7 stand-up cards.
For the Open Compute Project (OCP 3.0) cards, please refer to NVIDIA ConnectX-7 Adapter Cards for
OCP 3.0 User Manual.
ConnectX-7 PCIe x16 Stand-up Adapter Cards
NVIDIA
SKU
Legacy
OPN
Form Factor Data
Transmission
Rate
No. of
Ports and
Type
PCIe Support Secure
Boot
Cryp
to
Bracket
Type
900-9X766
-003N-SQ0
MCX7531
0AAS-
NEAT
PCIe Half
Height, Half
Length
2.71 in. x 6.6
in. (68.90mm
x 167.65 mm)
InfiniBand:
NDR 400Gb/s
(Default
speed)
Ethernet:
400GbE
Single-port
OSFP
PCIe x16 Gen
4.0/5.0 @
SERDES 16GT/
s/32GT/s
✔- Tall
Bracket
900-9X766
-003N-SR0
MCX7531
0AAC-
NEAT
PCIe Half
Height, Half
Length
2.71 in. x 6.6
in. (68.90mm
x 167.65 mm)
InfiniBand:
NDR 400Gb/s
(Default
speed)
Ethernet:
400GbE
Single-port
OSFP
PCIe x16 Gen
4.0/5.0 @
SERDES 16GT/
s/32GT/s
✔ ✔ Tall
Bracket
900-9X766
-003N-ST0
MCX7531
0AAS-
HEAT
PCIe Half
Height, Half
Length
2.71 in. x 6.6
in. (68.90mm
x 167.65 mm)
InfiniBand:
NDR200
200Gb/s
(Default
speed)
Ethernet:
200GbE
Single-port
OSFP
PCIe x16 Gen
4.0/5.0 @
SERDES 16GT/
s/32GT/s
✔- Tall
Bracket
900-9X7A
H-0088-
ST0
MCX7131
06AC-
VEAT
PCIe Half
Height, Half
Length
2.71 in. x 6.6
in. (68.90mm
x 167.65 mm)
Ethernet:
200GbE
Dual-port
QSFP112
PCIe x16 Gen
4.0/5.0 @
SERDES 16GT/
s/32GT/s
✔ ✔ Tall
Bracket
900-9X7A
H-0078-
ST0
MCX7131
06AS-
VEAT
PCIe Half
Height, Half
Length
2.71 in. x 6.6
in. (68.90mm
x 167.65 mm)
Ethernet:
200GbE
Dual-port
QSFP112
PCIe x16 Gen
4.0/5.0 @
SERDES 16GT/
s/32GT/s
✔- Tall
Bracket
900-9X7A
H-0086-
SQ0
MCX7131
06AC-
CEAT
PCIe Half
Height, Half
Length
2.71 in. x 6.6
in. (68.90mm
x 167.65 mm)
Ethernet:
100GbE
Dual-port
QSFP112
PCIe x16 Gen
4.0/5.0 @
SERDES 16GT/
s/32GT/s
✔ ✔ Tall
Bracket

9
NVIDIA
SKU
Legacy
OPN
Form Factor Data
Transmission
Rate
No. of
Ports and
Type
PCIe Support Secure
Boot
Cryp
to
Bracket
Type
900-9X7A
H-0076-
ST0
MCX7131
06AS-
CEAT
PCIe Half
Height, Half
Length
2.71 in. x 6.6
in. (68.90mm
x 167.65 mm)
Ethernet:
100GbE
Dual-port
QSFP112
PCIe x16 Gen
4.0/5.0 @
SERDES 16GT/
s/32GT/s
✔- Tall
Bracket
900-9X7A
O-00C3-
STZ
MCX7131
04AC-
ADAT
PCIe Half
Height, Half
Length
2.71 in. x 6.6
in. (68.90mm
x 167.65 mm)
Ethernet:
50/25GbE
Quad-port
SFP56
PCIe x16 Gen
4.0 @ SERDES
16GT/s
✔ ✔ Tall
Bracket
900-9X7A
O-0003-
ST0
MCX7131
04AS-
ADAT
PCIe Half
Height, Half
Length
2.71 in. x 6.6
in. (68.90mm
x 167.65 mm)
Ethernet:
50/25GbE
Quad-port
SFP56
PCIe x16 Gen
4.0 @ SERDES
16GT/s
✔-Tall
Bracket
ConnectX-7 for Telecommunication Applications
NVIDIA
SKU
Legacy
OPN
Form
Factor
Data
Transmis
sion
Rate
No. of
Ports
and Type
PCIe
Support
Secure
Boot
Cryp
to
Timing
Capabiliti
es
Bracke
t Type
900-9X7A
H-004N-
CT0
MCX713
114TC-
GEAT
PCIe Full
Height, Half
Length
4.53 in. x
6.6 in.
(115.15 mm
x 167.65
mm)
Ethernet:
50/25GbE
Quad-
port
SFP56
PCIe x16
Gen 4.0 @
SERDES
16GT/s
✔ ✔ PPS In /
Out, SMAs,
SycnE
& PTP GM
Support
Tall
Bracket
900-9X7A
H-004N-
GT0
MCX713
114GC-
GEAT
PCIe Full
Height, Half
Length
4.53 in. x
6.6 in.
(115.15 mm
x 167.65
mm)
Ethernet:
50/25GbE
Quad-
port
SFP56
PCIe x16
Gen 4.0 @
SERDES
16GT/s
✔ ✔ Enhanced-
SyncE &
PTP Grand
Master
support
and GNSS/
PPS Out
Tall
Bracket

10
ConnectX-7 Socket Direct Ready Cards for Dual-Slot
Servers
NVIDIA
SKU
Legacy
OPN
Form
Factor
Data
Transmissi
on Rate
No. of
Ports
and
Type
PCIe
Support
Socket
Direct
Ready -
PCIe
Extension
Option
Secur
e Boot
Cry
pto
Bracke
t Type
900-9X7A
H-0039-
STZ
MCX715
105AS-
WEAT
PCIe Half
Height,
Half Length
2.71 in. x
6.6 in.
(68.90mm x
167.65 mm)
InfiniBand:
NDR
400Gb/s
Ethernet:
400GbE
(Default
Speed)
Single-
port
QSFP112
PCIe x16
Gen 4.0/5.0
@ SERDES
16GT/s/
32GT/s
Optional:
PCIe x16
Gen 4.0 @
SERDES
16GT/s
✔-Tall
Bracket
900-9X72
1-003N-
DT1
MCX755
10AAS-
NEAT
PCIe Half
Height,
Half Length
2.71 in. x
6.6 in.
(68.90mm x
167.65 mm)
InfiniBand:
NDR
400Gb/s
Single-
port
OSFP
PCIe x16
Gen 4.0/5.0
@ SERDES
16GT/s/
32GT/s
Optional:
PCIe x16
Gen 4.0 @
SERDES
16GT/s
✔-Tall
Bracket
900-9X72
1-003N-
DT1
MCX755
10AAS-
HEAT
PCIe Half
Height,
Half Length
2.71 in. x
6.6 in.
(68.90mm x
167.65 mm)
InfiniBand:
NDR200
200Gb/s
Single-
port
OSFP
PCIe x16
Gen 4.0/5.0
@ SERDES
16GT/s/
32GT/s
Optional:
PCIe x16
Gen 4.0 @
SERDES
16GT/s
✔-Tall
Bracket
900-9X7A
H-0078-
DTZ
MCX755
106AS-
HEAT
PCIe Half
Height,
Half Length
2.71 in. x
6.6 in.
(68.90mm x
167.65 mm)
InfiniBand:
NDR200
200Gb/s
Ethernet:
200GbE
(Default
Speed)
Dual-port
QSFP112
PCIe x16
Gen 4.0/5.0
@ SERDES
16GT/s/
32GT/s
Optional:
PCIe x16
Gen 4.0 @
SERDES
16GT/s
✔-Tall
Bracket
900-9X7A
H-0079-
DTZ
MCX755
106AC-
HEAT
PCIe Half
Height,
Half Length
2.71 in. x
6.6 in.
(68.90mm x
167.65 mm)
InfiniBand:
NDR200
200Gb/s
Ethernet:
200GbE
(Default
Speed)
Dual-port
QSFP112
PCIe x16
Gen 4.0/5.0
@ SERDES
16GT/s/
32GT/s
Optional:
PCIe x16
Gen 4.0 @
SERDES
16GT/s
✔ ✔ Tall
Bracket
For more information, please refer toPCIe Auxiliary Card Kit.

11
•
•
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough
the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchased NVIDIA Global Support Services, please see your contract for details
regarding Technical Support.
Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek
assistance through their reseller.

12
Related Documentation
MLNX_OFED for
Linux User Manual
and Release Notes
User Manual describing OFED features, performance, band diagnostic, tools content
and configuration. SeeMLNX_OFED for Linux Documentation.
WinOF-2 for
WindowsUser
Manual and Release
Notes
User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools
content and configuration. SeeWinOF-2 for Windows Documentation.
NVIDIA VMware for
Ethernet User
Manual
User Manual and release notes describing the various components of the NVIDIA
ConnectX® NATIVE ESXi stack. See VMware® ESXi Drivers Documentation.
NVIDIA Firmware
Utility (mlxup) User
Manual and Release
Notes
NVIDIA firmware update and query utility used to update the firmware. Refer to
Firmware Utility (mlxup) Documentation.
NVIDIA Firmware
Tools (MFT) User
Manual
User Manual describing the set of MFT firmware management tools for a single node.
SeeMFT User Manual.
InfiniBand
Architecture
Specification
Release 1.2.1, Vol 2
- Release 1.4, and
Vol 2 - Release 1.5
InfiniBand Specifications
IEEE Std 802.3
Specification
IEEE Ethernet Specifications
PCI Express 5.0
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications. Refer to
PCI-SIG Specifications.
LinkX Interconnect
Solutions
LinkX cables and transceivers are designed to maximize the performance of High-
Performance Computing networks, requiring high-bandwidth, low-latency connections
between compute nodes and switch nodes. NVIDIA offers one of the industry’s most
complete line of 10, 25, 40, 50, 100, 200, and 400GbE in Ethernet and EDR, HDR, and
NDR, includingDirect Attach Copper cables (DACs), copper splitter cables, Active
Optical Cables (AOCs) and transceivers in a wide range of lengths from 0.5m to 10km.
In addition to meeting Ethernet and IBTA standards, NVIDIA tests every product in an
end-to-end environment ensuring a Bit Error Rate of less than 1E-15.Read more at
LinkX Cables and Transceivers.
NVIDIA ConnectX-7
Electrical and
Thermal
Specifications
Electrical and Thermal specifications are provided in the "NVIDIA ConnectX-7 Adapter
Cards Electrical and Thermal Specifications"document available at NVOnline following
login.

13
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in MegaBytes.
The use of Mb or Mbits (small b) indicates size in MegaBits.IB is used in this document to mean
InfiniBand. In this document, PCIe is used to mean PCI Express.

14
Revision History
A list of the changes made to this document is provided in Document Revision History.

15
•
•
•
•
•
Introduction
Product Overview
The NVIDIA ConnectX-7 family of network adapters supports both the InfiniBand and Ethernet
protocols. It enables a wide range of smart, scalable, and feature-rich networking solutions that
address traditional enterprise needs up to the world’s most demanding AI, scientific computing, and
hyperscale cloud data center workloads.
ConnectX-7 network adapters are offered in two form factors and various flavors: stand-up PCIe and
Open Compute Project (OCP) Spec 3.0 cards. This user manual covers the PCIe stand-up cards, for
the OCP 3.0 cards, please refer to NVIDIA ConnectX-7 Cards for OCP Spec 3.0 User Manual.
PCIe x16 Stand-up Adapter Cards
ConnectX-7 HCAs are available in various configurations; Single-port 400Gb/s or 200Gb/s, with octal
small form-factor pluggable (OSFP) connectors or Dual-port 100 or 200Gb/s with quad small form-
factor pluggable (QSFP112) connectors on PCIe standup half-height, half-length (HHHL) form factor,
with options for NVIDIA Socket Direct. Also available, Dual-port 50/25 GbE with quad small form-
factor pluggable (SFP56) connectors on PCIe standup full-height, half-length (FHHL) form factor,
with timing capabilities.
ConnectX-7 cards can either support both InfiniBand and Ethernet, or Ethernet only, as described in
the below table. The inclusive list of OPNs is available here.
Supported Protocols Port Type Supported Speed
Ethernet Only Card Dual-port QSFP112 200GbE
100GbE
Quad-port SFP56 50/25GbE
InfiniBand and Ethernet Single-port OSFP NDR 400Gb/s and 400GbE
NDR200 200Gb/s and 200GbE
Socket Direct Ready Cards
The Socket Direct technology offers improved performance to dual-socket servers by enabling direct
access from each CPU in a dual-socket server to the network through its dedicated PCIe interface.
NVIDIA offers ConnectX-7 Socket Direct adapter cards, which enable 400Gb/s or 200Gb/s
connectivity, and also for servers with PCIe Gen 4.0 capability. The adapter’s 32-lane PCIe bus is
split into two 16-lane buses, with one bus accessible through a PCIe x16 edge connector and the
other bus through an x16 Auxiliary PCIe Connection card. The two cards should be installed into two
PCIe x16 slots and connected using two Cabline SA-II Plus harnesses.
To use this card in the Socket-Direct configuration, please order the additional PCIe Auxiliary Card
kit according to the desired harness length. Cards that support socket direct can function as
separate x16 PCIe cards.
Make sure to use a PCIe slot capable of supplying the required power and airflow to the
ConnectX-7, as stated in the Specifications chapter.


16
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Socket Direct cards can support both InfiniBand and Ethernet, or InfiniBand only, as described
below.
Supported Protocols Port Type Supported Speed
InfiniBand Only Single-port OSFP NDR 400Gb/s
NDR200 200Gb/s
InfiniBand and Ethernet Dual-port QSFP112 NDR200 200Gb/s and 200GbE
Single-port QSFP112 NDR 400Gb/s and 400GbE
For more information on the passive PCIe Auxiliary kit, please refer toPCIe Auxiliary Card Kit.
System Requirements
Item Description
PCI
Express
slot
In PCIe x16 Configuration
PCIe Gen 5.0 (32GT/s) through x16 edge connector.
In Socket Direct Configuration (2x PCIe x16)
PCIe Gen 4.0/5.0 SERDES @16/32GT/s through edge connector
PCIe Gen 4.0 SERDES @16GT/s through PCIe Auxiliary Connection Card
System
Power
Supply
Refer to Specifications
Operating
System
In-box drivers for major operating systems:
Linux: RHEL, Ubuntu
Windows
Virtualization and containers
VMware ESXi (SR-IOV)
Kubernetes
OpenFabrics Enterprise Distribution (OFED)
OpenFabrics Windows Distribution (WinOF-2)
Connectiv
ity
Interoperable with 1/10/25/40/50/100/200/400 Gb/s Ethernet switches and SDR/DDR/
EDR/HDR100/HDR/NDR200/NDR InfiniBand switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support
Package Contents
Category Qty Item
Cards 1 ConnectX-7 adapter card
Accessories 1 Adapter card short bracket
1 Adapter card tall bracket (shipped assembled on the
card)

17
•
•
Features and Benefits
PCI
Express
(PCIe)
According to the OPN you have purchased, the card uses the following PCIe express interfaces:
PCIe x16 configurations:
PCIe Gen 4.0/5.0 (16GT/s / 32GT/s) through x16 edge connector.
2x PCIe x16 configurations (Socket-Direct):
PCIe Gen 4.0/5.0 ( SERDES @ 16GT/s / 32GT/s) through x16 edge connector
PCIe Gen 4.0 SERDES @ 16GT/s through PCIe Auxiliary Connection Card
InfiniBand
Architect
ure
Specificat
ion v1.5
compliant
ConnectX-7 delivers low latency, high bandwidth, and computing efficiency for high-performance
computing (HPC), artificial intelligence (AI), and hyperscale cloud data center applications.
ConnectX-7 is InfiniBand Architecture Specification v1.5 compliant.
InfiniBand Network Protocols and Rates:
Protocol Standard Rate (Gb/s) Comments
4x Port
(4 Lanes)
2x Ports
(2 Lanes)
NDR/NDR200 IBTA Vol2 1.5 425 212.5 PAM4 256b/
257b
encoding
and RS-FEC
HDR/HDR100 IBTA Vol2 1.4 212.5 106.25 PAM4 256b/
257b
encoding
and RS-FEC
EDR IBTA Vol2 1.3.1 103.125 51.5625 NRZ 64b/
66b
encoding
FDR IBTA Vol2 1.2 56.25 N/A NRZ 64b/
66b
encoding
Make sure to use a PCIe slot capable of supplying the required power and airflow to the
ConnectX-7 cards as stated in the Specifications chapter.

This section describes hardware features and capabilities. Please refer to the relevant
driver and firmware release notes for feature availability.


18
•
•
Up to 400
Gigabit
Ethernet
ConnectX-7 adapter cards comply with the following IEEE 802.3 standards:
400GbE / 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE
Protocol MAC Rate
IEEE802.3ck 100/200/400Gb/s Gigabit Ethernet
(Include ETC enhancement)
IEEE802.3cd
IEEE802.3bs
IEEE802.3cm
IEEE802.3cn
IEEE802.3cu
50/100/200/400Gb/s Gigabit Ethernet
(Include ETC enhancement)
IEEE 802.3bj
IEEE 802.3bm
100 Gigabit Ethernet
IEEE 802.3by
Ethernet Consortium25
25/50 Gigabit Ethernet
IEEE 802.3ba 40 Gigabit Ethernet
IEEE 802.3ae 10 Gigabit Ethernet
IEEE 802.3cb 2.5/5 Gigabit Ethernet
(For 2.5: support only 2.5 x1000BASE-
X)
IEEE 802.3ap Based on auto-negotiation and KR
startup
IEEE 802.3ad
IEEE 802.1AX
Link Aggregation
IEEE 802.1Q
IEEE 802.1P VLAN tags and priority
IEEE 802.1Qau (QCN)
Congestion Notification
IEEE 802.1Qaz (ETS)
EEE 802.1Qbb (PFC)
IEEE 802.1Qbg
IEEE 1588v2
IEEE 802.1AE (MACSec)
Jumbo frame support (9.6KB)
Memory
Compone
nts
SPI - includes 256Mbit SPI Quad Flash device.
FRU EEPROM - Stores the parameters and personality of the card. The EEPROM capacity is
128Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus. (Note:
Address 0x58 is reserved.)
Overlay
Networks
In order to better scale their networks, datacenter operators often create overlay networks that
carry traffic from individual virtual machines over logical tunnels in encapsulated formats such
as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from
the hardware offloading engines, placing higher loads on the host CPU. ConnectX-7 effectively
addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that
encapsulate and de-capsulate the overlay protocol.
Quality of
Service
(QoS)
Support for port-based Quality of Service enabling various application requirements for latency
and SLA.

19
•
•
•
•
•
•
•
•
Hardware
-based I/
O
Virtualiza
tion
ConnectX-7 provides dedicated adapter resources and guaranteed isolation and protection for
virtual machines within the server.
Storage
Accelerat
ion
A consolidated compute and storage network achieves significant cost-performance advantages
over multi-fabric networks. Standard block and file access protocols can leverage:
RDMA for high-performance storage access
NVMe over Fabric offloads for the target machine
NVMe over TCP acceleration
SR-IOV ConnectX-7 SR-IOV technology provides dedicated adapter resources and guaranteed isolation
and protection for virtual machines (VM) within the server.
High-
Performa
nce
Accelerat
ions
Collective operations offloads
Vector collective operations offloads
MPI tag matching
MPI_Alltoall offloads
Rendezvous protocol offload
RDMA
Message
Rate
330-370 million messages per second.
Secure
Boot
The secure boot process assures booting of authentic firmware/software that is intended to run
on ConnectX-7. This is achieved using cryptographic primitives using asymmetric cryptography.
ConnectX-7 supports several cryptographic functions in its HW Root-of-Trust (RoT) that has its
key stored in on-chip FUSES.
Secure
Firmware
Update
The Secure firmware update feature enables a device to verify digital signatures of new
firmware binaries to ensure that only officially approved versions can be installed from the host,
the network, or a Board Management Controller (BMC). The firmware of devices with “secure
firmware update” functionality (secure FW), restricts access to specific commands and registers
that can be used to modify the firmware binary image on the flash, as well as commands that
can jeopardize security in general.
For further information, refer to theMFT User Manual.
Advanced
storage
capabiliti
es
Block-level encryption and checksum offloads.

20
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Host
Managem
ent
ConnectX-7 technology maintains support for host manageability through a BMC. ConnectX-7
PCIe stand-up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe
protocols as if it is a standard NVIDIA PCIe stand-up adapter card. For configuring the adapter for
the specific manageability solution in use by the server, please contact NVIDIA Support.
Protocols: PLDM, NCSI
Transport layer – RBT, MCTP over SMBus and MCTP over PCIe
Physical layer: SMBus 2.0 / I2C interface for device control and configuration, PCIe
PLDM for Monitor and Control DSP0248
PLDM for Firmware Update DSP026
IEEE 1149.6
SecuredFW update
FWRecovery
NIC reset
Monitoring and control
Network port settings
Boot setting
Accurate
timing
NVIDIA offers a full IEEE 1588v2 PTP software solution, as well as time-sensitive related features
called “5T”. NVIDIA PTP and 5T software solutions are designed to meet the most demanding
PTP profiles. ConnectX-7 incorporates an integrated Hardware Clock (PHC) that allows
ConnectX-7 to achieve sub 20u Sec accuracy and also offers many timing-related functions such
as time-triggered scheduling or time-based SND accelerations (time-based ASAP²). Furthermore,
5T technology enables the software application to transmit fronthaul (ORAN) compatible in high
bandwidth. The PTP part supports the subordinate clock, master clock, and boundary clock.
ConnectX-7 PTP solution allows you to run any PTP stack on your host.
With respect to testing and measurements, selected NVIDIA adapters allow you to use the PPS-
out signal from the onboard SMA connecter, ConnectX-7 also allows measuring PTP in scale, with
a PPS-In signal. The PTP HW clock on the Network adapter will be sampled on each PPS-In signal,
and the timestamp will be sent to the SW.
RDMA and
RDMA
over
Converge
d
Ethernet
(RoCE)
ConnectX-7, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged
Ethernet) technology, delivers low-latency and high-performance over InfiniBand and Ethernet
networks. Leveraging datacenter bridging (DCB) capabilities as well as ConnectX-7 advanced
congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services
over Layer 2 and Layer 3 networks.
NVIDIA
PeerDirec
t™
PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary
internal data copies between components on the PCIe bus (for example, from GPU to CPU), and
therefore significantly reduces application run time. ConnectX-7 advanced acceleration
technology enables higher cluster efficiency and scalability to tens of thousands of nodes.
CPU
Offload
Adapter functionality enables reduced CPU overhead allowing more available CPU for
computation tasks.
Flexible match-action flow tables
Open VSwitch (OVS) offload using ASAP2®
Tunneling encapsulation/decapsulation