Nvidia BlueField-2 BF2500 User manual

Exported on Oct/31/2023 01:37 PM
NVIDIA BlueField-2 BF2500 Ethernet DPU Controller User Manual

2
Table of Contents
Introduction............................................................................................................................................ 9
System Requirements ........................................................................................................................................... 9
Package Contents ...............................................................................................................................................10
Card Package ................................................................................................................................................ 10
Accessories Kit .............................................................................................................................................. 10
Product Overview ...............................................................................................................................................11
Ideal Solution for JBOF and JBOD Systems.............................................................................................................. 11
Features and Benefits ...................................................................................................................................... 11
Supported Interfaces ................................................................................................................................ 14
Layout and Interfaces ..........................................................................................................................................14
Interfaces Detailed Description...............................................................................................................................17
DPU............................................................................................................................................................ 17
Encryption ............................................................................................................................................... 18
Ethernet QSFP56 Interface ................................................................................................................................ 18
PCI Express Interface....................................................................................................................................... 18
DDR4 SDRAM On-Board Memory........................................................................................................................... 19
NC-SI Management Interface .............................................................................................................................. 19
UART Interface Connectivity .............................................................................................................................. 19
USB Interfaces ............................................................................................................................................... 20

3
1GbE OOB Management Interface ........................................................................................................................ 20
1GbE OOB Management LEDs Interface.............................................................................................................. 21
RTC Battery .................................................................................................................................................. 21
eMMC Interface.............................................................................................................................................. 21
External PCIe Power Supply Connector.................................................................................................................. 21
Networking Ports LEDs Interface ......................................................................................................................... 22
Pin Description ....................................................................................................................................... 23
PCI Express Interface...........................................................................................................................................23
Power Sequencing........................................................................................................................................... 28
NC-SI Management Interface ..................................................................................................................................28
External Power Supply Connector (6-pin ATX Connector).................................................................................................35
Hardware Installation ............................................................................................................................... 36
Safety Warnings .................................................................................................................................................36
Installation Procedure Overview..............................................................................................................................36
System Requirements ..........................................................................................................................................37
Hardware Requirements ................................................................................................................................... 37
Airflow Requirements ...................................................................................................................................... 37
Software Requirements .................................................................................................................................... 38
Safety Precautions ..............................................................................................................................................38
Pre-Installation Checklist ......................................................................................................................................38
Installation Instructions........................................................................................................................................39

4
Cables and Modules.............................................................................................................................................39
Networking Cables .......................................................................................................................................... 39
RJ45 Cable for the 1GbE OOB Management Interface................................................................................................. 40
USB Interface Cabling ...................................................................................................................................... 41
Bring-Up and Driver Installation ................................................................................................................... 42
BlueField Software..............................................................................................................................................42
Software On eMMC..............................................................................................................................................43
Yocto Distribution Installation ............................................................................................................................ 43
PXE Server Configuration on Host Side ......................................................................................................................44
Installing Linux on BlueField Controller Card...............................................................................................................44
Software Requirements .................................................................................................................................... 45
Preparing Host-Side Environment .................................................................................................................... 45
Setup Procedure With Installation Script............................................................................................................ 46
Setup Procedure Without Installation Script........................................................................................................ 46
Flashing BlueField Controller Card Bootloader Code .................................................................................................. 50
Opening Terminal Connection to BlueField Controller Card...................................................................................... 51
Using Initial Install Bootstream ....................................................................................................................... 51
Installing CentOS 7.4 on BlueField Controller Card.................................................................................................... 53
Full PXE Boot Installation.............................................................................................................................. 53
Non-PXE Boot Installation ............................................................................................................................. 60
MLNX_OFED Installation........................................................................................................................................60

5
Installing MLNX_OFED on Arm Cores ..................................................................................................................... 61
Prerequisite Packages for Installing MLNX_OFED................................................................................................... 61
Removing Pre-installed Kernel Module .............................................................................................................. 62
Installing MLNX_OFED on the BlueField Controller Card.......................................................................................... 62
Updating BlueField Controller Card Firmware ..................................................................................................... 64
Troubleshooting ...................................................................................................................................... 66
Specifications......................................................................................................................................... 67
MBF2H516B-CEEOT / MBF2H516B-CENOT Specifications..................................................................................................67
BlueField-2 BF2500 Mechanical Drawing and Dimensions.................................................................................................68
Finding the MAC on the Controller Card.......................................................................................................... 70
Thermal Sensors...................................................................................................................................... 71
Document Revision History ......................................................................................................................... 72



6
About This Manual
It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and a step-by-step plan of
how to bring up the BlueField-2 DPU.
EOL'ed (End of Life) Ordering Part Numbers
OPN F
o
r
m
F
a
c
t
o
r
Series/ Core Speed Max Speed No. of Ports PCIe Support Crypto Secure Boot 1GbE OOB O
n
-
b
o
a
r
d
D
D
R
M
e
m
o
r
y
On-board eMMC
Memory
MBF2H516B-CEEOT F
H
H
L
*
P-Series / 2.75GHz 100GbE 2xQSFP56 Gen 4.0 x16 Crypto Enabled - √1
6
G
B
64GB
MBF2H516B-CENOT F
H
H
L
P-Series / 2.75GHz 100GbE 2xQSFP56 Gen 4.0 x16 Crypto Disabled - √1
6
G
B
64GB
*Full-Height Half-Length

7
•
•
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic familiarity with Ethernet network and architecture
specifications.
Technical Support
Customers who purchased NVIDIA products directly from NVIDIA are invited to contact usthrough the following methods:
E-mail:[email protected]
Enterprise Support page:https://www.nvidia.com/en-us/support/enterprise
Related Documentation
InfiniBand Architecture Specification InfiniBand Trade Association (IBTA) InfiniBand® specification Release 1.3.1, November 2, 2016 and Vol. 2, Release 1.4 ,
and Vol 2 - Release 1.5.
IEEE Std 802.3 Specification IEEE Ethernet specification.
PCI Express Specifications Industry Standard PCI Express Base and Card Electromechanical Specifications.
NVIDIA LinkX Interconnect Solutions The NVIDIA® LinkX® product family of cables and transceivers provide the industry’s broadest portfolio of QDR/FDR10
(40Gb/s), FDR (56Gb/s), EDR/HDR100 (100Gb/s), HDR (200Gb/s) and NDR (400Gb/s) cables, including Direct Attach
Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths
from 0.5m to 10km. In addition to meeting IBTA standards, NVIDIA tests every product in an end-to-end environment
ensuring a Bit Error Rate of less than 1E-15.
BlueField DPU Platform BSP Documentation This guide provides product release notes as well as information on the BSP and how to develop and/or customize
applications, system software, and file system images for the BlueField platform.
DOCA SDK Software Documentation NVIDIA DOCA SDK software.
Document Conventions
When discussing memory sizes, GB and GBytes are used in this document to mean size in giga-bytes. The use of Gb or Gbits (small b) indicates size in giga-
bits. In this document PCIe is used to mean PCI Express.

8
Revision History
A list of the changes made to this document are provided inDocument Revision History.

9
•
•
•
Introduction
This is the User Manual for NVIDIA® BlueField®-2 BF2500 DPU Controller. This document provides details of the product interfaces, specifications, required
software and firmware for operating the board, and a step-by-step plan of how to bring up the BlueField-2 BF2500 DPU Controller.
System Requirements
Item Description
Main-boardPCI Express slot x16 Gen 4.0 slot.
System Power Supply Minimum 75W or greater system power supply for all cards.
These PCIe Gen 4.0 x16 DPU controllersrequire additional 75W through a supplementary 6-pin ATX power supply
connector.
NOTE:The connector is not included in the package. It should be part of system wiring or it can be ordered separately as a
system accessory.
Operating System BlueField-2 DPU is shipped with Ubuntu – a Linux commercial operating system – which includes the NVIDIAOFED stack
(MLNX_OFED), and is capable of running all customer-based Linux applications seamlessly. BlueField-2 DPU also supports
CentOS and has an out-of-band 1GbE management interface.For more information, please refer to the DOCA SDK
documentation or NVIDIA BlueField-2 Software User Manual.
Connectivity Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support
For detailed information, see Specifications.

10
Package Contents
Before installing your new system, unpack it and check against the below tables that all the parts have been sent. Check the parts for visible damage that
may have occurred during shipping.
Card Package
Item Description
Cards 1x BlueField-2 DPU Controller card with an assembled tall bracket
Accessories Kit
The accessories kit should be ordered separately. Earlier controller versions require the kit OPN MBF20-DKIT, while newer versions require kit OPN MBF25-
DKIT.
Kit OPN Contents
MBF20-DKIT 1x USB 2.0 Type A to mini-USB Type B cable
1x USB 2.0 Type A to 30pin Flat Socket
If anything is damaged or missing, contact your reseller.

These DPU controllers, you need a6-pin ATX power supply connector cable to activate the card. The cable is not included in the package. For
further details, please refer toExternal PCIe Power Supply Connector.


11
Product Overview
BlueField-2 BlueField-2BF2500 DPU Controllerfeatures the second generation BlueField-2 data processing Unit (DPU) – aninnovative and high-performance
programmable networking engine. The DPU integrates an array ofeight powerful 64-bit Arm v8 A72 cores interconnected by a coherent mesh with a DDR4
memorycontroller and a dual-port Ethernet network controller.Providing unmatched scalability and efficiency, NVIDIA BF2500 DPU Controlleris the ideal
adapter to accelerate the most demanding workloads in data center, cloud, service provider and storage environments.
Ideal Solution for JBOF and JBOD Systems
NVIDIA BlueField-2 DPU is a highly integrated and efficient controller, optimized for NVMe storage systems, Network Functions Virtualization (NFV), Cloud
and Machine Learning workloads. BlueField-2 integrates all the discrete components of a storage system appliance into a single chip, including Arm core
CPUs, PCIe switch and a network controller, making it the premier solution for building Just-a-Bunch-Of-Flash (JBOF) systems, All-Flash-Array and storage
appliances for NVMe over Fabrics. With an integrated NVMe-oF offload accelerator, the BF2500 DPU Controller has a superior performance advantage over
existing JBOF systems, significantly reducing storage transaction latency, while increasing IOPs (I/O operations per second).
Features and Benefits
This section describes hardware features and capabilities.
Feature Description
PCI Express (PCIe) Uses PCIe Gen 4.0 (16GT/s) through an x16 edge connector, compatible with Gen 3.0, 2.0 and 1.1.
The BlueField BF2500 DPU Controller should be installed only in a JBOF and JBOD Systems as it functions as a PCIe root-complex (RC) initiating
PCIe bus operations. Installing it in a regular host system may damage the card.

It is recommended to upgrade your BlueField product to the latest software and firmware versions available in order to enjoy the latest features
and bug fixes.
Please refer to the software release notes for feature availability.


12
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Feature Description
Up to 200 Gigabit Ethernet The adapters comply with the following IEEE 802.3 standards: 200Gb/s / 100Gb/s / 50Gb/s/ 40Gb/s/ 25Gb/s/ 10Gb/s/ 1Gb/s
IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
IEEE 802.3ba 40 Gigabit Ethernet
IEEE 802.3by 25 Gigabit Ethernet
IEEE 802.3ae 10 Gigabit Ethernet
IEEE 802.3ap based auto-negotiation and KR startup
IEEE 802.3ad, 802.1AX Link Aggregation
IEEE 802.1Q, 802.1P VLAN tags and priority
IEEE 802.1Qau (QCN)
Congestion Notification
IEEE 802.1Qaz (ETS)
IEEE 802.1Qbb (PFC)
IEEE 802.1Qbg
IEEE 1588v2
Jumbo frame support (9.6KB)
On-board Memory Quad SPI NOR FLASH - includes 256Mbit for Firmware image.
UVPS EEPROM - includes 1Mbit.
FRU EEPROM - Stores the parameters and personality of the card. The EEPROM capacity is 128Kbit. FRU I2C address is (0x50) and is accessible
through the PCIe SMBus.
eMMC - x8 NAND flash
DDR4 SDRAM - 16GB @3200MT/s single-channel DDR4 SDRAM memory. Solder down on-board. 64bit + 8bit ECC.
BlueField-2 DPU The BlueField-2 DPU integrates eight 64-bit Armv8 A72 cores interconnected by a coherent mesh network, one DRAM controller, an RDMA intelligent network
adapter supporting up to 200Gb/s, an embedded PCIe switch with endpoint and root complex functionality, and up to 16 lanes of PCIe Gen 4.0.
Overlay Networks In order to better scale their networks, data center operators often create overlay networks that carry traffic from individual virtual machines over logical
tunnels in encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware
offloading engines, placing higher loads on the host CPU. DPU effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading
engines that encapsulate and de-capsulate the overlay protocol.
RDMA and RDMA overConverged
Ethernet (RoCE)
DPU, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high-performance
over Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as advanced congestion control hardware mechanisms, RoCE provides
efficient low-latency RDMA services over Layer 2 and Layer 3 networks.
NVIDIA PeerDirect NVIDIA PeerDirectcommunication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe
bus (for example, from GPU to CPU), and therefore significantly reduces application run time. DPU advanced acceleration technology enables higher cluster
efficiency and scalability to tens of thousands of nodes.

13
•
•
Feature Description
Quality of Service (QoS) Support for port-based Quality of Service enabling various application requirements for latency and SLA.
Storage Acceleration A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access
protocols can leverage RDMA for high-performance storage access.
NVMe over Fabric offloads for the target machine
T10-DIF Signature Handover
BlueField-2 DPU may operate as a co-processor offloading specific storage tasks from the host, isolating part of the storage media from the host, or enabling
abstraction of software-defined storage logic
using the BlueField-2 Arm cores. On the storage initiator side, BlueField-2 DPU can prove an efficient solution for hyper-converged systems to enable the
host CPU to focus
on compute while all the storage interface is handled through the Arm cores.
NVMe-oF Nonvolatile Memory Express (NVMe) over Fabrics is a protocol for communicating block storage IO requests over RDMA to transfer data between a host
computer and a target solid-state storage device or system over a network. BlueField-2 DPU may operate as a co-processor offloading specific storage tasks
from the host using its powerful NVMe over Fabrics Offload accelerator.
SR-IOV DPU SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within the server.
GPU Direct The latest advancement in GPU-GPU communications is GPUDirect RDMA. This new technology provides a direct P2P (Peer-to-Peer) data path between the
GPU Memory directly to/from the HCA devices. This provides a significant decrease in GPU-GPU communication latency and completely offloads the CPU,
removing it from all GPU-GPU communications across the network. DPU uses high-speed DMA transfers to copy data between P2P devices resulting in more
efficient system applications
Crypto The BlueField-2 DPU crypto enabled versions include a BlueField-2 IC which supports accelerated cryptographic operations. In addition to specialized
instructions for bulk cryptographic processing in the Arm cores, an offload hardware engine accelerates public-key cryptography and random number
generation are enabled.
Security Accelerators A consolidated compute and network solution based on DPU achieves significant advantages over a centralized security server solution. Standard encryption
protocols and security applications can leverage BlueField-2 compute capabilities and network offloads for security application solutions such as Layer4
Stateful Firewall.
Out-of-Band Management The BlueField-2 DPU incorporates a 1GbE RJ45 out-of-band port that allows the network operator to establish trust boundaries in accessing the management
function to apply it to network resources. It can also be used to ensure management connectivity (including the ability to determine the status of any
network component) independent of the status of other in-band network components.

14
Supported Interfaces
Layout and Interfaces
This section describes the DPU supported interfaces. Each numbered interface that is referenced in the figures is described in the following table with a
link to detailed information.
Component Side
The below figures are for illustration purposes only and might not reflect the current revision of the BF2500 card.


15
Print Side

16

17
Item Interface Description
1 DPU DPU IC 8 cores
2 Ethernet QSFP56 Interface The ethernet traffic is transmitted through the DPU QSFP56 connectors. The QSFP56 connectors allow the use of modules, optical and passive cable
interconnect solutions.By default, the ports of this group of OPNs are set to operate in QSFP28 mode (default card firmware setting).
3 PCI Express Interface PCIe Gen 4.0 through an x16 edge connector
4 DDR4 SDRAM On-Board Memory Units of SDRAM for a total of 16 GB @ 3200MT/s single DDR4 channel,64bit + 8bit ECC, solder-down memory
5 NC-SI Management Interface BMC connectivity for remote management
6 Mini USB Type B Interface Used for OS image loading
7 1GbE OOB Management
Interface
1GbE BASE-T OOB management interface
8 External PCIe Power Supply
Connector
An external 12V power connection through a 6-pin ATX connector.
9 Networking Ports LEDs Interface One bi-color I/O LED per port to indicate link and physical status
10 RTC Battery CR621 battery holder for RTC
11 eMMC Interface x8 NAND flash
Interfaces Detailed Description
DPU
NVIDIA® BlueField®-2 DPUis a family of advanced DPU IC solutions that integrate a coherent mesh of 64-bit Arm v8 A72 cores,anNVIDIA® ConnectX®-6
Dxnetwork adapter front-end and a PCI Express switch into a single chip. The powerful DPU IC architecture includes an Arm v8 multicore processor array
and enables customers to develop sophisticated applications and highly differentiated feature sets. leverages the rich Arm software ecosystem and
introduces the ability to offload the x86 software stack.

18
•
•
•
•
At the heartBlueField-2,the ConnectX-6 Dx network offload controller with RDMA and RDMA over Converged Ethernet (RoCE) technology delivers cutting-
edge performance for networking and storage applications such as NVMe over Fabrics. Advanced features include an embedded virtual switch with
programmable access lists (ACLs), transport offloads and stateless encaps/decaps of NVGRE, VXLAN, and MPLS overlay protocols.
Encryption
DPU addresses the concerns of modern data centers by combining hardware encryption accelerators with embedded software and fully integrated
advanced network capabilities, making it an ideal platform for developing proprietary security applications. It enables a distributed security architecture
by isolating and protecting each individual workload and providing flexible control and visibility at the server and workload level, controlling risk at the
server access layer. builds security into the DNA of the data center and enables prevention, detection, and response to potential threats in real-time. DPU
is capable of delivering powerful functionality, including encryption of data-in-motion, bare-metal provisioning, stateful L4 firewall and more.
Ethernet QSFP56 Interface
The network ports of the DPU are compliant with the IEEE 802.3 Ethernet standards listed inFeatures and Benefits. Ethernet traffic is transmitted through
the cards' QSFP56 connectors. Note that the ports operate in QSFP28 mode by default.
PCI Express Interface
The DPU supports PCI Express Gen4.0 (3.0, 2.0, and 1.1 compatible) through an x16 edge connector.The following lists PCIe interface features:
PCIe Gen 4.0 compliant, and 3.0, 2.0 and 1.1 compatible
2.5, 5.0, or 8.0, or 16.0 GT/s link rate x16 lanes
Auto-negotiates to x16, x8, x4, x2, or x1
Support for MSI/MSI-X mechanisms
Applies to Crypto enabled OPNs.


19
DDR4 SDRAM On-Board Memory
The DPU incorporates 16GB @ 3200MT/s single DDR4 channel,64bit + 8bit ECC, solder-down memory.
NC-SI Management Interface
The DPU enables the connection of a Baseboard Management Controller (BMC) to a set of Network Interface Controller (NICs) for the purpose of enabling
out-of-band remote manageability. The NC-SI management is supported over RMII and has a connector on the DPU. Please refer toNC-SI Management
Interfacefor pins.
UART Interface Connectivity
A UART debug interface is available on the DPU cards via 3 pins of a 30-pin NC-SI connector (described in NC-SI Management Interface). The connectivity is
shown in the following table:
NC-SI Connector Pin # Signal on DPU
30 BF_UART0_RX
28 BF_UART0_TX
26 GND
The UART interface is compliant with TTL 3.3V voltage level. AUSB to UART cable that supports TTL voltage levels should be used to connect the UART
Interface for Arm console access - see example below.

20
USB Interfaces
The controllers use a mini-USB Type B connector to load operating system images.
1GbE OOB Management Interface
The DPU incorporates a 1GbE RJ45 out-of-band port that allows the network operator to establish trust boundaries in accessing the management function
to apply it to network resources. It canalsobe used to ensure management connectivity (including the ability to determine the status of any network
component) independent of the status of other in-band network components.
It is prohibited to directly connect any RS-232 cable! Only TTL 3.3V voltage level cables are supported.

The USB to UART cable is not used for NC-SI management purposes.

10Mb/s and 100Mb/s modes are not supported on this interface.
