Nvidia BlueField-2 BF2500 User manual

Exported on Oct/31/2023 01:37 PM
NVIDIA BlueField-2 BF2500 Ethernet DPU Controller User Manual

2
Table of Contents
Introduction............................................................................................................................................ 9
System Requirements ........................................................................................................................................... 9
Package Contents ...............................................................................................................................................10
Card Package ................................................................................................................................................ 10
Accessories Kit .............................................................................................................................................. 10
Product Overview ...............................................................................................................................................11
Ideal Solution for JBOF and JBOD Systems.............................................................................................................. 11
Features and Benefits ...................................................................................................................................... 11
Supported Interfaces ................................................................................................................................ 14
Layout and Interfaces ..........................................................................................................................................14
Interfaces Detailed Description...............................................................................................................................17
DPU............................................................................................................................................................ 17
Encryption ............................................................................................................................................... 18
Ethernet QSFP56 Interface ................................................................................................................................ 18
PCI Express Interface....................................................................................................................................... 18
DDR4 SDRAM On-Board Memory........................................................................................................................... 19
NC-SI Management Interface .............................................................................................................................. 19
UART Interface Connectivity .............................................................................................................................. 19
USB Interfaces ............................................................................................................................................... 20

3
1GbE OOB Management Interface ........................................................................................................................ 20
1GbE OOB Management LEDs Interface.............................................................................................................. 21
RTC Battery .................................................................................................................................................. 21
eMMC Interface.............................................................................................................................................. 21
External PCIe Power Supply Connector.................................................................................................................. 21
Networking Ports LEDs Interface ......................................................................................................................... 22
Pin Description ....................................................................................................................................... 23
PCI Express Interface...........................................................................................................................................23
Power Sequencing........................................................................................................................................... 28
NC-SI Management Interface ..................................................................................................................................28
External Power Supply Connector (6-pin ATX Connector).................................................................................................35
Hardware Installation ............................................................................................................................... 36
Safety Warnings .................................................................................................................................................36
Installation Procedure Overview..............................................................................................................................36
System Requirements ..........................................................................................................................................37
Hardware Requirements ................................................................................................................................... 37
Airflow Requirements ...................................................................................................................................... 37
Software Requirements .................................................................................................................................... 38
Safety Precautions ..............................................................................................................................................38
Pre-Installation Checklist ......................................................................................................................................38
Installation Instructions........................................................................................................................................39

4
Cables and Modules.............................................................................................................................................39
Networking Cables .......................................................................................................................................... 39
RJ45 Cable for the 1GbE OOB Management Interface................................................................................................. 40
USB Interface Cabling ...................................................................................................................................... 41
Bring-Up and Driver Installation ................................................................................................................... 42
BlueField Software..............................................................................................................................................42
Software On eMMC..............................................................................................................................................43
Yocto Distribution Installation ............................................................................................................................ 43
PXE Server Configuration on Host Side ......................................................................................................................44
Installing Linux on BlueField Controller Card...............................................................................................................44
Software Requirements .................................................................................................................................... 45
Preparing Host-Side Environment .................................................................................................................... 45
Setup Procedure With Installation Script............................................................................................................ 46
Setup Procedure Without Installation Script........................................................................................................ 46
Flashing BlueField Controller Card Bootloader Code .................................................................................................. 50
Opening Terminal Connection to BlueField Controller Card...................................................................................... 51
Using Initial Install Bootstream ....................................................................................................................... 51
Installing CentOS 7.4 on BlueField Controller Card.................................................................................................... 53
Full PXE Boot Installation.............................................................................................................................. 53
Non-PXE Boot Installation ............................................................................................................................. 60
MLNX_OFED Installation........................................................................................................................................60

5
Installing MLNX_OFED on Arm Cores ..................................................................................................................... 61
Prerequisite Packages for Installing MLNX_OFED................................................................................................... 61
Removing Pre-installed Kernel Module .............................................................................................................. 62
Installing MLNX_OFED on the BlueField Controller Card.......................................................................................... 62
Updating BlueField Controller Card Firmware ..................................................................................................... 64
Troubleshooting ...................................................................................................................................... 66
Specifications......................................................................................................................................... 67
MBF2H516B-CEEOT / MBF2H516B-CENOT Specifications..................................................................................................67
BlueField-2 BF2500 Mechanical Drawing and Dimensions.................................................................................................68
Finding the MAC on the Controller Card.......................................................................................................... 70
Thermal Sensors...................................................................................................................................... 71
Document Revision History ......................................................................................................................... 72



6
About This Manual
It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and a step-by-step plan of
how to bring up the BlueField-2 DPU.
EOL'ed (End of Life) Ordering Part Numbers
OPN F
o
r
m
F
a
c
t
o
r
Series/ Core Speed Max Speed No. of Ports PCIe Support Crypto Secure Boot 1GbE OOB O
n
-
b
o
a
r
d
D
D
R
M
e
m
o
r
y
On-board eMMC
Memory
MBF2H516B-CEEOT F
H
H
L
*
P-Series / 2.75GHz 100GbE 2xQSFP56 Gen 4.0 x16 Crypto Enabled - √1
6
G
B
64GB
MBF2H516B-CENOT F
H
H
L
P-Series / 2.75GHz 100GbE 2xQSFP56 Gen 4.0 x16 Crypto Disabled - √1
6
G
B
64GB
*Full-Height Half-Length

7
•
•
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic familiarity with Ethernet network and architecture
specifications.
Technical Support
Customers who purchased NVIDIA products directly from NVIDIA are invited to contact usthrough the following methods:
E-mail:[email protected]
Enterprise Support page:https://www.nvidia.com/en-us/support/enterprise
Related Documentation
InfiniBand Architecture Specification InfiniBand Trade Association (IBTA) InfiniBand® specification Release 1.3.1, November 2, 2016 and Vol. 2, Release 1.4 ,
and Vol 2 - Release 1.5.
IEEE Std 802.3 Specification IEEE Ethernet specification.
PCI Express Specifications Industry Standard PCI Express Base and Card Electromechanical Specifications.
NVIDIA LinkX Interconnect Solutions The NVIDIA® LinkX® product family of cables and transceivers provide the industry’s broadest portfolio of QDR/FDR10
(40Gb/s), FDR (56Gb/s), EDR/HDR100 (100Gb/s), HDR (200Gb/s) and NDR (400Gb/s) cables, including Direct Attach
Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths
from 0.5m to 10km. In addition to meeting IBTA standards, NVIDIA tests every product in an end-to-end environment
ensuring a Bit Error Rate of less than 1E-15.
BlueField DPU Platform BSP Documentation This guide provides product release notes as well as information on the BSP and how to develop and/or customize
applications, system software, and file system images for the BlueField platform.
DOCA SDK Software Documentation NVIDIA DOCA SDK software.
Document Conventions
When discussing memory sizes, GB and GBytes are used in this document to mean size in giga-bytes. The use of Gb or Gbits (small b) indicates size in giga-
bits. In this document PCIe is used to mean PCI Express.

8
Revision History
A list of the changes made to this document are provided inDocument Revision History.

9
•
•
•
Introduction
This is the User Manual for NVIDIA® BlueField®-2 BF2500 DPU Controller. This document provides details of the product interfaces, specifications, required
software and firmware for operating the board, and a step-by-step plan of how to bring up the BlueField-2 BF2500 DPU Controller.
System Requirements
Item Description
Main-boardPCI Express slot x16 Gen 4.0 slot.
System Power Supply Minimum 75W or greater system power supply for all cards.
These PCIe Gen 4.0 x16 DPU controllersrequire additional 75W through a supplementary 6-pin ATX power supply
connector.
NOTE:The connector is not included in the package. It should be part of system wiring or it can be ordered separately as a
system accessory.
Operating System BlueField-2 DPU is shipped with Ubuntu – a Linux commercial operating system – which includes the NVIDIAOFED stack
(MLNX_OFED), and is capable of running all customer-based Linux applications seamlessly. BlueField-2 DPU also supports
CentOS and has an out-of-band 1GbE management interface.For more information, please refer to the DOCA SDK
documentation or NVIDIA BlueField-2 Software User Manual.
Connectivity Interoperable with 1/10/25/40/50/100/200 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support
For detailed information, see Specifications.

10
Package Contents
Before installing your new system, unpack it and check against the below tables that all the parts have been sent. Check the parts for visible damage that
may have occurred during shipping.
Card Package
Item Description
Cards 1x BlueField-2 DPU Controller card with an assembled tall bracket
Accessories Kit
The accessories kit should be ordered separately. Earlier controller versions require the kit OPN MBF20-DKIT, while newer versions require kit OPN MBF25-
DKIT.
Kit OPN Contents
MBF20-DKIT 1x USB 2.0 Type A to mini-USB Type B cable
1x USB 2.0 Type A to 30pin Flat Socket
If anything is damaged or missing, contact your reseller.

These DPU controllers, you need a6-pin ATX power supply connector cable to activate the card. The cable is not included in the package. For
further details, please refer toExternal PCIe Power Supply Connector.
