Nvidia BlueField BF1500 User manual

Exported on Oct/31/2023 01:39 PM
NVIDIA BlueField BF1500 Ethernet Controller Card User Guide

2
Table of Contents
Introduction............................................................................................................................................ 7
Product Overview ................................................................................................................................................ 7
Features and Benefits ........................................................................................................................................... 8
Operating Systems/Distributions .............................................................................................................................11
Connectivity .....................................................................................................................................................11
Package Contents ...............................................................................................................................................11
Supported Interfaces ................................................................................................................................ 12
BF1500 Controller Card Layout and Interface Information ...............................................................................................12
Interfaces Detailed Description...............................................................................................................................14
BlueField System-on-Chip (SoC) .......................................................................................................................... 14
Ethernet QSFP28 Interface ................................................................................................................................ 14
PCI Express Interface....................................................................................................................................... 14
DDR4 SDRAM On-Board Memory........................................................................................................................... 15
NC-SI Management Interface .............................................................................................................................. 15
3 Pin Header Interface ..................................................................................................................................... 16
UART Interface Connectivity .......................................................................................................................... 16
Mini USB Type B Interface ................................................................................................................................. 17
JTAG CoreSight 10 Interface .............................................................................................................................. 17
LEDs Interface ............................................................................................................................................... 17

3
Pin Description ....................................................................................................................................... 19
PCI Express Pins Description...................................................................................................................................19
NC-SI Management Interface ..................................................................................................................................22
JTAG CoreSight 10 ..............................................................................................................................................24
Hardware Installation ............................................................................................................................... 26
Safety Warnings .................................................................................................................................................26
Installation Procedure Overview..............................................................................................................................26
System Requirements ..........................................................................................................................................27
Hardware Requirements ................................................................................................................................... 27
Airflow Requirements .................................................................................................................................. 27
Software Requirements .................................................................................................................................... 28
Safety Precautions ..............................................................................................................................................28
Unpacking the Package ........................................................................................................................................29
Pre-Installation Checklist ......................................................................................................................................29
Bracket Replacement Instructions ...........................................................................................................................29
Installation Instructions........................................................................................................................................30
Cables and Modules.............................................................................................................................................33
Cable Installation ........................................................................................................................................... 33
Identify the BF1500 Controller Card in your System ......................................................................................................34
Network Adapter Environment............................................................................................................................ 34
On Windows.............................................................................................................................................. 34

4
On Linux .................................................................................................................................................. 35
Uninstalling the BF1500 Controller Card ....................................................................................................................35
Bring-Up and Driver Installation ................................................................................................................... 38
Troubleshooting ...................................................................................................................................... 39
General Troubleshooting .......................................................................................................................................39
Specifications......................................................................................................................................... 40
MBF1L516B-CSNAT Specifications.............................................................................................................................40
Mechanical Drawing and Dimensions.........................................................................................................................42
Mechanical Drawing of the BF1500 Controller Card - Component Side............................................................................. 42
Mechanical Drawing of the BF1500 Controller Card - Print Side..................................................................................... 43
Bracket Mechanical Drawing ..................................................................................................................................44
Finding the MAC on the Adapter Card ............................................................................................................ 45
Document Revision History ......................................................................................................................... 46



5
•
•
About This Manual
This User Manual describes NVIDIA® BlueField® Ethernet BF1500 Controller Card. It provides details as to the interfaces of the board, specifications,
required software and firmware for operating the board, and a step-by-step plan of how to bring up the BlueField Controller Card.
EOL'ed (End of Life) Ordering Part Numbers
The table below provides the ordering part numbers (OPNs) for available BlueField BF1500 Controller Card.
NVIDIA SKU OPN Marketing Description
900-9D1A2-0056-SN0 MBF1L516B-CSNAT BlueField® Controller card, Dual-Port 100GbE QSFP28, BlueField® G-
Series 16 cores, PCIe Gen3.0/4.0 x16, Crypto disabled, 16GB on-board
DDR, FHHL, Single Slot, Tall Bracket
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic familiarity with Ethernet network and architecture
specifications.
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough the following methods:
URL:https://www.nvidia.com> Support
E-mail:[email protected]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first seekassistance through their reseller.
Related Documentation
IEEE Std 802.3
Specification
IEEE Ethernet specification athttp://standards.ieee.org
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications athttps://pcisig.com/specifications

6
IEEE Std 802.3
Specification
IEEE Ethernet specification athttp://standards.ieee.org
Mellanox LinkX
Interconnect
Solutions
Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the performance of High-Performance Computing networks, requiring high-bandwidth, low-
latency connections between compute nodes and switch nodes. Mellanox offers one of the industry’s broadest portfolio of QDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/
HDR100 (100Gb/s), and HDR (200Gb/s) cables, including Direct Attach Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a
wide range of lengths from 0.5m to 10km. In addition to meeting IBTA standards,Mellanox tests every product in an end-to-end environment ensuring a Bit Error Rate of
less than 1E-15.Read more athttps://www.nvidia.com/en-us/networking/interconnect/
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega Bytes. The use of Mb or Mbits (small b) indicates size in
mega bits. In this document PCIe is used to mean PCI Express.
Revision History
A list of the changes made to this document are provided inDocument Revision History.

7
Introduction
Product Overview
The BlueField BF1500 Controller Card is the perfect solution for managing NVMestorage drives in storage and hyper-converged systems. The BF1500
Controller Card delivers thehighest NVMe-oF target performance, reducing TCO and increasing ROI. It can transform existingJBOF systems into NVMe-oF
compliant solutions, simply by plugging the card into an existingPCIe slot. The card’s small form factor allows customers to install multiple controller
cardsin a single system to support a larger number of SSDs and high-availability storage architecture.
BlueField Multicore System-on-chip (SoC)
BlueField SoC is a highly integrated and efficient controller, optimized for NVMe storagesystems, Network Functions Virtualization (NFV), Cloud, and AI
workloads.BlueField SoC integrates all the discrete components of a storage system appliance into a singlechip (CPU, PCIe switch, and Network
controller), making it the premier SoC solution for buildingJust-a-Bunch-Of-Flash (JBOF) systems, All-Flash-Array, and storage appliances for NVMeover-
Fabrics.
With an integrated NVMe-oF offload accelerator, the BlueField SoC gives the BF1500 ControllerCard a superior performance advantage over existing JBOF
systems, significantly reducingstorage transaction latency, while increasing IOPs (I/O operations per second).
BlueField SoC also offers cost-effective and integrative solutions for Machine Learning appliances,enabling efficient data delivery for real-time analytics
and data insights based on superiorRDMA and GPUDirect® RDMA technologies. Support for PCIe Gen4.0 helps future-proof thenext generation of high-
performance GPU cards.
BlueField BF1500 Ethernet Controller Card Card
Part Number MBF1L516A-CSNAT
Form Factor Full-Height, Half-Length (FHHL)
The BF1500 Controller card connectivity to NVMe SSD devices is possible through the PCIe switch.

The BlueField Controller Card should be installed only in a JBOF System as it functions as a PCIe root-complex (RC) initiating PCIe bus operations.
Installing it in a regular host system may damage the card.


8
Ethernet Data Rate 1/10/25/40/50/100 GbE
Network Connector Type Dual-port QSFP28
PCI Express Connectors PCIe Gen 3.0 (8GT/s) / Gen 4.0 (16GT/s) x16 lanes
On-board DDR4 Memory(a) Single-channel with 8 DDR4 8 bit + ECC (64bit + 8bit ECC)
16GB @ 2400MT/s
BlueField IC OPN MT41M16E23A0-CDCR-TTEV
RoHS RoHS Compliant
Crypto Disabled
BlueField IC Cores
Series and NIC/Arm
Frequency
16 Cores
G-Series - 275/800MHz
For more detailed information seeSpecifications.
Features and Benefits
Feature Description
PCI Express (PCIe) Uses PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x16 edge connector. Gen 1.1 and 2.0 compatible.
Up to 100 Gigabit
Ethernet
NVIDIA Controllers comply with the following IEEE 802.3 standards:
– 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
– IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
– IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature
availability.


9
•
•
•
•
•
Feature Description
– IEEE 802.3ba 40 Gigabit Ethernet
– IEEE 802.3by 25 Gigabit Ethernet
– IEEE 802.3ae 10 Gigabit Ethernet
– IEEE 802.3az Energy Efficient Ethernet
– IEEE 802.3ap based auto-negotiation and KR startups
– Proprietary Ethernet protocols (20BASE-R2)
– IEEE 802.3ad, 802.1AX Link Aggregation
– IEEE 802.1Q, 802.1P VLAN tags and priority
– IEEE 802.1Qau (QCN)
– Congestion Notification
– IEEE 802.1Qaz (ETS)
– IEEE 802.1Qbb (PFC)
– IEEE 802.1Qbg
– IEEE 1588v2
– Jumbo frame support (9.6KB)
On-board Memory Quad SPI NOR FLASH - includes 128Mbit for Firmware image (W25Q128FVSIG device by Winbond-Nuvoton).
UVPS EEPROM - includes 512Kbit (M24512-DFMC6TG by STMICROELECTRONICS).
FRU EEPROM - includes a 128Kbit (M24128-DFMC6TG by STMICROELECTRONICS)
eMMC - x8 16GB NAND flash Memory for SoC BIOS.
DDR4 SDRAM - single-channel DDR4 SDRAM memory with 8 components of 8 bit + ECC (64bit + 8bit ECC), 8/16GB @2400MT/s.
BlueField SoC The BlueField SoC integrates four, eight or sixteen 64-bit Armv8 A72 cores interconnected by a coherent mesh network, one DRAM controller, a dual port RDMA intelligent
Ethernet adapter supporting up to 100Gb/s, an embedded PCIe switch with endpoint and root complex functionality, and up to 8 lanes of PCIe Gen 3.0/4.0.
Overlay Networks In order to better scale their networks, data center operators often create overlay networks that carry traffic from individual virtual machines over logical tunnels in
encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher
loads on the host CPU. NVIDIA BF1500 Controller Card effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and
de-capsulate the overlay protocol.otocol.
RDMA and RDMA
overConverged
Ethernet (RoCE)
NVIDIA BF1500 Controller Card, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, delivers low-latency and high-
performance over Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as advanced congestion control hardware mechanisms, RoCE provides
efficient low-latency RDMA services over Layer 2 and Layer 3 networks.
NVIDIA
PeerDirect™
PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from
GPU to CPU), and therefore significantly reduces application run time. NVIDIA BF1500 Controller Cardadvanced acceleration technology enables higher cluster efficiency
and scalability to tens of thousands of nodes.

10
•
•
•
•
•
Feature Description
Quality of Service
(QoS)
Support for port-based Quality of Service enabling various application requirements for latency and SLA.
Storage
Acceleration
A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can
leverage RDMA for high-performance storage access.
NVMe over Fabric offloads for the target machine
T10-DIF Signature Handover
BlueField BF1500 Controller Card may operate as a co-processor offloading specific storage tasks from the host, isolating part of the storage media from the host, or
enabling abstraction of software-defined storage logic
using the BlueField Arm cores. On the storage initiator side, BlueField BF1500 card can prove an efficient solution for hyper-converged systems to enable the host CPU to
focus
on compute while all the storage interface is handled through the Arm cores.
NVMe-oF Nonvolatile Memory Express (NVMe) over Fabrics is a protocol for communicating block storage IO requests over RDMA to transfer data between a host computer and a
target solid-state storage device or system over a network. BlueField BF1500 Controller Card may operate as a co-processor offloading specific storage tasks from the host
using its powerful NVMe over Fabrics Offload accelerator.
SR-IOV NVIDIA BF1500 Controller Card SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VM) within the server.
High-
PerformanceAcce
lerations
Tag Matching and Rendezvous Offloads
Adaptive Routing on Reliable Transport
Burst Buffer Offloads for Background Checkpointing
GPU Direct The latest advancement in GPU-GPU communications is GPUDirect RDMA. This new technology provides a direct P2P (Peer-to-Peer) data path between the GPU Memory
directly to/from the NVIDIA HCA devices. This provides a significant decrease in GPU-GPU communication latency and completely offloads the CPU, removing it from all
GPU-GPU communications across the network. NVIDIA BF1500 Controller Card uses high-speed DMA transfers to copy data between P2P devices resulting in more efficient
system applications
Security
Accelerators
A consolidated compute and network solution based on BF1500 Controller Card achieves significant advantages over a centralized security server solution. Standard
encryption protocols and security applications can leverage BlueField compute capabilities and network offloads for security application solutions such as Layer4 Statefull
Firewall.

11
•
•
•
Operating Systems/Distributions
The BF1500 Controller Card is shipped with Linux based Operating System burned on it which includes all needed drivers. For more information, please
refer to the BlueField Software User Manual.
Connectivity
Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support
Package Contents
Category Quantity Item
Cards 1 BF1500 Controller Card
Cables 1 USB 2.0 Type A to Mini USB Type B cable
1 3 pin UART to USB cable
Accessories 1 Tall bracket (shipped assembled)
1 Short bracket

12
Supported Interfaces
BF1500 Controller Card Layout and Interface Information
The below figures show the component and print sides of BF1500 Controller Card. Each numbered interface that is referenced in the figures is described in
the following table with a link to detailed information.
BF1500 Controller Card Interfaces - Component Side
BF1500 Controller Card Interface - Print Side
The BF1500 Controller Card include special circuits to protect from ESD shocks to the card/server when plugging copper cables

The below figures are for illustration purposes only and might not reflect the current revision of the BF1500 Controller Card.


13
Item Interface Description
1"BlueField System-on-Chip (SoC)”Mellanox BlueField SoC 16 cores
2“Ethernet QSFP28 Interface” Ethernet traffic is transmitted through the BF1500 Controller Card QSFP28 connectors. The
QSFP28 connectors allow for the use of modules, optical and passive cable interconnect solutions
3“PCI Express Interface” PCIe Gen 3.0/4.0 through an x16 edge connector
4“DDR4 SDRAM On-Board Memory” Single channel with 8 DDR4 8 bit + ECC (64bit + 8bit ECC)
16GB @ 2400MT/s
5“NC-SI Management Interface” BMC connection for remote management
6 Access to Arm console.
6“Mini USB Type B Interface” Mounted on the Controller Card for OS image loading
7“JTAG CoreSight 10 Interface” Arm debug tool
8“LEDs Interface” One bi-color LED per port for the link and physical status

14
•
•
•
•
Interfaces Detailed Description
BlueField System-on-Chip (SoC)
BlueField is a Mellanox family of advanced SoC solutions that integrate a coherent mesh of 64-bit Arm v8 A72 cores, a ConnectX network adapter front-end
and a PCI Express switch into a single chip. The powerful SoC architecture includes an ARMv8 multicore processor array and enables customers to develop
sophisticated applications and highly differentiated feature sets. BlueField leverages the rich ARM software ecosystem and introduces the ability to offload
the x86 software stack.
At the heart of BlueField is the ConnectX-5 network offload controller with RDMA and RDMA over Converged Ethernet (RoCE) technology, delivering cutting-
edge performance for networking and storage applications such as NVMe over Fabrics. Advanced features include an embedded virtual switch with
programmable access lists (ACLs), transport offloads and stateless encaps/decaps of NVGRE, VXLAN, and MPLS overlay protocols.
Ethernet QSFP28 Interface
The network ports ofBF1500 Controller Card are compliant with the IEEE 802.3 Ethernet standards listed inFeatures and Benefits. Ethernet traffic is
transmitted through the cards' QSFP28 connectors.
PCI Express Interface
The BF1500 Controller Card supports PCI Express Gen 3.0/4.0 (1.1 and 2.0 compatible) through an x16 edge connector. The device is a slave responding to
the PCIe bus operations (end-point). Please refer toPCI Express Pins Descriptionfor pinouts description.
The following lists PCIe interface features:
PCIe Gen 3.0 and 4.0 compliant, 2.0 and 1.1 compatible
2.5, 5.0, or 8.0, or 16.0 GT/s link rate x16 lanes
Auto-negotiates to x16, x8, x4, x2, or x1
Support for MSI/MSI-X mechanisms

15
DDR4 SDRAM On-Board Memory
The BF1500 Controller Card incorporates a single channel DDR4 SDRAM memory with 8 components of 8 bit + ECC (64bit + 8bit ECC), 16GB @2400MT/s.
NC-SI Management Interface
The BlueField Controller Card enables the connection of a Baseboard Management Controller(BMC) to a set of Network Interface Controller (NICs) for the
purpose of enabling out-of-bandremote manageability. The NC-SI management is supported over RMII and has a connector onthe Controller Card.Please
refer toNC-SI Management Interfacefor pins.

16
•
•
•
3 Pin Header Interface
The 3 pin header interface enables access to the Arm console through UART0 through the followingpins:
Pin # 1 - UART0 RX
Pin # 2 -GND
Pin # 3 - UART0 TX
UART Interface Connectivity
The USB to UART cable that supports TTL voltage levels can be used to connect the UART Interface. See an example of TTL 3.3V voltage levels cable in the
below figure.
USBto UART Cable - Example
It is prohibited to directly connect any RS-232 cable! Only TTL 3.3V voltage level cables are supported!


17
Mini USB Type B Interface
The BF1500 Controller Card uses a Mini USB Type B. The USB connector is mounted on the card. The purpose of the USB connector is loading operating
system images for the operating systems. In order to use this interface, please use a standard Mini USB Type-B to Mini USB Type A cable.
JTAG CoreSight 10 Interface
The BF1500 Controller Card uses JTAG CoreSight 10-Pin Header. The purpose of the JTAG CoreSight 10 is for debugging the Arm with Arm DSTREAM debug
tool. For JTAG CoreSight 10 pins, please refer JTAG CoreSight 10.
LEDs Interface
There is a one bi-color I/O LED per port to indicate link and physical status.
Physical and Logical Link Indications (Ethernet Mode)
LED Color and State Description
Off A link has not been established
Blinking Amber(a) 4 Hz blinking amber indicates a problem with the link
Solid Green Indicates a valid link with no active traffic
Blinking Green Indicates a valid logical link with active traffic

18
Note a. 1Hz Blinking amber occurs due to running a beacon command for locating the adapter card.

19
Pin Description
PCI Express Pins Description
The below table lists the PCI Express pins description. For further details, please refer toPCI Express Interface.
PCI Express Pin Description
Pin # Signal Name Description Pin # Signal Name Description
A1 PRSNT1# Mechanical Present B1 12V
A2 12V B2 12V
A3 12V B3 12V
A4 GND B4 GND
A5 TCK JTAG B5 SMCLK Host SMBus
A6 TDI JTAG B6 SMDAT Host SMBus
A7 TDO JTAG B7 GND
A8 TMS JTAG B8 3.3V
A9 3.3V B9 TRST# JTAG
A10 3.3V B10 3.3V_AUX
A11 PERST# PCIe Reset B11 WAKW#/RSVD

20
Pin # Signal Name Description Pin # Signal Name Description
A12 GND B12 RSVD
A13 REFCLK+ Host Reference Clock B13 GND
A14 REFCLK- Host Reference Clock B14 PETP0
A15 GND B15 PETN0
A16 PERP0 B16 GND
A17 PERN0 B17 RSVD
A18 GND B18 GND
A19 RSVD B19 PETP1
A20 GND B20 PETN1
A21 PERP1 B21 GND
A22 PERN1 B22 GND
A23 GND B23 PETP2
A24 GND B24 PETN2
A25 PERP2 B25 GND
A26 PERN2 B26 GND
A27 GND B27 PETP3