Nvidia BlueField BF1600 User manual

Exported on Oct/31/2023 01:39 PM
NVIDIA BlueField BF1600 InfiniBand/Ethernet Controller Card User
Manual

2
Table of Contents
Introduction............................................................................................................................................ 9
Product Overview ................................................................................................................................................ 9
BlueField Multicore System-on-chip (SoC) ..................................................................................................................10
Features and Benefits ..........................................................................................................................................11
Operating Systems/Distributions .............................................................................................................................13
Package Contents ...............................................................................................................................................14
BF1600 Controller Card with Auxiliary Connection Card.............................................................................................. 14
BF1600 Controller Card without Auxiliary Connection Card.......................................................................................... 15
Supported Interfaces ................................................................................................................................ 16
Layout and Connector Information...........................................................................................................................16
BlueField BF1600 Controller Card ........................................................................................................................ 16
PCIe Auxiliary Connection Card........................................................................................................................... 20
PCIe Auxiliary Connection Card - Print Side ........................................................................................................ 22
Interfaces ........................................................................................................................................................23
BlueField System On Chip (SoC) .......................................................................................................................... 23
Ethernet QSFP56 Interfaces ............................................................................................................................... 23
QSFP28 Port Over-current Protection................................................................................................................ 23
PCI Express Interfaces...................................................................................................................................... 24
PCIe x16 Gen 3.0/4.0 Edge Connector............................................................................................................... 24

3
PCIe Gen 3.0 through Two CABLINE-CA II PLUS Connectors....................................................................................... 24
NCSI Management Interface ............................................................................................................................... 25
USB Connector............................................................................................................................................... 26
UART0 Interface............................................................................................................................................. 26
I²C0 Interface ............................................................................................................................................... 27
JTAG CoreSight 10 .......................................................................................................................................... 27
External PCIe Power Supply Interfaces .................................................................................................................. 28
External PCIe Power Supply Interface on the BF1600 Controller Card ......................................................................... 28
External PCIe Power Supply Interface on the Auxiliary PCIe Connection Card ................................................................ 29
CPLD Interface .............................................................................................................................................. 30
eMMC Interface.............................................................................................................................................. 30
DDR4 SODIMM Memory...................................................................................................................................... 31
SPI Flash Memory ........................................................................................................................................... 32
Networking LED Interfaces ................................................................................................................................ 32
CPLD Revision LEDs Interface ............................................................................................................................. 32
CABLINE-CA II PLUS Connector LED Interfaces ......................................................................................................... 33
Power-Good Signal LED Interface......................................................................................................................... 34
PCI Link Status LEDs ........................................................................................................................................ 34
Pin Description ....................................................................................................................................... 35
PCI Express Interface...........................................................................................................................................35
PCIe x16 Gen 3.0/4.0 Edge Connector................................................................................................................... 35

4
PCIe x16 Edge Connector Pins Description - Print Side............................................................................................ 35
PCIe x16 Edge Connector Pinouts Description - Component Side................................................................................ 41
PCIe Gen 3.0 through Two CABLINE-CA II PLUS Connectors....................................................................................... 47
CABLINE-CA II PLUS Connector - Component Side ................................................................................................. 48
CABLINE-CA II PLUS Connector - Print Side.......................................................................................................... 52
Power Sequencing........................................................................................................................................... 57
NCSI Management Interface ...................................................................................................................................57
External Power Supply Interfaces ............................................................................................................................59
External +12V Mechanical Pinout Diagram .............................................................................................................. 60
UART0 Interface.................................................................................................................................................60
JTAG CoreSight 10 Interface ..................................................................................................................................61
DDR4 DRAM Memory Interface ................................................................................................................................62
Cables and Cabling Configurations ................................................................................................................ 64
UART Interface Cabling ........................................................................................................................................64
USB Interface Cabling ..........................................................................................................................................65
Configuration Scenarios............................................................................................................................. 67
Configuration 1: Direct Connectivity to 4-8 NVMe SSDs (No External PCIe Switch)...................................................................67
Configuration 2: Direct Connectivity to 8-16 NVMe SSDs with (No External PCIe Switch) ...........................................................69
Configuration 3: Connectivity to 8-16 NVMe SSDs (with External PCIe Switch) .......................................................................70
Configuration 4: Connectivity to 24 NVMe SSDs (with External PCIe Switch)..........................................................................70
Clocks Distribution..............................................................................................................................................71

5
Thermal Sensors...................................................................................................................................... 72
Hardware Installation ............................................................................................................................... 73
Installation Procedure Overview..............................................................................................................................73
System Requirements ..........................................................................................................................................74
Hardware Requirements ................................................................................................................................... 74
Airflow Requirements .................................................................................................................................. 74
Software Requirements .................................................................................................................................... 75
Pre-Installation Checklist ......................................................................................................................................75
Bracket Replacement Instructions ...........................................................................................................................76
Removing the Existing Bracket............................................................................................................................ 76
Installing the New Bracket ................................................................................................................................ 76
Installation Instructions........................................................................................................................................76
BF1600 Controller Card Installation Instructions - PCIe x16.......................................................................................... 77
Connect the BF1600 Controller Card in an available PCI Express slot on the JBOF system. ................................................ 77
Secure the BF1600 Controller Card to the JBOF system. ......................................................................................... 79
BF1600 Controller and Auxiliary Connection Cards Installation Instructions - PCIe x32 ......................................................... 79
Connect the BF1600 Controller Card with the Auxiliary connection card using the supplied CABLINE-CAII PLUS harnesses........... 80
Connect the BF1600 Controller Card in an available PCI Express slot on the JBOF system. ................................................ 81
Connect the PCIe Auxiliary Connection Card in an available PCI Express slot on the JBOF system. ....................................... 82
Secure the BF1600 Controller and PCIe Auxiliary Connection Cards to the JBOF system.................................................... 83
Cables and Modules.............................................................................................................................................84

6
Networking Cables .......................................................................................................................................... 84
Bring-Up and Driver Installation ................................................................................................................... 86
Troubleshooting ...................................................................................................................................... 87
General Troubleshooting .......................................................................................................................................87
Specifications......................................................................................................................................... 88
Dual-slot Controller Cards Specifications ...................................................................................................................88
Single-slot Controller Card Specifications...................................................................................................................89
LED Interfaces ...................................................................................................................................................91
Networking LED Interfaces ................................................................................................................................ 91
CPLD Revision LEDs Interface ............................................................................................................................. 91
CABLINE-CA II PLUS Connector LED Interfaces ......................................................................................................... 92
BF1600 Controller Card Cabline-CA II Plus Connector LED ....................................................................................... 93
PCIe Auxiliary Connection Card Cabline-CA II Plus Connector LED .............................................................................. 93
Power-Good Signal LED Interface......................................................................................................................... 93
Finding the MAC on the Controller Card.......................................................................................................... 94
Document Revision History ......................................................................................................................... 99



7
•
About This Manual
This User Manual describes NVIDIA® BlueField® BF1600 Controller Card. It provides details as to the interfaces of the board, specifications, required
software and firmware for operating the card, hardware installation, driver installation and bring-up instructions.
Overview of Document Content
Section Section Description
Introduction Provides a general overview of the BlueField BF1600 Controller Cards and discusses its benefits and features.
Supported Interfaces Provides descriptions of all supported interfaces.
Pin Description Provides interfaces’ Pinouts description.
Configuration Scenarios Provides examples of different configuration scenarios for NVMe SSD connectivity.
Cables and Cabling Configurations Describes the required cables and cabling configurations.
Thermal Sensors Describes the available thermal sensors on the BlueField SoC
Hardware Installation Describes the procedures for installing and uninstalling the Controller Card.
Bring-Up and Driver Installation Describes driver installation and bring-up instructions
Troubleshooting Describes potential system problems - you can use this troubleshooting information to identify and resolve
the problem.
Specifications Lists the physical, electrical, operational and regulatory specifications of the BlueField
Intended Audience
This manual is intended for the installer and user of these cards.The manual assumes basic familiarity with Ethernet network and architecture
specifications.
Technical Support
Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough the following methods:
URL:https://www.nvidia.com> Support

8
• E-mail:[email protected]
Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails regarding Technical Support.
Customers who purchased Mellanox products through a Mellanox approved reseller should first seekassistance through their reseller.
Related Documentation
IEEE Std 802.3
Specification
IEEE Ethernet specification at http://standards.ieee.org
PCI Express
Specifications
Industry Standard PCI Express Base and Card Electromechanical Specifications at https://pcisig.com/specifications
Mellanox LinkX
Interconnect
Solutions
Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the performance of High-Performance Computing networks, requiring high-bandwidth, low-
latency connections between compute nodes and switch nodes. Mellanox offers one of the industry’s broadest portfolio of QDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/
HDR100 (100Gb/s), and HDR (200Gb/s) cables, including Direct Attach Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a
wide range of lengths from 0.5m to 10km. In addition to meeting IBTA standards,Mellanox tests every product in an end-to-end environment ensuring a Bit Error Rate of
less than 1E-15.Read more at https://www.nvidia.com/en-us/networking/interconnect/
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in MegaBytes. The use of Mb or Mbits (small b) indicates size in
MegaBits. In this document, PCIe is used to mean PCI Express.
Revision History
A list of the changes made to this document are provided inDocument Revision History.

9
Introduction
This is the User Manual for BlueField® BF1600 Controller Card. This document contains the complete installation instructions and specifications for the
BF1600 Controller Cards listed in the below table.
Ordering Part Numbers
The below table provides the ordering part numbers (OPN) for the available BlueField BF1600 Controller Cards:
OPN Marketing Description
MBF1M606A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, PCIe Gen4.0 x16, BlueField® E-Series 16 cores, Crypto disabled, No
Memory DIMM, FH¾L, Single Slot, Tall Bracket
MBF1M606A-ESNAT BlueField® Controller card, Dual Port InfiniBand/Ethernet 100Gb/s QSFP28, PCIe Gen4.0 x16, BlueField® E-Series 16 cores,
Crypto disabled, No Memory DIMM, FH¾L, Single Slot, Tall Bracket
MBF1M616A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, PCIe Gen4.0 x16, BlueField® E-Series 16 cores, Crypto disabled, 2x
16GB Memory DIMM, FH¾L, Single Slot, Tall Bracket
MBF1M626A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, BlueField E-Series 16 cores, PCIe Gen3.0 x16, Auxiliary Card PCIe
Gen3.0 x16, CABLINE-CA II PLUS 350mm, Crypto disabled, No Memory DIMM, FH¾L, Single Slot width, Tall Bracket
MBF1M636A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, BlueField E-Series 16 cores, PCIe Gen3.0 x16, Auxiliary Card PCIe
Gen3.0 x16, CABLINE-CA II PLUS 350mm, QSFP28, 2x 16GB Memory DIMM, FH¾L, Single Slot width, Tall Bracket
MBF1M646A-CSNAT BlueField® Controller card, Dual Port 100Gb/s, 100GbE dual-port QSFP28, PCIe Gen4.0 x16, Crypto disabled, 2x8GB Memory
DIMM, FH¾L, Single Slot, Tall Bracket
MBF1M656A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, BlueField® E-Series 16 cores, PCIe Gen3.0 x16, Auxiliary Card PCIe
Gen3.0 x16, 2 x I-PEX 350mm Cable, Crypto disabled, 2x 8GB Memory DIMM, FH¾L, Single Slot, Tall Bracket
Product Overview
The BlueField BF1600 Dual Port 100Gb/s Controller Card is the perfect solution for managing NVMe storage drives in storage and hyper-converged systems.
BlueField BF1600 Dual Port 100Gb/s Controller Card delivers the highest NVMe-oF target performance, reducing TCO and increasing ROI. The BlueField
BF1600 Controller Cards can transform existing JBOF systems into NVMe-oF compliant solutions, simply by plugging the card into an existing PCIe slot. The
card is available in two configurations: PCIe x16 and PCIe x32. This flexibility makes it easier to match different JBOF system configurations. The card’s

10
small form factor allows customers to install multiple controller cards in a single system to support a larger number of SSDs and high-availability storage
architecture.
BlueField Multicore System-on-chip (SoC)
BlueField SoC is a highly integrated and efficient controller, optimized for NVMe storage systems, Network Functions Virtualization (NFV), Cloud and AI
workloads.
BlueField SoC integrates all the discrete components of a storage system appliance into a single chip (CPU, PCIe switch, and Network controller), making it
the premier SoC solution for building Just-a-Bunch-Of-Flash (JBOF) systems, All-Flash-Array and storage appliances for NVMe over-Fabrics.
With an integrated NVMe-oF offload accelerator, the BlueField SoC gives the BlueField Dual Port 100Gb/s Controller Card a superior performance
advantage over existing JBOF systems, significantly reducing storage transaction latency, while increasing IOPs (I/O operations per second).
BlueField SoC also offers cost-effective and integrative solutions for Machine Learning appliances, enabling efficient data delivery for real-time analytics
and data insights based on superior RDMA and GPUDirect® RDMA technologies. Support for PCIe Gen4.0 helps future-proof the next generation of high-
performance GPU cards.
BlueField Controller Cards
Part Number MBF1M606A-CSNAT MBF1M646A-CSNAT MBF1M616A-CSNAT MBF1M626A-CSNAT MBF1M656A-CSNAT MBF1M636A-CSNAT
Form Factor PCIe Full Height, ¾ Length
9.13 in. x 4.37 in. (232.0 mm x 111.15mm)
Heatsink Height 10.1mm
Ethernet Data Rate 1/10/25/40/50/100 Gb/s Ethernet
Network Connector Type Dual-port QSFP28
PCIe Gen 3.0/4.0 through
x16 Edge Connectora
PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s
The BlueField BF1600 Controller Card should be installed only in a JBOF System as it functions as a PCIe root-complex (RC) initiating PCIe bus
operations. Installing it in a regular host system may damage the card.


11
PCIe Gen 3.0 through x16
Auxiliary Card
NA PCIe Gen 3.0 SERDES @ 8.0GT/s
On-board DDR4 Memory(a) NA 2x 8GB 2x 16GB NA 2x 8GB 2x 16GB
1GbE OOB Management √ √ √ √
RoHS RoHS Compliant
BlueField SoC BlueField E-Series 16 cores
a.Note: Refer to BlueField Software and Firmware release notes for the availability of PCIe Gen 4.0 capabilities.
For more detailed information seeSpecifications.
Features and Benefits
This section describes hardware features and capabilities. Please refer to the software release notes for feature availability.
Feature Description
PCI Express
(PCIe)a
Uses the following PCIe connectors:
• PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x16 edge connector. Gen 1.1 and 2.0 compatible
• Two PCIe Gen 3.0 (8GT/s) CABLINE-CA II PLUS connectors - available in MBF1M656A-CSNAT, MBF1M636A-CSNAT and MBF1M626A-CSNAT
Up to 100
Gigabit Ethernet
•100GbE/ 50GbE / 25GbE
•IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
•IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
•IEEE 802.3az Energy Efficient Ethernet
•IEEE 802.3ap based auto-negotiation and KR startup
•Proprietary Ethernet protocols (50GBASE-R4)
•IEEE 802.3ad, 802.1AX Link Aggregation
It is recommended to upgrade your BlueField product to the latest software and firmware versions available in order to enjoy the latest features
and bug fixes.


12
Feature Description
•IEEE 802.1Q, 802.1P VLAN tags and priority
•IEEE 802.1Qau (QCN)
•Congestion Notification
•IEEE 802.1Qaz (ETS)
•IEEE 802.1Qbb (PFC)
•IEEE 802.1Qbg
•IEEE 1588v2
•Jumbo frame support (9.6KB)
On-board
Memory
• Two DDR4 SODIMM sockets - size is 16GB each (MBF1M636A-CSNAT and MBF1M616A-CSNAT)
• Two DDR4 SODIMM sockets - size is 8GB each (MBF1M656A-CSNAT and MBF1M646A-CSNAT)
• 2x 16GB eMMC NAND flash memory for software
• SPI Flash for NIC firmware
• SPI Flash for CPLD firmware
• EEPROM - Size 128Kbit
BlueField SoC The BlueField-2 SoC integrates eight 64-bit Armv8 A72 cores interconnected by a coherent mesh network, one DRAM controller, an RDMA intelligent network adapter
supporting up to 200Gb/s, an embedded PCIe switch with endpoint and root complex functionality, and up to 16 lanes of PCIe Gen 3.0/4.0.
Overlay
Networks
In order to better scale their networks, data center operators often create overlay networks that carry traffic from individual virtual machines over logical tunnels in
encapsulated formats such as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from the hardware offloading engines, placing higher
loads on the host CPU. BlueField BF1600 Controller Card effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate
and de-capsulate the overlay protocol.
RDMA and RDMA
overConverged
Ethernet (RoCE)
The BlueField BF1600 Controller Cards, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged Ethernet) technology, deliver low-latency and
high-performance over Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as advanced congestion control hardware mechanisms, RoCE provides
efficient low-latency RDMA services over Layer 2 and Layer 3 networks.
NVIDIA
PeerDirect®
PeerDirect communication provides high-efficiency RDMA access by eliminating unnecessary internal data copies between components on the PCIe bus (for example, from
GPU to CPU), and therefore significantly reduces application run time. BlueField BF1600 Controller Card advanced acceleration technology enables higher cluster efficiency
and scalability to tens of thousands of nodes.
Quality of
Service (QoS)
Support for port-based Quality of Service enabling various application requirements for latency and SLA.

13
•
•
•
•
•
Feature Description
Storage
Acceleration
A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks. Standard block and file access protocols can
leverage RDMA for high-performance storage access.
• NVMe over Fabric offloads for the target machine
• T10-DIF Signature Handover
BlueField-2 SmartNIC may operate as a co-processor offloading specific storage tasks from the host, isolating part of the storage media from the host, or enabling
abstraction of software-defined storage logic
using the BlueField-2 Arm cores. On the storage initiator side, BlueField-2 SmartNIC can prove an efficient solution for hyper-converged systems to enable the host CPU to
focus
on compute while all the storage interface is handled through the Arm cores.
High-
PerformanceAcc
elerations
Tag Matching and Rendezvous Offloads
Adaptive Routing on Reliable Transport
Burst Buffer Offloads for Background Checkpointing
GPU Direct The latest advancement in GPU-GPU communications is GPUDirect RDMA. This new technology provides a direct P2P (Peer-to-Peer) data path between the GPU Memory
directly to/from the NVIDIA HCA devices. This provides a significant decrease in GPU-GPU communication latency and completely offloads the CPU, removing it from all
GPU-GPU communications across the network. BlueField BF1600 Controller Card uses high-speed DMA transfers to copy data between P2P devices resulting in more efficient
system applications.
Security
Accelerators
A consolidated compute and network solution based on BlueField Controller Card achieves significant advantages over a centralized security server solution. Standard
encryption protocols and security applications can leverage BlueField compute capabilities and network offloads for security application solutions such as:
• Connection Tracking
• Statefull firewall
a. Refer to BlueField Software and Firmware release notes for availability of PCIe Gen 4.0 capabilities
Operating Systems/Distributions
The BlueField® BF1600 Controller Card supports the following operating systems:
BlueField Software
OpenFabrics Enterprise Distribution (OFED)

14
•
•
•
Connectivity
Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches
Passive copper cable with ESD protection
Powered connectors for optical and active cable support
Package Contents
Before installing your new system, unpack it and check against the below tables that all the parts have been sent according to the OPN you purchased.
Check the parts for visible damage that may have occurred during shipping.
BF1600 Controller Card with Auxiliary Connection Card
Category Qty. Item
Cards 1 BF1600 Controller Card
1 PCIe Auxiliary Card
Cables 1 USB 2.0 Cable Type A to Type B
1 USB 2.0 Cable to UART
2 CABLINE-CAII PLUS 60 PIN harnesses (white and black) - 35cm
1 Tall bracket for the BF1600 Controller Card (shipped assembled)
If anything is damaged or missing, contact your sales representative at [email protected].

Applies to MBF1M626A-CSNAT, MBF1M656A-CSNAT and MBF1M636A-CSNAT.


15
Category Qty. Item
Accessories 1 Tall bracket for the PCIe Auxiliary card (shipped assembled)
1 Short bracket for the PCIe Auxiliary card
BF1600 Controller Card without Auxiliary Connection Card
Category Qty. Item
Cards 1 BF1600 Controller Card
Cables 1 USB 2.0 Cable Type A to Type B
1 USB 2.0 Cable to UART
Accessories 1 Tall bracket for the PCIe Auxiliary card (shipped assembled)
Applies to MBF1M606A-CSNAT, MBF1M646A-CSNAT and MBF1M616A-CSNAT.


16
Supported Interfaces
Layout and Connector Information
BlueField BF1600 Controller Card
The BF1600 Controller card is a 9.13 in. x 4.37 in. (232.0 mm x 111.15mm) card. The component height on the top and bottom of the card complies with
the PCIe specification. The below figures show the connector and LED designations.
BF1600 Controller Card - Component Side
The below figures are for illustration purposes only and might not reflect the current revision of the BF1600 card.


17

18
BF1600 Controller Card - Print Side
Callout Interface Description
1BlueField System On Chip (SoC) BlueField SoC with heatsink
2PCIe x16 Gen 3.0/4.0 Edge Connector The interface between the BlueField Controller and the JBOF system.
3PCIe Gen 3.0 through Two CABLINE-CA II PLUS
Connectors
The interface between the PCIe Auxiliary Card and the BlueField Controller Card.

19
4Ethernet QSFP56 Interfaces Ethernet traffic is transmitted through the controller’s QSFP28 connectors. The QSFP28 connectors allow for the use of Optical
modules and cable interconnect solutions.
5USB Connector USB Type-B Bracket mounted on card
6I²C0 Interface For ConnectX-5 Debug
7UART0 Interface BlueField UART0 (Console): connected to a transceiver and header
8JTAG CoreSight 10 For Arm debugging
9External PCIe Power Supply Interfaces Along with the PCIe interface, this connector provides power to the board.
10 NCSI Management Interface Connectivity channel between the BMC CPU and the BlueField Arm Array
11 CPLD Interface CPLD for board periphery control and power sequencing
12 eMMC Interface 16GB eMMC Flash memory for software
13 DDR4 SODIMM Memory 2 Channel DDR4 SODIMMs, 64bit +ECC
SPI Flash Memory SPI Flash for NIC firmware
14 Networking LED Interfaces One bi-color LED per port.
15 CPLD Revision LEDs Interface Four green LEDs indicating CPLD revision
16 CABLINE-CA II PLUS Connector LED Interfaces One green LED indicating a proper mating connection to cable
17 Power-Good Signal LED Interface One green LED indicating power-good signal
18 PCI Link Status LEDs 2 bi-color LEDs that indicate proper link status of PCI0 and PCI1

20
PCIe Auxiliary Connection Card
The PCIe Auxiliary Card is a 5.09in. x 2.32in. (129.30mm x 59.00mm) card. The below figuresprovide the connector designations.
PCIe Auxiliary Connection Card - Component Side
The PCIe Auxiliary Connection card is supplied with the following OPNs: MBF1M626A-CSNAT, MBF1M656A-CSNAT and MBF1M636A-CSNAT.

The below figures are for illustration purposes only and might not reflect the current revision of the BF1600 card.
