Nvidia DGX H100 User manual

NVIDIA DGX H100 User Guide
NVIDIA
May 26, 2023

Contents
1 Introduction to the NVIDIA DGX H100 System 1
1.1 HardwareOverview........................................... 2
1.1.1 DGX H100 Models and Component Descriptions . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 MechanicalSpecications..................................... 3
1.1.3 PowerSpecications........................................ 3
1.1.3.1 Support for 4+2 Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.4 DGX H100 Locking Power Cord Specication . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.5 Using the Locking Power Cords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.6 Environmental Specications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.7 Front Panel Connections and Controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.7.1 WithaBezel .......................................... 5
1.1.7.2 WiththeBezelRemoved................................... 7
1.1.8 RearPanelModules......................................... 7
1.1.9 Motherboard Connections and Controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.1.10 Motherboard Tray Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.1.11 GPUTrayComponents....................................... 10
1.2 Network Connections, Cables, and Adaptors . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2.1 NetworkPorts............................................ 11
1.2.2 Compute and Storage Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.2.3 NetworkModules.......................................... 13
1.2.4 Supported Network Cables and Adaptors . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.3 DGXH100SystemTopology ..................................... 15
1.4 DGXOSSoftware............................................ 15
1.5 CustomerSupport ........................................... 16
2 Connecting to the DGX H100 17
2.1 ConnectingtotheConsole ...................................... 17
2.1.1 DirectConnection.......................................... 17
2.1.2 Remote Connection through the BMC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2 SSHConnectiontotheOS ...................................... 21
3 First Boot Setup 23
3.1 SystemSetup .............................................. 23
3.2 PostSetupTasks ............................................ 25
3.2.1 Obtaining Software Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2.2 EnablingtheSRPDaemon .................................... 25
4 Quickstart and Basic Operation 27
4.1 Installation and Conguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2 Registration ............................................... 27
4.3 ObtaininganNGCAccount ...................................... 28
4.4 TurningDGXH100OnandO.................................... 28
4.4.1 StartupConsiderations ...................................... 28
i

4.4.2 ShutdownConsiderations..................................... 28
4.5 Verifying Functionality - Quick Health Check . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.6 RunningthePre-ightTest...................................... 29
4.7 Running NGC Containers with GPU Support . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.7.1 UsingNativeGPUSupport .................................... 30
4.7.2 Using the NVIDIA Container Runtime for Docker . . . . . . . . . . . . . . . . . . . . . . . 31
4.8 ManagingCPUMitigations ...................................... 32
4.8.1 Determining the CPU Mitigation State of the DGX System . . . . . . . . . . . . . . . . 32
4.8.2 DisablingCPUMitigations..................................... 33
4.8.3 Re-enabling CPU Mitigations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5 SBIOS Settings 35
5.1 AccessingtheSBIOSSetup...................................... 35
5.2 ConguringtheBootOrder...................................... 36
5.3 ConguringtheLocalTerminal.................................... 38
5.3.1 Linux.................................................. 38
5.3.2 WindowsandMacOS........................................ 38
5.4 PoweronorReboottheSystem................................... 38
6 Using the Baseboard Management Controller (BMC) 39
6.1 ConnectingtotheBMC ........................................ 39
6.2 OverviewofBMCControls ...................................... 40
6.3 Changing the BMC Login Credentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.4 UsingtheRemoteConsole ...................................... 43
6.5 Setting Up Active Directory, LDAP, or E-Directory . . . . . . . . . . . . . . . . . . . . . . . . 43
6.6 Conguring Platform Event Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.7 Uploading or Generating SSL Certicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.7.1 ViewingtheSSLCerticate.................................... 45
6.7.2 Generating the SSL Certicate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6.7.3 Uploading the SSL Certicate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.7.4 Updating the SBIOS Certicate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
7 Security 51
7.1 UserSecurityMeasures ........................................ 51
7.1.1 SecuringtheBMCPort ...................................... 51
7.2 SystemSecurityMeasures ...................................... 51
7.2.1 Secure Flash of DGX H100 Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.2.2 Encryption .............................................. 52
7.2.3 NVIDIA System Manager Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.3 SecureDataDeletion.......................................... 52
7.3.1 Prerequisites............................................. 52
7.3.2 Procedure............................................... 53
8 Redsh APIs Support 55
8.1 SupportedRedshFeatures ..................................... 55
8.2 RedshExamples............................................ 56
8.2.1 BMCManager ............................................ 56
8.2.2 FirmwareUpdate .......................................... 57
8.2.3 BIOSSettings ............................................ 59
8.2.4 Telemetry............................................... 60
8.2.5 Chassis ................................................ 60
8.2.6 SELLogs ............................................... 61
8.2.7 VirtualImage............................................. 61
8.2.8 CollectBMCDebugData ..................................... 61
ii

9 Safety 63
9.1 SafetyInformation ........................................... 63
9.2 SafetyWarningsandCautions.................................... 63
9.3 IntendedApplicationUses ...................................... 64
9.4 SiteSelection .............................................. 64
9.5 EquipmentHandlingPractices.................................... 65
9.6 ElectricalPrecautions ......................................... 65
9.6.1 Power and Electrical Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
9.6.2 PowerCordWarnings ....................................... 66
9.7 SystemAccessWarnings ....................................... 66
9.8 RackMountWarnings ......................................... 67
9.9 ElectrostaticDischarge ........................................ 68
9.10 OtherHazards.............................................. 68
9.10.1 CALIFORNIA DEPARTMENT OF TOXIC SUBSTANCES CONTROL . . . . . . . . . . . . . 68
9.10.2 NICKEL ................................................ 68
9.10.3 BatteryReplacement........................................ 68
9.10.4 CoolingandAirow......................................... 69
10 Compliance 71
10.1 UnitedStates .............................................. 71
10.2 UnitedStates/Canada ......................................... 71
10.3 Canada .................................................. 72
10.4 CE...................................................... 72
10.5 AustraliaandNewZealand ...................................... 73
10.6 Brazil.................................................... 73
10.7 Japan ................................................... 73
10.8 SouthKorea ............................................... 75
10.9 China.................................................... 76
10.10 Taiwan................................................... 78
10.11 Russia/Kazakhstan/Belarus...................................... 79
10.12 Israel.................................................... 80
10.13 India .................................................... 80
10.14 SouthAfrica ............................................... 81
10.15 Great Britain (England, Wales, and Scotland) . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
11 Third-Party License Notices 83
11.1 Micronmsecli .............................................. 83
11.2 Mellanox(OFED)............................................. 84
12 Notices 85
12.1 Notice ................................................... 85
12.2 Trademarks................................................ 86
12.3 Copyright................................................. 86
iii

iv

Chapter 1. Introduction to the NVIDIA
DGX H100 System
The NVIDIA DGX H100 System is the universal system purpose-built for all AI infrastructure and work-
loads, from analytics to training to inference. The system is built on eight NVIDIA H100 Tensor Core
GPUs.
1

NVIDIA DGX H100 User Guide
1.1. Hardware Overview
1.1.1. DGX H100 Models and Component Descriptions
There are two models of the NVIDIA DGX H100 system: the NVIDIA DGX H100 640GB system and the
NVIDIA DGX H100 320GB system.
Table 1: Table 1. Component Description
Component Description
GPU 8 x NVIDIA H100 GPUs that provide 640 GB total GPU memory
CPU 2 x Intel Xeon 8480C PCIe Gen5 CPU with 56 cores 2.0/2.9/3.8
GHz (base/all core turbo/Max turbo)
NVSwitch 4 x 4th generation NVLink that provide 900 GB/s GPU-to-GPU
bandwidth
Storage (OS) 2 x 1.92 TB NVMe M.2 SSD (ea) in RAID 1 array
Storage (Data Cache) 8 x 3.84 TB NVMe U.2 SED (ea) in RAID 0 array
Network (Cluster) card 4 x OSFP ports for 8 NVIDIA ConnectX-7 Single Port InniBand
cars that each provide the following speeds:
InniBand (default): Up to 400Gbps
Ethernet: 400GbE, 200GbE, 100GbE, 50GbE, 40GbE, 25GbE, and
10GbE
Network (Storage) card 2 x NVIDIA ConnectX-7 Dual Port Ethernet cards that each pro-
vide the following speeds:
Ethernet (default): 400GbE, 200GbE, 100GbE, 50GbE, 40GbE,
25GbE, and 10GbE
InniBand: Up to 400Gbps
System Memory (DIMM) 2 TB per 32 DIMMs
BMC (out-of-band system man-
agement)
1 GbE RJ45 interface
Supports Redsh, IPMI, SNMP, KVM, and Web user interface
In-band system management 50 GbE in slot 3 and 10 GbE RJ45 interface
Power Supply 6 x 3.3 kW
2 Chapter 1. Introduction to the NVIDIA DGX H100 System

NVIDIA DGX H100 User Guide
1.1.2. Mechanical Specications
Table 2: Table 2. Mechanical Specications
Feature Description
Form Factor 8U Rackmount
Height 14” (356 mm)
Width 19” (482.3 mm) max
Depth 35.3” (897.1 mm) max
System Weight 271.5 lbs (123.16 kg) max
1.1.3. Power Specications
The DGX H100 system contains six power supplies with balanced distribution of the power load.
Table 3: Table 3. Power Specications
Input Specication for Each Power Supply
200-240 volts AC 10.2 kW max. 3300 W @ 200-240 V, 16 A, 50-60 Hz
1.1.3.1 Support for 4+2 Redundancy
The DGX H100 includes six power supply units (PSU) congured for 4+2 redundancy. If two PSUs fail,
the system will continue to operate at full power with the remaining four PSUs.
▶If less than four PSUs are working, troubleshoot the cause for the loss of power from the other
PSUs and correct. If faulty PSUs need to be replaced, shut the system down and install working
PSUs.
▶The system will only boot if four PSUs are working. If fewer than four PSUs are available, only the
BMC will be available.
1.1.4. DGX H100 Locking Power Cord Specication
The DGX H100 is shipped with a set of six (6) locking power cords which have been qualied for use
with the DGX H100 to ensure regulatory compliance.
Warning: To avoid electric shock or re, only use the NVIDIA-provided power cords to connect
power to the DGX H100. For more details, refer to Electrical Precautions.
1.1. Hardware Overview 3

NVIDIA DGX H100 User Guide
Power Cord Specication
Power Cord Feature Specication
Electrical 250VAC, 16A
Plug Standard C19/C20
Dimension 1200mm length
Compliance Cord: UL62, IEC60227
Connector/Plug: IEC60320-1
1.1.5. Using the Locking Power Cords
This section provides information about how to use the locking power cords.
Locking and Unlocking the PDU Side
Power Distribution Unit side
▶To INSERT, push the cable into the PDU socket.
▶To REMOVE, press the clips together and pull the cord out of the socket.
Locking/Unlocking the PSU Side (Cords with Twist-Lock Mechanism)
Power Supply (System) side - Twist locking
▶To INSERT or REMOVE make sure the cable is UNLOCKED and push/ pull into/out of the socket.
4 Chapter 1. Introduction to the NVIDIA DGX H100 System

NVIDIA DGX H100 User Guide
1.1.6. Environmental Specications
Here are the environmental specications for your DGX H100 system.
Feature Specication
Operating Temperature 5° C to 30° C (41° F to 86° F)
Relative Humidity 20% to 80% non-condensing
Airow 1105 CFM Front-to-Back @ 80% fan PWM
Heat Output 38,557 BTU/hr
1.1.7. Front Panel Connections and Controls
This section provides information about the front panel, connections, and controls of the DGX H100
system.
1.1.7.1 With a Bezel
Here is an image of the DGX H100 system with a bezel.
1.1. Hardware Overview 5

NVIDIA DGX H100 User Guide
Control Description
Power Button Press to turn the DGX H100 system On or O.
▶Green ashing (1 Hz): Standby (BMC
booted)
▶Green ashing (4 Hz): POST in progress
▶Green solid On: Power On
ID Button Press to have the blue LED turn On or blink (con-
gurable through the BMC) as an identier dur-
ing servicing.
Also causes an LED on the back of the unit to
ash as an identier during servicing.
Fault LED Amber On: System or component faulted
6 Chapter 1. Introduction to the NVIDIA DGX H100 System

NVIDIA DGX H100 User Guide
1.1.7.2 With the Bezel Removed
Here is an image of the DGX H100 system without a bezel.
Important: Refer to the section First Boot Setup for instructions on how to properly turn the system
on or o.
1.1.8. Rear Panel Modules
Here is an image that shows the real panel modules on DGX H100.
1.1. Hardware Overview 7

NVIDIA DGX H100 User Guide
1.1.9. Motherboard Connections and Controls
Here is an image that shows the motherboard connections and controls in a DGX H100 system.
8 Chapter 1. Introduction to the NVIDIA DGX H100 System

NVIDIA DGX H100 User Guide
Table 4: Table 4. Motherboard Controls
Control Description
Power Button Press to turn the system On or O.
ID LED Button Blinks when ID button is pressed from the front of the unit as an aid in identifying
the unit needing servicing.
BMC Reset
button
Press to manually reset the BMC.
See Network Connections, Cables, and Adaptors for details on the network connections.
1.1.10. Motherboard Tray Components
Here is an image that shows the motherboard tray components in DGX H100.
1.1. Hardware Overview 9

NVIDIA DGX H100 User Guide
1.1.11. GPU Tray Components
Here is an image of the GPU tray components in a DGX H100 system.
10 Chapter 1. Introduction to the NVIDIA DGX H100 System

NVIDIA DGX H100 User Guide
1.2. Network Connections, Cables, and Adaptors
This section provides information about network connections, cables, and adaptors.
1.2.1. Network Ports
Here is an image that shows the network ports on a DGX H100 system.
1.2. Network Connections, Cables, and Adaptors 11

NVIDIA DGX H100 User Guide
Table 5: Table 5. Network Port Mapping
Port Designation
Port PCI Bus Default Optional RDMA
OSFP1P1 dc:00.0 ibp220s0 enp220s0 mlx5_11
OSFP1P2 9a:00.0 ibp154s0 enp154s0 mlx5_6
OSFP2P1 ce:00.0 ibp206s0 enp206s0 mlx5_10
OSFP2P2 c0:00.0 ibp192s0 enp192s0 mlx5_9
OSFP3P1 4f:00.0 ibp79s0 enp79s0 mlx5_4
OSFP3P2 40:00.0 ibp64s0 enp64s0 mlx5_3
OSFP4P1 5e:00.0 ibp94s0 enp94s0 mlx5_5
OSFP4P2 18:00.0 ibp24s0 enp24s0 mlx5_0
Slot1 P1 aa:00.0 ibp170s0f0 enp170s0f0 mlx5_7
Slot1 P2 aa:00.1 enp170s0f1np1 ibp170s0f1np1 mlx5_8
Slot2 P1 29:00.0 ibp41s0f0 enp41s0f0 mlx5_1
Slot2 P2 29:00.1 enp41s0f1np1 ibp41s0f1np1 mlx5_2
Slot3 P1 82:00.0 ens6f0 N/A irdma0
Slot3 P2 82:00.1 ens6f1 N/A irdma1
On-board 0b:00.0 eno3 N/A
1.2.2. Compute and Storage Networking
12 Chapter 1. Introduction to the NVIDIA DGX H100 System

NVIDIA DGX H100 User Guide
1.2.3. Network Modules
▶New form factor for aggregate PCIe network devices
▶Consolidates four ConnectX-7 networking cards into a single device
▶Two networking modules are installed on interposer board
▶Interposer board connects to CPUs on one end and to GPU tray on the other
▶DensiLink cables are used to go directly from ConnectX-7 networking cards to OSFP connectors
at the back of the system
Each DensiLink cable has two ports, one from each ConnectX-7 card
Table 6: Table 6. Network Modules
Port ConnectX Device Network Module/CPU GPU Default RDMA
OSFP1P1 CX0 1 7 ibp220s0 mlx5_11
OSFP1P2 CX1 1 4 ibp154s0 mlx5_6
OSFP2P1 CX2 1 6 ibp206s0 mlx5_10
OSFP2P2 CX3 1 5 ibp192s0 mlx5_9
OSFP3P1 CX2 0 2 ibp79s0 mlx5_4
OSFP3P2 CX3 0 1 ibp64s0 mlx5_3
OSFP4P1 CX0 0 3 ibp94s0 mlx5_5
OSFP4P2 CX1 0 0 ibp24s0 mlx5_0
1.2. Network Connections, Cables, and Adaptors 13

NVIDIA DGX H100 User Guide
1.2.4. Supported Network Cables and Adaptors
The DGX H100 system is not shipped with network cables or adaptors. You will need to purchase
supported cables or adaptors for your network.
The ConnectX-7 rmware determines which cables and adaptors are supported. For a list of cables
and adaptors compatible with the NVIDIA ConnectX cards installed in the DGX H100 system,
1. Visit the NVIDIA Adapter Firmware Release page.
2. From the left navigation menu, select the ConnectX model and corresponding rmware included
in the DGX H100.
3. Select Firmware Compatible Products.
14 Chapter 1. Introduction to the NVIDIA DGX H100 System