Nvidia DGX H100 User manual

NVIDIA DGX H100 Service Manual
NVIDIA
May 26, 2023

Contents
1 Introduction 1
1.1 Customer-replaceable Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 RecommendedTools.......................................... 2
1.3 CustomerSupport ........................................... 2
1.4 RunningthePre-ightTest...................................... 3
2 Front Fan Module Replacement 5
2.1 Front Fan Module Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Identifying a Failed Fan Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Replacing and Returning the Front Fan Module . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 Power Supply Replacement 11
3.1 Power Supply Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Identifying the Failed Power Supply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3 Preparing the Power Supply for Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 ReplacingthePowerSupply ..................................... 16
3.5 LockingPowerCords.......................................... 17
4 Motherboard Tray - Opening and Closing the IO door 19
4.1 Preparing the Motherboard for Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.2 ReleasetheMotherboard....................................... 20
4.3 Pull Motherboard from Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.4 OpentheMotherboardIODoor ................................... 22
4.5 ClosetheMotherboardIODoor ................................... 23
4.6 LocktheMotherboardLid....................................... 24
4.7 InserttheMotherboard ........................................ 24
4.8 FinalizeMotherboardClosing..................................... 26
5 Motherboard Tray - Removal and Installation 27
5.1 Preparing the Motherboard for Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.2 ReleasetheMotherboard....................................... 28
5.3 Pull Motherboard from Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.4 Remove the Motherboard Tray Lid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.5 Close the Motherboard Tray Lid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.6 Insert the Motherboard Tray into the Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.7 InserttheMotherboard ........................................ 34
5.8 FinalizeMotherboardClosing..................................... 35
6 U.2 NVMe Cache Drive Replacement 37
6.1 U.2 NVMe Cache Drive Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.2 Identifying the Failed U.2 NVMe SSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.3 Identifying the NVMe Manufacturer and Model . . . . . . . . . . . . . . . . . . . . . . . . . 38
6.4 ReplacingtheU.2NVMeDrive .................................... 39
6.5 InserttheU.2NVMeDrive....................................... 41
i

6.6 NextSteps ................................................ 42
7 U.2 NVMe Cache Drive Post-Installation Tasks 43
7.1 Recreating the Cache RAID 0 Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
7.2 ReturningtheNVMeDrive ...................................... 44
8 M.2 NVMe Boot Drive Replacement 45
8.1 M.2 NVMe Boot Drive Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
8.2 IdentifytheFailedM.2NVMe..................................... 46
8.3 Remove the M.2 Boot Drive Carrier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
8.4 RemovetheM.2Drive ......................................... 48
8.5 ReplacetheM.2Drive ......................................... 50
8.6 Install the M.2 Boot Drive Carrier and Close the System . . . . . . . . . . . . . . . . . . . . 51
8.7 Integrate the New Drive and Complete Installation . . . . . . . . . . . . . . . . . . . . . . . 53
9 M.2 Boot Drive Assembly Replacement 55
9.1 M.2 Boot Drive Riser Assembly Replacement Overview . . . . . . . . . . . . . . . . . . . . . 55
9.2 Preparing the System for Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
9.3 Remove the M.2 Boot Drive Carrier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
9.4 Install the M.2 Boot Drive Carrier and Close the System . . . . . . . . . . . . . . . . . . . . 58
9.5 Re-Install the System and Complete the Procedure . . . . . . . . . . . . . . . . . . . . . . . 60
10 DIMM Replacement 61
10.1 DIMMReplacementOverview .................................... 61
10.2 IdentifyingtheFailedDIMM ..................................... 62
10.3 ReplacingtheDIMM .......................................... 62
10.4 FinalizeDIMMReplacement...................................... 64
11 Network Interface Card Replacement 67
11.1 Network Card Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
11.2 Prepare the System for Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
11.3 Remove the Non-Functional Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
11.4 Install the New Card and Close the Lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
11.5 Finalize the Network Interface Card Replacement . . . . . . . . . . . . . . . . . . . . . . . . 71
12 Updating the ConnectX-7 Firmware 73
13 ConnectX-7 I/O Replacement 75
13.1 ConnectX-7 I/O Card Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
13.2 Prepare the System for Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
13.3 Remove the I/O Card above the ConnectX Card to be replaced . . . . . . . . . . . . . . . . 76
13.4 RemovetheConnectXCard...................................... 76
13.5 RemoveanIPEXCable......................................... 77
13.6 InsertanIPEXCable .......................................... 78
13.7 InstallConnectXCard ......................................... 79
13.8 Install the I/O Card above the ConnectX Card . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
13.9 Power on the System and Conrm the Replacement . . . . . . . . . . . . . . . . . . . . . . 80
14 Front Console Board Replacement 81
14.1 Front Console Board Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
14.2 Front Console Board Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
15 Motherboard Tray Battery Replacement 85
15.1 Motherboard Tray Battery Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . 85
15.2 IdentifyaFailedBattery........................................ 86
ii

15.3 Prepare the System for Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
15.4 RemovethePCIEthernetCard.................................... 86
15.5 RemovetheConnectXCard...................................... 88
15.6 RemoveanIPEXCable......................................... 89
15.7 ReplacetheBattery........................................... 90
15.8 InsertanIPEXCable .......................................... 91
15.9 InstallConnectXCard ......................................... 92
15.10 InstallthePCIEthernetCard..................................... 93
15.11 Power On the System and Conrm Replacement . . . . . . . . . . . . . . . . . . . . . . . . 95
16 Trusted Platform Module Replacement 97
16.1 Trusted Platform Module Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . 97
16.2 Prepare the System for Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
16.3 ReplacetheTPMModule ....................................... 98
16.4 InstallOSFPCarrierModule......................................100
16.5 FinalizeTPMreplacement.......................................101
17 Removing and Attaching the Bezel 103
17.1 BezelRemoval ..............................................103
17.2 BezelInstallation ............................................104
18 Rack Mount Kit Replacement 107
18.1 RackMountKitDescription......................................107
18.2 Remove Rack Mount Kit - Front Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
18.3 RemoveRackMountKit-Rear....................................109
18.4 Conrm Necessary Screws and Washers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
18.5 Install Cage Nuts Using Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
18.6 InstallRackMountKit-Front.....................................114
18.7 InstallRackMountKit-Rear.....................................115
19 Safety 117
19.1 SafetyInformation ...........................................117
19.2 Safety Warnings and Cautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
19.3 IntendedApplicationUses ......................................118
19.4 SiteSelection ..............................................118
19.5 Equipment Handling Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
19.6 ElectricalPrecautions .........................................119
19.6.1 Power and Electrical Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
19.6.2 PowerCordWarnings .......................................120
19.7 SystemAccessWarnings .......................................120
19.8 RackMountWarnings .........................................121
19.9 ElectrostaticDischarge ........................................122
19.10 OtherHazards..............................................122
19.10.1 CALIFORNIA DEPARTMENT OF TOXIC SUBSTANCES CONTROL . . . . . . . . . . . . . 122
19.10.2NICKEL ................................................122
19.10.3BatteryReplacement........................................122
19.10.4CoolingandAirow.........................................123
20 Compliance 125
20.1 UnitedStates ..............................................125
20.2 UnitedStates/Canada .........................................125
20.3 Canada ..................................................126
20.4 EU......................................................126
20.5 AustraliaandNewZealand ......................................127
20.6 Brazil....................................................127
iii

20.7 Japan ...................................................127
20.8 SouthKorea ...............................................129
20.9 China....................................................130
20.10 Taiwan...................................................132
20.11 Russia/Kazakhstan/Belarus......................................133
20.12 Israel....................................................134
20.13 India ....................................................134
20.14 SouthAfrica ...............................................135
20.15 Great Britain (England, Wales, and Scotland) . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
21 Third-Party License Notices 137
21.1 Micronmsecli ..............................................137
21.2 Mellanox(OFED).............................................138
22 Notices 139
22.1 Notice ...................................................139
22.2 Trademarks................................................140
22.3 Copyright.................................................140
iv

Chapter 1. Introduction
This document contains instructions for replacing NVIDIA DGX H100 system components. Be sure to
familiarize yourself with the NVIDIA Terms and Conditions documents before attempting to perform
any modication or repair to the DGX H100 system. These Terms and Conditions for the DGX H100
system can be found through the NVIDIA DGX Systems Support page.
Contact NVIDIA Enterprise Support to obtain an RMA number for any system or component that needs
to be returned for repair or replacement. When replacing a component, use only the replacement
supplied to you by NVIDIA.
1.1. Customer-replaceable Components
List of customer-replaceable components in the NVIDIA DGX H100.
Be sure to familiarize yourself with the NVIDIA Terms & Conditions documents before attempting to
perform any modication or repair to the DGX H100 system. These Terms & Conditions for the DGX
H100 system can be found through the NVIDIA DGX Systems Support page.
Customer Replaceable Units
Contact NVIDIA Enterprise Support to obtain an RMA number for any system or component that needs
to be returned for repair or replacement. When replacing a component, use only the replacement
supplied to you by NVIDIA.
You can obtain the following components for replacement in your data center.
▶Bezel
▶Locking power cords
▶Power supply
▶Fan module
▶Front Console Board
▶U.2 data drive
▶M.2 boot (OS) storage drive
▶Riser assembly with 2 M.2 drives
▶ConnectX-7 PCI card (Storage Network)
▶50 Gb Ethernet NIC replacement
1

NVIDIA DGX H100 Service Manual
▶DIMMs
▶Rackmount kit
▶Trusted Platform Module
▶Battery
Contact NVIDIA Enterprise Support for replacement instructions and guidance for specic compo-
nents if those instructions are not included in this document.
1.2. Recommended Tools
Here are the tools that are required for the procedures outlined in the presentation. Each procedure
will indicate which tool is needed for the dierent actions.
Important: Hearing protection is mandatory when working with or in the vicinity of this equipment
▶Laptop
▶USB key with tools and drivers
▶USB key imaged with the DGX Server OS ISO
▶Screwdrivers
▶Phillips #1
▶Phillips #2
▶Torx T15
▶Torx T10
▶KVM Crash Cart
▶Electro Static Discharge strap and/or mat
▶Masking tape or label maker
▶Tie wraps or VELCRO® Brand for cable management
▶Box cutter
▶Black Permanent Marker or Labels/labeler for cable identication
▶Packing materials
1.3. Customer Support
Contact NVIDIA Enterprise Support for assistance in reporting, troubleshooting, or diagnosing prob-
lems with your DGX H100 system. Also contact NVIDIA Enterprise Support for assistance in installing
or moving the DGX H100 system.
For details on how to obtain support, visit the NVIDIA Enterprise Support web site (https://www.nvidia.
com/en-us/support/enterprise/ ).
2 Chapter 1. Introduction

NVIDIA DGX H100 Service Manual
1.4. Running the Pre-ight Test
Instructions for running the DGX stress test.
NVIDIA recommends running the pre-ight stress test before putting a system into a production envi-
ronment or after servicing. You can specify running the test on the GPUs, CPU, memory, and storage,
and also specify the duration of the tests.
To run the tests, use NVSM.
Syntax:
sudo nvsm stress-test [--usage] [--force] [--no-prompt] [<test>...] [DURATION]
For help on running the test, issue the following.
sudo nvsm stress-test --usage
Recommended Command
The following command runs the test on all supported components (GPU, CPU, memory, and storage),
and takes approximately 20 minutes.
sudo nvsm stress-test --force
1.4. Running the Pre-ight Test 3

NVIDIA DGX H100 Service Manual
4 Chapter 1. Introduction

Chapter 2. Front Fan Module
Replacement
2.1. Front Fan Module Replacement Overview
This is a high-level overview of the steps needed to replace the front fan modules.
1. Identify failed front fan module through BMC or with the fan module LED and submit a service
ticket
2. Get replacement from NVIDIA Enterprise Support
3. Remove failed fan module
4. Insert new fan module
5. Conrm new fan module is working correctly through BMC or the operating system tools
6. Return/ship the failed unit to NVIDIA Enterprise Support using the packaging provided
2.2. Identifying a Failed Fan Module
You can identify a failed fan module using any of the following methods:
▶Remove the system bezel and visually inspect the fan module LEDs.
▶Run the nvsm show fans command and view the command output.
▶Access the BMC web user interface and view the sensor data from the fans. If a fan is running
at an abnormal speed, then that fan needs to be replaced.
5

NVIDIA DGX H100 Service Manual
Viewing the Fan Module LEDs
1. Removing and Attaching the Bezel to expose the fan modules.
After you remove the bezel, the system looks like the following gure.
2. Identify the failed fan using the fan module fault LED as shown in the following gure.
3. Look for the lit fault LED on the upper right corner of the faulty fan module as shown in the
6 Chapter 2. Front Fan Module Replacement

NVIDIA DGX H100 Service Manual
following gure.
Running the Show Fans command
▶From the operating system, run:
sudo nvsm show fans
View the command output for any alerts, failures, or an unhealthy status.
Viewing Fan Modules from the BMC web user interface
1. Identify the faulty fan module using the BMC dashboard.
2. Log on to the BMC.
3. Click Sensor from the left navigation menu.
4. Review the Normal Sensors section.
5. Look for abnormal fan speeds in the right column.
2.2. Identifying a Failed Fan Module 7

NVIDIA DGX H100 Service Manual
There are two fans in the fan module, identied by SPD_FAN_SYSn_F and SPD_FAN_SYSn_R,
where nis the module ID. If either fan fails, then the entire module must be replaced.
6. Use the nvsm command to conrm the fan issue.
8 Chapter 2. Front Fan Module Replacement

NVIDIA DGX H100 Service Manual
sudo nvsm show fans
View the output and conrm that the status is unhealthy for the same fan.
2.3. Replacing and Returning the Front Fan
Module
1. Remove the new fan module from its packaging and be ready to install it.
Important: Replace the old fan with the new one within 30 seconds to avoid overheating of the
system components.
2. Refer to Removing and Attaching the Bezel to expose the fan modules.
3. Unlock the fan module by pressing the release button, as shown in the following gure.
4. Replace the failed fan module with the new one.
2.3. Replacing and Returning the Front Fan Module 9

NVIDIA DGX H100 Service Manual
5. Conrm that the fan module is healthy working properly by performing the following actions:
▶Using the BMC web user interface
▶Verifying that the amber LED on the fan module is extinguished
▶Running the sudo nvsm show fans command
▶Install the bezel as described in the bezel section
10 Chapter 2. Front Fan Module Replacement

Chapter 3. Power Supply Replacement
This section describes how to replace one of the DGX H100 system power supplies (PSUs).
3.1. Power Supply Replacement Overview
This is a high-level overview of the steps needed to replace a power supply.
1. Identify the broken power supply either by the amber color LED or by the power supply number
2. Request a replacement from NVIDIA Enterprise Support.
3. Remove the locking power cord from the power supply
4. Replace the power supply
5. Install the locking power cord
6. Conrm that both LEDs light up green on the power supply
7. Make sure the BMC reports no power supply failures
8. If requested, ship back the failed unit to NVIDIA Enterprise Support using the packaging provided
3.2. Identifying the Failed Power Supply
You can identify a failed power supply using any of the following methods:
▶Visually inspect the the LEDs on the power supplies from the rear of the system when the system
is powered on.
▶Run the nvsm show psus command and view the command output.
▶Access the BMC web user interface and view the sensor data.
NVIDIA Enterprise Support might ask for this or similar information to conrm the power supply needs
to be replaced.
The nvsm command output and the BMC web user interface identify each power supply as PSUx, where
xis 0to 5. The following diagram shows the physical location of each PSU.
11

NVIDIA DGX H100 Service Manual
Viewing the Power Supply LEDs
▶Access the rear of the system and view the status LEDs while the system is powered on.
Both LEDs are solid green if the PSU is good. If either of the LEDs are not green or they blink,
contact NVIDIA Enterprise Support to troubleshoot the issue.
12 Chapter 3. Power Supply Replacement

NVIDIA DGX H100 Service Manual
Running the Show PSUs Command
▶Run the following command to display information about the PSUs:
sudo nvsm show psus
The output shows information for each PSU. Look for any that do not report Status_Health=OK.
Viewing PSUs from the BMC web user interface
1. Access the BMC web user interface and select Sensors from the left hand column.
▶Conrm PSU presence:
▶Conrm power output:
▶Conrm fan speeds:
3.2. Identifying the Failed Power Supply 13

NVIDIA DGX H100 Service Manual
▶Conrm the PSU temperature readings:
2. Run the ipmitool command to view information about the PSUs:
sudo ipmitool sdr | grep -i psu
Look for power supplies with no temperature reading or an output reading that is close to, or
equal to, zero.
14 Chapter 3. Power Supply Replacement