Nvidia DGX A100 User manual

NVIDIA DGX A100 Service Manual
NVIDIA
Jun 23, 2023

Contents
1 Introduction 1
1.1 Customer-replaceable Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 RecommendedTools.......................................... 2
1.3 CustomerSupport ........................................... 2
2 Front Fan Module Replacement 3
2.1 Front Fan Module Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Identifying the Failed Fan Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2.1 ViewingtheFanModuleLED................................... 4
2.2.2 Using the BMC Dashboard and NVSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Replacing and Returning the Front Fan Module . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 Power Supply Replacement 7
3.1 Power Supply Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2 Identifying the Failed Power Supply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2.1 Identifying the Failed Power Supply from the Back . . . . . . . . . . . . . . . . . . . . . 7
3.2.2 Identifying the Failed Power Supply from the Console . . . . . . . . . . . . . . . . . . . 8
3.2.3 Determining the Manufacturer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.3 ReplacingthePowerSupply ..................................... 10
4 Motherboard Tray - Accessing in Place 11
4.1 Accessing the Motherboard Tray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.2 Replacing the Motherboard Tray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5 Motherboard Tray - Removal and Installation 17
5.1 Removing the Motherboard Tray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.2 Reinstalling the Motherboard Tray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
6 U.2 NVMe Cache Drive Upgrade from 4 to 8 25
6.1 U.2 NVMe Cache Drive Upgrade Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.2 Determining the NVMe Drive to Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.3 Installing the Optional NVMe Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
7 U.2 NVMe Cache Drive Replacement 29
7.1 U.2 NVMe Cache Drive Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 29
7.2 Identifying the Failed U.2 NVMe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
7.2.1 Identifying the Failed NVMe from the Front . . . . . . . . . . . . . . . . . . . . . . . . . . 29
7.2.2 Identifying the Failed NVMe from the Console . . . . . . . . . . . . . . . . . . . . . . . . 30
7.2.3 Identifying the NVMe Manufacturer and Model . . . . . . . . . . . . . . . . . . . . . . . 31
7.3 ReplacingtheU.2NVMeDrive .................................... 31
8 U.2 NVMe Cache Drive Upgrade to 7.68 TB Drives 33
8.1 U.2 NVMe Cache Drive 7.68 TB Upgrade Overview . . . . . . . . . . . . . . . . . . . . . . . . 33
8.2 Installing the 7.68 TB NVMe Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
i

9 U.2 NVMe Cache Drive Post-Installation Tasks 37
9.1 Recreating the Cache RAID 0 Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
9.2 ReturningtheNVMeDrive ...................................... 38
10 M.2 NVMe Boot Drive Replacement 39
10.1 M.2 NVMe Boot Drive Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
10.2 Identifying the Failed M.2 NVMe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
10.3 ReplacingtheM.2NVMeDrive.................................... 40
10.4 Rebuilding the Boot Drive RAID 1 Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
10.5 ReturningtheNVMeDrive ...................................... 45
11 M.2 Boot Drive Riser Assembly Replacement 47
11.1 M.2 Boot Drive Riser Assembly Replacement Overview . . . . . . . . . . . . . . . . . . . . . 47
11.2 Determining a Failed M.2 NVMe Riser Assembly . . . . . . . . . . . . . . . . . . . . . . . . . 48
11.3 Replacing the M.2 NVMe Riser Assembly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
11.4 ReturningtheRiserAssembly .................................... 50
12 DIMM Replacement 51
12.1 DIMMReplacementOverview .................................... 51
12.2 IdentifyingtheFailedDIMM ..................................... 51
12.3 ReplacingtheDIMM .......................................... 52
13 DIMM Upgrade 57
13.1 DIMMUpgradeOverview ....................................... 57
13.2 UpgradingtheDIMM.......................................... 57
14 Network Card Replacement 63
14.1 Network Card Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
14.2 Identifying the Failed Network Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
14.3 Replacing the Vertical Network Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
14.4 Replacing the Horizontal Network Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
15 Adding the Optional Dual-port Horizontal Network Card 71
15.1 Dual-port Network Card Upgrade Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
15.2 Adding the Horizontal Network Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
16 Updating the Mellanox Network Card Firmware 77
17 Front Console Board Replacement 79
17.1 Front Console Board Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
17.2 Replacing the Front Console Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
18 Motherboard Tray Battery Replacement 83
18.1 Motherboard Tray Battery Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . 83
18.2 Replacing the Motherboard Tray Battery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
19 Trusted Platform Module Replacement 89
19.1 Trusted Platform Module Replacement Overview . . . . . . . . . . . . . . . . . . . . . . . . 89
19.2 Replacing the Trusted Platform Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
20 Removing and Attaching the Bezel 93
21 Installing the Rack Mount Kit 95
21.1 InstallingtheRails............................................ 95
21.2 InstallingtheCageNuts........................................ 97
ii

22 Third-Party License Notices 99
22.1 Micronmsecli .............................................. 99
22.2 Mellanox(OFED).............................................100
23 Notices 101
23.1 Notice ...................................................101
23.2 Trademarks................................................102
23.3 Copyright.................................................102
iii

iv

Chapter 1. Introduction
This document contains instructions for replacing NVIDIA DGX™ A100 system components. Be sure
to familiarize yourself with the NVIDIA Terms & Conditions documents before attempting to perform
any modication or repair to the DGX A100 system. These Terms & Conditions for the DGX A100
system can be found through the NVIDIA DGX Systems Support page.
Contact NVIDIA Enterprise Support to obtain an RMA number for any system or component that needs
to be returned for repair or replacement. When replacing a component, use only the replacement
supplied to you by NVIDIA.
1.1. Customer-replaceable Components
List of customer-replaceable components in the NVIDIA DGX A100.
You can obtain the following components for replacement in your data center.
▶Fan Modules
▶Power Supplies
▶Cache (U.2) NVMe Drives
▶Boot (M.2) NVMe Drive
▶Boot (M.2) Riser Assembly
▶DIMMs
▶Network Card (vertical, single or dual port)
▶Network Card (horizontal, dual port)
▶Front Console Board
▶Battery
▶Trusted Platform Module (TPM)
▶Bezel
▶Rack Mount Kit
Contact NVIDIA Enterprise Support for replacement instructions and guidance for specic compo-
nents if those instructions are not included in this document.
1

NVIDIA DGX A100 Service Manual
1.2. Recommended Tools
List of recommended tools needed to service the NVIDIA DGX A100.
▶Laptop
▶USB key with tools and drivers
▶USB key imaged with the DGX Server OS ISO
▶Screwdrivers (Phillips #1 and #2, small at head)
▶KVM Crash Cart
▶Anti-static wrist strap
▶Masking tape or label maker
▶Tie wraps or VELCRO® Brand for cable management
▶Box cutter
▶Black Permanent Marker
▶Packing materials
1.3. Customer Support
Contact NVIDIA Enterprise Support for assistance in reporting, troubleshooting, or diagnosing prob-
lems with your DGX A100 system. Also contact NVIDIA Enterprise Support for assistance in installing
or moving the DGX A100 system.
For details on how to obtain support, visit the NVIDIA Enterprise Support web site (https://www.nvidia.
com/en-us/support/enterprise/).
2 Chapter 1. Introduction

Chapter 2. Front Fan Module
Replacement
2.1. Front Fan Module Replacement Overview
This is a high-level overview of the steps needed to replace the front fan modules.
1. Identify the failed front fan module through the BMC or the fan module LED and submit a service
ticket to NVIDIA Enterprise Support.
2. Get a replacement from NVIDIA Enterprise Support.
3. Remove the failed fan module using the fan numbering diagram as a reference.
4. Insert the new fan module.
5. Conrm the new fan module is working correctly through the BMC or NVSM. sudo nvsm show
health
6. Return the bad fan module using the packaging from the new fan module.
2.2. Identifying the Failed Fan Module
There are several ways to determine the faulty fan module to replace.
3

NVIDIA DGX A100 Service Manual
2.2.1. Viewing the Fan Module LED
Look for the lit fault LED on the upper right corner of the faulty fan module.
2.2.2. Using the BMC Dashboard and NVSM
1. Identify the faulty fan module using the BMC dashboard.
1. Log on to the BMC.
2. Click Sensor from the left navigation menu, then review the Normal Sensors section.
There are two fans in the fan module, identied by SPD_FAN_SYSn_F and SPD_FAN_SYSn_R,
where nis the module ID. If either fan fails, then the entire module must be replaced.
2. Use NVSM to conrm the fan issue.
4 Chapter 2. Front Fan Module Replacement

NVIDIA DGX A100 Service Manual
$ sudo nvsm show fans
In the output, look for the ‘unhealthy’ status for the same fan.
2.3. Replacing and Returning the Front Fan
Module
1. Remove the new fan module from its packaging and be ready to install it.
2. Remove the failed fan module by pressing on the release button on the top of the module and
pulling on the handle.
3. Quickly insert the new fan module, observing that the handle release mechanism is facing up.
Caution: Replace the fan module within 30 seconds to prevent overheating of the system
components.
4. Conrm that the fan module is healthy working properly by
▶Verifying that the fan module fault LED is not lit.
▶Viewing the state of the fan module on he BMC dashboard.
▶Using NVSM (sudo nvsm show fans)
5. Use packaging to pack up the bad fan and follow the shipping instructions to return the bad fan
to NVIDIA Enterprise Support.
2.3. Replacing and Returning the Front Fan Module 5

NVIDIA DGX A100 Service Manual
6 Chapter 2. Front Fan Module Replacement

Chapter 3. Power Supply Replacement
This chapter describes how to replace one of the DGX A100 system power supplies (PSUs).
3.1. Power Supply Replacement Overview
This is a high-level overview of the steps needed to replace a power supply.
1. Identify failed power supply through the BMC and submit a service ticket.
2. Get replacement power supply from NVIDIA Enterprise Support.
3. Identify the power supply using the diagram as a reference and the indicator LEDs.
4. Remove the power cord from the power supply that will be replaced.
5. Replace the failed power supply with the new power supply.
6. Insert the power cord and make sure both LEDs light up green (IN/OUT).
7. Use the BMC to conrm that the power supply is working correctly.
8. Ship back the failed unit to NVIDIA Enterprise Support using the packaging provided.
3.2. Identifying the Failed Power Supply
3.2.1. Identifying the Failed Power Supply from the Back
If physical access to the system is available, you can identify a failed PSU by the inspecting the LEDs
on the power supply when the system is powered on.
Both LEDs should be solid green. If either of the LEDs are not green or if they are blinking, contact
NVIDIA Enterprise Support to troubleshoot the issue.
7

NVIDIA DGX A100 Service Manual
3.2.2. Identifying the Failed Power Supply from the
Console
There are several ways to identify the failed PSU from the DGX A100 console.
▶Use the NVSM CLI as follows.
$ sudo nvsm show psus
The output shows information for each PSU. Look for any that do not report Status_Health=OK.
▶View the PSU status from the BMC.
Click Sensor from the left side menu and inspect the PSU information from the Normal Sensors
section.
▶Use ipmitool.
$ sudo ipmitool sdr |grep -i psu
Look for power supplies with no temperature reading or an output reading close to or equal to
zero.
Both NVSM and the BMC identify each power supply as PSUx, where x is from 0 to 5. The following
diagram shows the physical location of each PSU.
8 Chapter 3. Power Supply Replacement

NVIDIA DGX A100 Service Manual
3.2.3. Determining the Manufacturer
Important: All PSUs in the system must be from the same manufacturer.
Issue the following to determine the PSU manufacturer.
$ sudo nvsm show ∕chassis∕localhost∕power∕PSUX
Where Xcorresponds to the PSU identier. The following examples uses PSU0, and shows that the
manufacturer is “Delta”.
$ sudo nvsm show ∕chassis∕localhost∕power∕PSU0
∕chassis∕localhost∕power∕PSU0
Properties:
FirmwareVersion =01.05.01.05.01.05
LastPowerOutputWatts =312
Manufacturer =Delta
MemberId =PSU0
Model =ECD16010092
Name =PSU0
Oem_PSU_Error =<NOT_SET>
PowerSupplyType =AC
SerialNumber =DTHTCP200807M
Status_Health =OK
Status_State =Present
Targets:
Verbs:
cd
show
Obtain the replacement PSU (of the same manufacturer) from NVIDIA Enterprise Support.
3.2. Identifying the Failed Power Supply 9

NVIDIA DGX A100 Service Manual
3.3. Replacing the Power Supply
1. Be sure you have obtained the replacement PSU and that you have saved the packaging to use
when sending back the failed PSU.
2. Determine whether you need to shut down the system.
▶If the three remaining PSUs are working and energized, then you do not need to shut down
power to the DGX A100 system.
▶If fewer than three PSUs are working and energized, then shut down power to the DGX A100
system.
3. Unlock the power cord and then unplug it from the PSU to be replaced.
You may need to dislodge the power cord from the retaining clip.
4. Remove the PSU.
1. Push on the green tab to release the lock.
2. Pull on the black handle to remove the PSU from the chassis.
5. Install the new power supply.
1. Insert the new power supply into the chassis and push it all the way in, making sure that the
green locking mechanism engages.
2. Plug in the power cord and lock it in place.
3. If needed, power on the system.
6. Conrm the installation by
▶Viewing the PSU status from the BMC dashboard->``Sensors`` page.
▶Running nvsm show health to conrm all power supplies are healthy.
Pack the old power supply and ship it back to NVIDIA Enterprise Support.
10 Chapter 3. Power Supply Replacement

Chapter 4. Motherboard Tray -
Accessing in Place
You will need to access the motherboard tray in order to service the following components. This pro-
cess provides access to the motherboard components while the motherboard remains attached to
the server.
▶M.2 NVMe drives
▶M.2 module riser card
▶Motherboard tray battery
▶Single-port or dual port CX-6 PCI network adapter card
4.1. Accessing the Motherboard Tray
1. Loosen the two motherboard thumbscrews and then pull the handles out to eject the mother-
board tray.
11

NVIDIA DGX A100 Service Manual
2. Pull the motherboard tray out of the system until it locks, then loosen the two thumbscrews
holding the lid in place.
3. Lift the rear section of the motherboard lid.
12 Chapter 4. Motherboard Tray - Accessing in Place

NVIDIA DGX A100 Service Manual
4.2. Replacing the Motherboard Tray
1. Close the lid to the motherboard tray.
2. Tighten the two thumbscrews and then push the motherboard tray into the system.
4.2. Replacing the Motherboard Tray 13

NVIDIA DGX A100 Service Manual
3. Close the handles to secure the motherboard tray in place.
4. Tighten the motherboard tray thumbscrews to complete the motherboard insertion.
14 Chapter 4. Motherboard Tray - Accessing in Place