Supermicro SuperServer SYS-821GE-TNHR User manual

®
SYS-821GE-TNHR
USER’S MANUAL
Revision 1.0

The information in this User’s Manual has been carefully reviewed and is believed to be accurate. The vendor assumes
no responsibility for any inaccuracies that may be contained in this document, and makes no commitment to update
or to keep current the information in this manual, or to notify any person or organization of the updates. Please Note:
For the most up-to-date version of this manual, please see our website at www.supermicro.com.
Super Micro Computer, Inc. ("Supermicro") reserves the right to make changes to the product described in this manual
at any time and without notice. This product, including software and documentation, is the property of Supermicro and/
or its licensors, and is supplied only under a license. Any use or reproduction of this product is not allowed, except
as expressly permitted by the terms of said license.
IN NO EVENT WILL Super Micro Computer, Inc. BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL,
SPECULATIVE OR CONSEQUENTIALDAMAGESARISING FROM THE USE OR INABILITYTO USE THIS PRODUCT
OR DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN PARTICULAR, SUPER
MICRO COMPUTER, INC. SHALL NOT HAVE LIABILITY FOR ANY HARDWARE, SOFTWARE, OR DATA STORED
OR USED WITH THE PRODUCT, INCLUDING THE COSTS OF REPAIRING, REPLACING, INTEGRATING,
INSTALLING OR RECOVERING SUCH HARDWARE, SOFTWARE, OR DATA.
Any disputes arising between manufacturer and customer shall be governed by the laws of Santa Clara County in the
State of California, USA. The State of California, County of Santa Clara shall be the exclusive venue for the resolution
of any such disputes. Supermicro's total liability for all claims will not exceed the price paid for the hardware product.
FCC Statement: This equipment has been tested and found to comply with the limits for a Class A or Class B digital
device pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against
harmful interference when the equipment is operated in industrial environment for Class A device or in residential
environment for Class B device. This equipment generates, uses, and can radiate radio frequency energy and, if not
installed and used in accordance with the manufacturer’s instruction manual, may cause harmful interference with
radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in
which case you will be required to correct the interference at your own expense.
California Best Management Practices Regulations for Perchlorate Materials: This Perchlorate warning applies only
to products containing CR (Manganese Dioxide) Lithium coin cells. Perchlorate Material-special handling may apply.
See www.dtsc.ca.gov/hazardouswaste/perchlorate.
WARNING: This product can expose you to chemicals including
lead, known to the State of California to cause cancer and birth
defects or other reproductive harm. For more information, go
to www.P65Warnings.ca.gov.
!
The products sold by Supermicro are not intended for and will not be used in life support systems, medical equipment,
nuclear facilities or systems, aircraft, aircraft devices, aircraft/emergency communication devices or other critical

property damage. Accordingly, Supermicro disclaims any and all liability, and should buyer use or sell such products
for use in such ultra-hazardous applications, it does so entirely at its own risk. Furthermore, buyer agrees to fully
indemnify, defend and hold Supermicro harmless for and against any and all claims, demands, actions, litigation, and
proceedings of any kind arising out of or related to such ultra-hazardous use or sale.
Manual Revision 1.0
Release Date: April 25, 2023
Unless you request and receive written permission from Super Micro Computer, Inc., you may not copy any part of this

to herein are trademarks or registered trademarks of their respective companies or mark holders.
Copyright © 2023 by Super Micro Computer, Inc.
All rights reserved.
Printed in the United States of America

3
Preface
Preface
About this Manual
This manual is written for professional system integrators and PC technicians. It provides
information for the installation and use of this server. Installation and maintenance should be
performed by experienced technicians only.
page on our website for updates
on supported memory, processors and operating systems (www.supermicro.com).
Notes
For your system to work properly, please follow the links below to download all necessary
drivers/utilities and the user’s manual for your server.
• Supermicro product manuals: https://www.supermicro.com/support/manuals/
• Product drivers and utilities: https://www.supermicro.com/wdl/
• Product safety info: https://www.supermicro.com/en/about/policies/safety-information
If you have any questions, please contact our support team at: [email protected]m
This manual may be periodically updated without notice. Please check the Supermicro website
for possible updates to the manual revision level.
Secure Data Deletion
A secure data deletion tool designed to fully erase all data from storage devices can be found
on our website: https://www.supermicro.com/about/policies/disclaimer.cfm?url=/wdl/utility/
Lot9_Secure_Data_Deletion_Utility/
Warnings
Special attention should be given to the following symbols used in this manual.
Warning! Indicates high voltage may be encountered when performing a procedure.
Warning! Indicates important information given to prevent equipment/property damage


Contents
4
Contents
Chapter 1 Introduction
1.1 Overview...............................................................................................................................9
1.2 System Features ................................................................................................................10
Front View .........................................................................................................................10
Drive Carrier Indicators..................................................................................................11
Control Panel.................................................................................................................12
Rear View..........................................................................................................................14
Power Supply Indicator..................................................................................................16
Fan Module Indicator.....................................................................................................16
Top View, Motherboard Tray and Switch Module .............................................................17
Top View, GPU Tray..........................................................................................................19
1.3 System Block Diagram.......................................................................................................20
1.4 Motherboard Layout ...........................................................................................................21
Quick Reference ...............................................................................................................22
Motherboard Block Diagram .............................................................................................24
Chapter 2 Server Installation
2.1 Overview.............................................................................................................................25
2.2 Unpacking the System .......................................................................................................25
2.3 Preparing for Setup ............................................................................................................25
Choosing a Setup Location...............................................................................................25
Rack Precautions ..............................................................................................................26
Server Precautions............................................................................................................26
Rack Mounting Considerations .........................................................................................26
Ambient Operating Temperature....................................................................................26
 ............................................................................................................................26
Mechanical Loading.......................................................................................................27
Circuit Overloading ........................................................................................................27
Reliable Ground.............................................................................................................27
2.4 Installing the Rails ............................................................................................................28
2.5 Installing the Server ............................................................................................................30
Chapter 3 Maintenance and Component Installation
3.1 Removing Power ...............................................................................................................31
3.2 Accessing the System........................................................................................................32
3.3 Processor and Heatsink .....................................................................................................35

5
Contents
Processor Overview ..........................................................................................................35
Installation Overview .........................................................................................................36
Installation Procedure Overview....................................................................................36
Create the Processor Carrier Assembly ...........................................................................37
Assemble the Processor Heatsink Module ......................................................................41
Remove the Socket Cover................................................................................................43
Install the PHM..................................................................................................................44
Removing the PHM...........................................................................................................47
Removing the Carrier Assembly from the Heatsink ......................................................49
Removing the Processor from the Carrier Assembly....................................................50
3.4 Memory...............................................................................................................................51
Memory Support................................................................................................................51
Memory Population Guidelines .........................................................................................53
Guidelines Regarding Mixing DIMMs ............................................................................53
DIMM Construction........................................................................................................53
Installing Memory ..............................................................................................................54
3.5 Motherboard Battery...........................................................................................................55
3.6 Storage Drives....................................................................................................................56
Installing Hot-Swap Drives................................................................................................56
M.2 Drives .........................................................................................................................58
Hot-Swap for NVMe Drives...............................................................................................59
Checking the Temperature of an NVMe Drive..................................................................60
3.7 System Cooling ..................................................................................................................61
Changing a System Fan ...................................................................................................61
Motherboard Air Shroud....................................................................................................62
GPU Air Blockers ..............................................................................................................63
3.8 Power Supply .....................................................................................................................64
Power Supply LEDs ..........................................................................................................64
3.9 PCIe Expansion Cards......................................................................................................66
Installing Expansion Cards (1–8)......................................................................................67
Installing Riser Expansion Cards (9–12) ..........................................................................68
Chapter 4 Motherboard Connections
4.1 Rear Connectors ................................................................................................................69
4.2 Power Connections ............................................................................................................69
4.3 Headers and Connectors ...................................................................................................70

Contents
6
4.4 Jumpers..............................................................................................................................72
4.5 LED Indicators....................................................................................................................73
Chapter 5 Software
5.1 Microsoft Windows OS Installation.....................................................................................74
5.2 Driver Installation................................................................................................................76
5.3 SuperDoctor®5...................................................................................................................77
5.4 BMC....................................................................................................................................78
BMC ADMIN User Password ............................................................................................78
Chapter 6 Optional Components
6.1 Power Supply Modules ......................................................................................................79
6.2 NIC Kit................................................................................................................................79
Chapter 7 Troubleshooting and Support
7.1 Information Resources ........................................................................................................80
Website .............................................................................................................................80
Direct Links for the SYS-821GE-TNHR System ...........................................................80
Direct Links for General Support and Information .......................................................80
7.2 BMC Interface .....................................................................................................................81
7.3 Troubleshooting Procedures .............................................................................................82
General Technique ............................................................................................................82
No Power ..........................................................................................................................82
No Video ...........................................................................................................................82
System Boot Failure .........................................................................................................83
Memory Errors ..................................................................................................................83
 ...........................................................................83
When the System Becomes Unstable ..............................................................................83
7.4 BIOS Error Beep (POST) Codes .......................................................................................85
Additional BIOS POST Codes ..........................................................................................85
7.5 Crash Dump Using the BMC Dashboard............................................................................86
7.6 UEFI BIOS Recovery .........................................................................................................87
Overview ...........................................................................................................................87
Recovering the UEFI BIOS Image....................................................................................87
Recovering the Main BIOS Block with a USB Device......................................................87
7.7 CMOS Clear .......................................................................................................................92
7.8 BMC Reset.........................................................................................................................92
7.9 Where to Get Replacement Components..........................................................................93

7
Contents
7.10 Reporting an Issue............................................................................................................93
Technical Support Procedures ..........................................................................................93
Returning Merchandise for Service...................................................................................93
Vendor Support Filing System ..........................................................................................94
7.11 Feedback...........................................................................................................................94
7.12 Contacting Supermicro......................................................................................................95
Appendix A Standardized Warning Statements for AC Systems
Appendix B System Specications

Contents
8
Contacting Supermicro
Headquarters
Address: Super Micro Computer, Inc.
980 Rock Ave.
San Jose, CA 95131 U.S.A.
Tel: [phone]
Fax: [phone]
Email: [email protected] (General Information)
[email protected] (Sales Inquiries)
[email protected] (Gov. Sales Inquiries)
[email protected] (Technical Support)
[email protected] (RMA Support)
[email protected] (Webmaster)
Website: www.supermicro.com
Europe
Address: Super Micro Computer B.V.
Het Sterrenbeeld 28, 5215 ML
's-Hertogenbosch, The Netherlands
Tel: [phone]
Fax: [phone]
Email: [email protected] (Sales Inquiries)
[email protected] (Technical Support)
[email protected] (RMA Support)
Website: www.supermicro.nl
Asia-Pacic
Address: Super Micro Computer, Inc.
3F, No. 150, Jian 1st Rd.
Zhonghe Dist., New Taipei City 235
Taiwan (R.O.C)
Tel: [phone]
Fax: [phone]
Email: [email protected] (Sales Inquiries)
[email protected] (Technical Support)
[email protected] (RMA Support)
Website: www.supermicro.com.tw

9
Chapter 1: Introduction
Chapter 1
Introduction
1.1 Overview
This chapter provides an outline of the functions and features of the SuperServer
SYS-821GE-TNHR. This is a Universal GPU system that mainly supports NVIDIA Restone

System Overview
Motherboard X13DEG-OAD
Chassis CSE-GP801TS
Processors
Supports dual 4th Gen Intel Xeon Scalable processors, ini Socket E (LGA-4677) with four
UPIs (16GT/s max.) and a thermal design power (TDP) of up to 350W.
Supports SP XCC, SP MCC, and Max Series (HBM) SKUs.
Memory Up to 8TB 3DS RDIMM/RDIMM DDR5 ECC memory with speeds up to 4800MT/s (1PDC)
or 4400MT/s (2DPC) in 32 DIMM slots
GPUs HGX H100 8-GPU SXM5 Multi-GPU Board PCIe 5.0 x16 CPU-to-GPU Interconnect,
NVIDIA NVLink with NVSwitch
Storage Twenty-four 2.5" hot-swap front drive bays: 16 NVMe and 8 SATA
Two M.2 NVMe SSDs
Expansion Slots Twelve PCIe 5.0 slots (see Section 3.9for details)
I/O Ports One RJ45 dedicated BMC LAN port
Two USB 3.0 ports; one VGA port
System Cooling Eighteen heavy duty fans with speed control, including fans as part of the power modules
GPU air blockers, motherboard air shrouds
Power Six redundant power supplies with the option for two more; 3000W (PER power supply),
80Plus Titaniium level
Form Factor 8U; 17.2” x 14” x 33.2” (depth) (43.7 x 356 x 843 cm)
A Quick Reference Guide can be found on the product page of the Supermicro website.

compliant with UL and CSA: GP801-Q30X13, GP801-GPU, GP801-30.

10
Chapter 1: Introduction
4 5
1.2 System Features
The following views of the system display the main features. Refer to Appendix B for additional

Front View
Figure 1-1. Front View
Service/Asset Tag
with BMC Password
Control Panel
0 8654321 11 12 13 14 15 16 17 18 19 20 21 22 2310 97
Tray Release Handles
Tray Release Handles
Logical Storage Drive Numbers
Item Description
0to 15 Sixteen 2.5-inch hot-swap NVMe drive bays
16 to 23 Eight 2.5-inch hot-swap SATA drive bays
Fans
GPU Tray
Motherboard Tray
3
21

11
Chapter 1: Introduction
Drive Carrier Indicators
Each drive carrier has two LED indicators: an activity indicator and a status indicator. For
            
             
supported, such as hot spare.
Drive Carrier LED Indicators
Color Blinking Pattern Behavior for Device
Activity
LED
Blue Solid On Idle SAS/NVMe drive installed
Blue Blinking I/O activity
󰀨 Idle SATA drive installed
Status
LED
Red Solid On Failure of drive with RSTe support
Red Blinking at 1 Hz Rebuild drive with RSTe support
Red Blinking with two blinks
and one stop at 1 Hz
Hot spare for drive with RSTe support
Red 
󰀨
Power on for drive with RSTe support
Red Blinking at 4 Hz Identify drive with RSTe support
Green Solid on Safe to remove NVMe drive
Amber Blinking at 1Hz Do not remove NVMe drive

12
Chapter 1: Introduction
Control Panel Features
Features Description
Power Button The main power switch applies or removes primary power from the power supply to the
server but maintains standby power.
UID button/LED/
BMC button
󰀨
LED and a blue LED on the rear of the chassis.
This button can also be used to reset the BMC.
Power LED
Steady on – Power on
Blinking at 4Hz – Checking BIOS/BMC integrity


Blinking at 1Hz and "i" LED red – Fault detected
HDD LED 
NIC LEDs Non-functional, as there are no onboard network connections.
Information LED Alerts operator to several states (noted in the table below).
Power Fail LED Indicates a power supply module has failed.
Information LED
Color, Status Description
Red, solid An overheating condition has occurred
Red, blinking at 1Hz Fan failure, check for an inoperative fan
Red, blinking at 0.25Hz Power failure, check for a non-operational
power supply
Red, blinking at 10Hz CPLD recovery mode error
Control Panel
Figure 1-2. Control Panel
Power Button
UID Button/LED/
BMC Button
Power LED
Power Fail LED
HDD LED
NIC LED 1
Information LED
(Table continued on next page)
NIC LED 2

13
Chapter 1: Introduction
Information LED
Color, Status Description
Blue, solid Unit ID has been activated by switch
Blue, blinking at 1Hz Unit ID has been activated using the BMC
Blue, blinking at 2Hz, and BMC Heartbeat
LED on the motherboard is green BMC is resetting
Blue, blinking at 4Hz BMC is setting factory defaults
Blue, blinking at 10Hz BIOS/BMC is recovering or updating
Red, blinking at 10Hz and the rear UID LED
is blue, blinking at 10Hz CPLD recovery or update is in progress
Front Control Panel LED Indicators
Event Power HDD NIC UID Info Power Fail
Power On Solid On
HDD Activity Blinking
NIC Activity Blinking
Overheat Solid On
Fan Fail Blinking 1Hz
Power Fail Blinking 1/4Hz Solid On
Local UID On Solid On
Remote UID On Blinking 1Hz
Checking BMC/BIOS Blinking at 4HZ
Recovering/Updating
BMC Blinking at 4HZ
BMC 2 Blinks at 4Hz,
1 Pause at 2Hz
󰀨󰀨
BIOS/BMC
Blinking @10Hz
Flash Not Detected or
Golden Image Check
Failed
BMC/BIOS Blinking at
1HZ
CPLD Recovery Mode Blinking @10Hz
(MB UID LED)
Blinking at 10Hz
(FP Red LED)

14
Chapter 1: Introduction
Rear View
Figure 1-3. System: Rear View
System Features: Rear
Feature Description
Fans Five exhaust fans, showing logical numbering
Power Supply/Fans Eight combination power supply modules with system cooling fans, showing logical
numbering
PCIe Slots Twelve slots for PCIe expansion cards, showing logical numbering
Input/Output Panel See next page for details
10
11
Input/Output Panel
9
2 1
3
4
65
7
8
Fans
Power Supply/Fans
Power Supply/Fans
12
Expansion Tray
10
8
679
7
65
4
3
21
8

15
Chapter 1: Introduction
Expansion Slot Locations
Slot Description
1PCIe 5.0 x16 (LP) from PLX switch to GPUs
2PCIe 5.0 x16 (LP) from PLX switch to GPUs
3PCIe 5.0 x16 (LP) from PLX switch to GPUs
4PCIe 5.0 x16 (LP) from PLX switch to GPUs
5PCIe 5.0 x16 (LP) from PLX switch to GPUs
6PCIe 5.0 x16 (LP) from PLX switch to GPUs
7PCIe 5.0 x16 (LP) from PLX switch to GPUs
8PCIe 5.0 x16 (LP) from PLX switch to GPUs
9PCIe 5.0 x16 (FHFL)
10 PCIe 5.0 x16 (FHFL)
11 PCIe 5.0 x16 (FHFL)
12 PCIe 5.0 x16 (FHFL)
Figure 1-4. Input/Output Panel
BMC LAN Port USB VGAUID LED
UID Switch
BMC LAN Port
A dedicated LAN port provides access to the BMC.
Unit Identier Switch/UID LED Indicator

󰀨
using the BMC.

16
Chapter 1: Introduction
Power Supply Indicator
Power Supply Indicator
LED Color and State Power Supply Condition
Solid Green Indicates that the power supply is on and working
Blinking Green 󰀨
Solid Amber Indicates failure or needs attention
󰀨 No AC power to module
Fan Module Indicator
Fan Module Indicator
LED Color and State Condition
Solid Green Indicates that the fan is on and working
Solid Amber Indicates failure or needs attention
Solid Red Indicates that the fan is receiving standby power
󰀨 󰀨

17
Chapter 1: Introduction
Top View, Motherboard Tray and Switch Module
Figure 1-5. Motherboard Tray: Top View
Motherboard Tray Features
Feature Description
Switch Module Platform for switches and I/O
PCIe Switches Four switches to facilitate direct communication between GPUs
M.2 Slots Two slots for PCIe 3.0 x2 M.2 NVMe
Processors Dual 4th Gen Intel Xeon Scalable with active heatsinks
DIMM Slots Thirty-two memory slots
DIMM Slots
Processors/
Heatsinks
M.2 Slots
(Under Switch module)
PCIe Switches
Switch Module

18
Chapter 1: Introduction
Figure 1-6. Motherboard Tray: Components
Better Faster Greener ™ © 2022 Supermicro10
CSE-GP801 CPU Tray
AOM-GP801 - SW/AD
P
AOM-GP801-SW/ADP
AOM-SXM5-IO
MB-X13DEG
BPN-PDB-GP802
BPN-PDB-GP802
Motherboard Tray Features
Feature Description
AOM-GP801-SW/ADP Two add-on modules to facilitate the switch module OR
NIC adaptor board with PLX switch for Delta-Next GPU system
AOM-SXM5-IO PCIe Gen5 switch board for 8 SXM5 GPU system
Storage Backplane
BPN-NVME5-HS219N-S24 NVMe Gen5 24-slot direct-attached backplane
BPN-PDB-GP802 Two power distribution boards (extension board)
Storage Backplane

19
Chapter 1: Introduction
Top View, GPU Tray
Figure 1-7. Nvidia HGX H100 8-GPU baseboard: Top View

20
Chapter 1: Introduction
1.3 System Block Diagram
The block diagram below shows the connections and relationships between the subsystems

Figure 1-8. System Block Diagram
2x M.2 NVMe