Inspur NF5888M5 User manual

Inspur Server User Manual
NF5888M5 (AGX-5)
V1.1

© Copyright Inspur 2018. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means
without prior written consent of Inspur.
The information in this manual is subject to change without notice.
Inspur is the registered trademark of Inspur. All the other trademarks or registered
trademarks mentioned in this manual are the property of their respective holders.
Edion:1.1
Sep.2019

Abstract
This manual contains technical informaon such as specicaons, hardware operaons,
soware conguraon, fault diagnosis, etc., that are relevant to the maintenance and
operaon of this server.
It is recommended that server installaon, conguraon and maintenance is performed by
experienced technicians only.
Target Audience
This manual is intended for:
●Technical support engineers
●Product maintenance engineers
●Technicians
Warnings:
This manual introduces the NF5180M5 server’s technical features, system installaon and setup,
which will help the user to understand how best to ulize the server and all its funconalies.
1.For your safety, please do not disassemble the server’s components arbitrarily. Please do
not extend conguraon or connect other peripheral devices arbitrarily. If needed, please
contact Inspur for our support and guidance.
2.Before disassembling the server’s components, please be sure to disconnect all the power
cords connected to the server.
3.BIOS and BMC setup is a signicant factor in correctly conguring your server. If there are
no special requirements, it is suggested to use the Default Values and not alter the parameter
sengs arbitrarily. Aer the rst login, please change the BMC user password in me.
4.Please install the product-compable operang system and use the driver provided by
Inspur. If you use an incompable operang system or non-Inspur driver, it may cause
compability issues and aect the normal use of the product, Inspur will not assume any
responsibility or liability.
Inspur is not responsible for any damages, including loss of prots, loss of informaon,
interrupon of business, personal injury, and/or any damage or consequenal damage
without limitaon, incurred before, during, or aer the use of our products.

Contents
1 Safety Instructions......................................................................................................................... 1
2 Product Specification .................................................................................................................... 5
2.1 Overview .................................................................................................................................... 5
2.2 Features and specifications ........................................................................................................ 5
3 Component identification.............................................................................................................. 7
3.1 Front panel components ............................................................................................................ 7
3.2 Front control panel buttons and indicators................................................................................ 8
3.3 Hard drive tray indicators ........................................................................................................... 8
3.4 Rear panel components.............................................................................................................. 9
3.5 Motherboard components ......................................................................................................... 10
4 Operations..................................................................................................................................... 13
4.1 Power up the server ................................................................................................................... 13
4.2 Power down the Server .............................................................................................................. 13
4.3 Extend the server from the rack................................................................................................. 13
4.4 Remove the access panel ........................................................................................................... 14
4.5 Install the access panel............................................................................................................... 15
4.6 Install the server......................................................................................................................... 15
5 Setup ............................................................................................................................................. 17
5.1 Optimum environment............................................................................................................... 17
5.2 Rack warnings............................................................................................................................. 19
5.3 Identifying the contents of the server shipping carton ..............................................................20
5.4 Installing hardware options........................................................................................................ 20
5.5 Installing the server into the rack............................................................................................... 20
5.6 Installing the operating system .................................................................................................. 21
6 Hardware options installation ....................................................................................................... 22
6.1 Overview .................................................................................................................................... 22

6.2 Processor option......................................................................................................................... 22
6.3 Memory option .......................................................................................................................... 24
6.4 Hard drive option ....................................................................................................................... 26
6.5 Power supply option................................................................................................................... 27
6.6 Air baffle option ......................................................................................................................... 28
7 Cabling........................................................................................................................................... 30
8 BIOS setup ..................................................................................................................................... 32
8.1 Common operations................................................................................................................... 32
8.2 BIOS parameter description ....................................................................................................... 50
8.3 Firmware update ........................................................................................................................ 92
9 BMC settings.................................................................................................................................. 97
9.1 Motherboard BMC settings ........................................................................................................ 97
9.2 Switch board BMC settings......................................................................................................... 125
10 Common faults and troubleshooting........................................................................................... 152
10.1 Hardware problems.................................................................................................................. 152
10.2 Software problems ................................................................................................................... 155
11 Battery replacement.................................................................................................................... 157
12 Regulatory compliance notices ................................................................................................... 158
12.1 Regulatory compliance identification numbers........................................................................ 158
12.2 Federal communications commission notice ........................................................................... 158
12.3 European Union regulatory notice ........................................................................................... 159
12.4 Disposal of waste equipment by users in the European Union................................................159
12.5 Korean notice ........................................................................................................................... 160
12.6 Chinese notice .......................................................................................................................... 160
12.7 Battery replacement notice...................................................................................................... 160
13 Electrostatic discharge................................................................................................................. 162
13.1 Preventing electrostatic discharge ........................................................................................... 162

13.2 Grounding methods to prevent electrostatic discharge........................................................... 162
14 Warranty...................................................................................................................................... 163
14.1 Introduction.............................................................................................................................. 163
14.2 Warranty service ...................................................................................................................... 163
14.3 Warranty exclusions ................................................................................................................. 164

1
Safety Introductions
1. Safety instructions
WARNING: Please be advised to follow the instructions below for safety. Failure to do so
could result to potential dangers that may cause property loss, personal injury or death.
1. The power supplies in the system may produce high voltages and energy hazards that
may cause personal injury. For your safety, please do not attempt to remove the cover
of the system to remove or replace any component without assistance provided by
Inspur. Only service technicians trained by Inspur are authorized to remove the cover of
the host, and to remove and replace internal components.
2. Please connect the equipment to the appropriate power sup0ply. Use only power supplies
with the correct voltage and electrical specifications according to the label. To protect your
equipment from damages caused by a momentary spike or plunge of the voltage, please
use relevant voltage stabilizing equipment, or uninterruptible power supplies.
3. If you must use an extension cable, please use a three-core cable with properly
grounded plugs. Observe extension cable ratings. Ensure that the total rating of all
equipment plugged into the extension cable does not exceed 80 percent of the ratings
limit for the extension cable.
4. Please be sure to use the power supply components that come with the server, such as
power lines, power socket (if provided with the server) etc. For your safety, please do
not replace power cables or plugs randomly.
5. To prevent electric shock dangers caused by leakage in the system, please make sure
that the power cables of the system and peripheral equipment are correctly connected
to the earthed/grounded power socket. Please connect the three-core power line plug
to the three-core AC power socket that is well earthed and easy to access. Be sure
to use earthing /grounding pin of power lines and do not use the patch plug or the
earthing/grounding pin unplugged with cables. In the case that the earthing/grounding
conductors are not installed and it is uncertain whether there are appropriate earthing/
grounding protections, please do not use or attempt to operate the equipment. Contact
and consult an electrician.
6. Please do not push any objects into the openings of the system. Doing so may cause
fire or electric shock.
7. Please place the system far away from the cooling plate and heat sources, and be sure

2
not to block the air vents.
8. Please be sure not to scatter food or liquid in the system or on other components, and
do not use the product in humid or dusty environments.
9. Using an incompatible battery may cause explosion. When battery replacement is
required, please consult the manufacturer first, and choose batteries of the same or
equivalent type. Do not disassemble, crush, puncture the batteries or make the external
connection point short circuit, and do not expose them in the environment over 60°C.
Never throw batteries into fire or water. Please do not attempt to open or repair the
batteries. Dispose of used batteries according to instructions. For battery recycling,
please contact the local waste recycling center.
10. Before installing equipment into the rack, please install all front and side stabilizers on
the independent rack first. Please install the front stabilizers first, if connecting with
other racks. Please install stabilizers before installing equipment into the rack. Failure to
install the corresponding stabilizers before installing equipment into the rack may cause
the cabinet to tip over, possibly resulting to severe injury. After installing the equipment
and other components into the rack, only one component can be pulled out from the
rack through its sliding part at one time. Pulling out several components at the same
time may cause the rack to turn over, resulting to serious personal injury.
11. A minimum of two people are required to safely move a rack. The racks are extremely
awkward and heavy, moving them without adequate, trained personnel could result in
severe injury or death.
12. It is prohibited to directly short-circuit the copper busbar. Please do not touch the
copper busbar when the rack is powered on.
13. This is Class A product, and may cause radio interference. In such case, users may need
to take necessary measures to mitigate the interference.
14. The equipment is intended for installation in a Restricted Access Location.
Note: The following considerations may help avoid the occurrence of problems that could
damage the components or cause data loss, etc.
1. In the event of the following, please unplug the power line plug from the power socket
and contact Inspur’s customer service department:
1) The power cables, extension cables or power plugs are damaged.
2) The products get wet.

3
Safety Introductions
3) The products have fallen or have been damaged.
4) Other objects have fallen into the products.
5) The products do not or are unable to function normally even when attempting to
operate according to the instructions.
2. If the system becomes wet or damp, please follow these steps:
1) Power off the equipment, disconnect them with the power socket, wait for 10 to 20
seconds, and then open the host cover.
2) Move the equipment to a well-ventilated place to dry the system at least for 24 hours
and make sure that the system is fully dried.
3) Close the host cover, reconnect the system to the power socket, and then power on.
4) In case of operation failure or other abnormal situations, please contact Inspur and get
technical support.
3. Pay attention to the position of system cables and power cables-avoid placing wires in
high foot traffic locations. Please do not place objects on the cables.
4. Before removing the host cover, and/or touching the internal components, please
allow for the equipment to cool first. To avoid damaging the mainboard, please power
off the system and wait for five seconds, and then remove the components from the
mainboard and/or disconnect the peripheral device from the system. Please remember
that only service technicians trained by Inspur are authorized to remove the cover of
the host, and to remove and replace internal components.
5. If there is modem, telecom or LAN options installed in the equipment, please pay
attention to the followings:
1) In the case of thunder and lightning, please do not connect or use the modem.
2) Never connect or use the modem in a damp environment.
3) Never insert the modem or telephone cables into the socket of network interface
controller (NIC).
4) Before unpacking the product package, installing internal components, touching
uninsulated cables or jacks of the modem, please disconnect the modem cables.
6. In order to prevent electrostatic discharge from damaging the electronic components in
the equipment, please pay attention to the followings:
1) Please remove any static electricity on your body before dismounting or touching any
electronic component in the equipment, to prevent the static electricity from conducting
itself to the sensitive components. You may remove the static electricity on the body by

4
touching the metal earthing objects (such as the unpainted metal surface on the rack).
2) Please do not take electrostatic sensitive components that are not ready to be installed
for application out of the antistatic package materials.
3) While working, please touch the earthing conductor or the unpainted metal surface on
the cabinet regularly to remove any static electricity from the body that may damage
the internal components.
7. Upon receiving the proper authorization from Inspur and dismounting the internal
components, please pay attention to the following:
1) Switch the system power supply off and disconnect the cables, including all connections
of the system. When disconnecting the cables, please hold the connector of the cables
and slowly pull the plugs out. Never pull on the cables.
2) The products need to completely cool down before dismounting the host cover or
touching the internal components.
3) During the dismounting process, avoid making large movement ranges to prevent
damage to the components or scratching arms.
4) Handle components and plug-in cards with care. Please do not touch the components
or connection points on the plug-in cards. When handling the plug-in cards or
components, firmly grab the edges of the plug-in cards and components, and/or their
metal fixed supports.
8. During the process of rack installation and application, please pay attention to the followings:
1) After the rack installation is finished, please ensure that the stabilizers have been fixed
to the rack and supported to ground, and the weight of the rack is firm on ground.
2) Always load from the bottom up, and load the heaviest items first.
3) When pulling out the components from the rack, apply slight force to keep the rack balanced.
4) When pressing down the release latch and the rail of components is sliding, please be
careful; as the sliding may hurt your fingers.
5) Do not overload the AC power supply branch circuits in the rack. The total load of the
rack should not exceed 80% of the ratings of the branch circuits.
6) Ensure that components in the rack have good ventilation conditions.
7) When repairing components in the rack, never step on any other components.
9. After receiving the server, please refer to the label on the top cover to remove the shipping
screws on both sides of the chassis and then install it on the rack, to avoid damage to the
handles when you open the GPU BOX later without removing the screws.

5
Product Specication
2 Product specifications
2.1 Overview
Inspur server NF5888M5 (AGX-5) is an artificial intelligence (AI) oriented supercomputer
platform. The 8U chassis supports up to 16 fully interconnected NVIDIA(R) Tesla (R) V100
GPUs and provides 2-petaFLOPS AI computing performance. The global GPU interconnection
bandwidth is up to 2400GB/s, which guarantees near linear ultra high speed-up ratio for
any AI training program in the architecture. A single system may shorten the training period
from one week to twelve hours. The 3000w (2+2)*2 power supplies and whole modular
design create high reliability and ultra maintainability of the entire system in the AI cloud
scenarios.
Fig. 2.1 NF5888M5 appearance
2.2 Features and specifications
Processor
Processor type 2* new-generaon Intel® Xeon® scalable processors, TDP 205W
Chipset
Chipset type Intel® C620 series chipset (Lewisburg-2)
Memory
Memory type DDR4 RDIMM/LRDIMM 2666MHz
Number of memory slots 24

6
Total memory capacity Supports up to 3,072GB (128G per bank)
I/O
USB interface 2* front USB3.0 interfaces, 1* rear SUV serial interface (including 2* USB2.0
interfaces)
PCIe interface 4* external PCIe connectors
VGA interface 1* front VGA interface, 1* SUV serial interface (including 1* VGA interface)
Management network
interface 1* front and 1* rear RJ45 IPMI interfaces
PCIE card 4* PCIe x16 cards, 1* PCIe x8 card
Display
Controller type Integrated in Aspeed 2500 chip, supporng resoluon up to 1900*1200
Hard drive
Hard drive type
SATA/NVME disk, supporng
8* 2.5” SATA disks
Or 4* 2.5” SATA disks + 4* 2.5” NVMe disks
Power
Specicaons 8* 3000W 80Plus Planum PSU, supporng upper/lower (2+2)*2 redundancy
Power input Please refer to the power input on the nameplate label of the host.
Physical
External packing
dimensions 722mm (W) x 586 mm (H) x 1168mm (D)
Mainframe dimensions 448mm (W) x 351.6mm (H) x 850mm (D)
Weight
Full conguraon
NW 130kg, GW 160kg (Gross weight including mainframe + packing box + guide
rail + accessory box)
Environmental
Working temperature 10℃～ 35℃
Storage and shipment
temperature -40℃～ 60℃
Working humidity 20% ～80% relave humidity
Storage and shipment
humidity 20% ～93% (40℃) relave humidity

7
Component Identication
3 Component identification
3.1 Front panel components
No. Module name
1 Front panel buttons and indicators
2UID indicator and button
3 System reset button
4 System fault indicator
5-12 Hard disk 0-7
13 USB3.0 slot x2
14 BMC management interface
15 Network interface x2
16 VGA interface
17 UID indicator and button
18 BMC debug interface
19-24 NVLINK Bridge 1-6
25 Front control panel 1
26 Front control panel 2
27 GPU BOX 1
28 GPU BOX 2

8
3.2 Front control panel buttons and indicators
No. Module name Descripon
1 Front control panel 1 When the equipment works as a GPU server, front control panel 1 is
used to control the equipment.
When the equipment works as a GPU BOX, front control panel 1 is
used to control GPU BOX 1 (upper), front control panel 2 controlling
GPU BOX 2 (lower).
2 Front control panel 2
3On/off button and indicator
Green on in power-on status
Orange on in standby status
Long-press to force shutdown
4UID indicator and button Activate/deactivate UID, blue on
5 System reset button Short press to force system rest
6 System fault indicator
O in normal status
Steady red in case of power supply failure
Flashing red in case of abnormal power supply status
3.3 Hard drive tray indicators

9
Component Identication
No. Module name Descripon
1Hard disk fault alarming
indicator
Steady red: Hard disk in fault
Steady blue: Hard disk posioning
Steady pink: Working with RAID Rebuilding
2Hard disk acvity status
indicator
Steady green: Normal
Flashing green: Hard disk in reading/wring
3.4 Rear panel components
No. Module name
1-4 PSU0-3
5 FAN 0-11
6I/O box A
7 External PCIe connector A0-A3
8 PCIE interface 4
9 PCIE interface 0
10-11 PSU4-5
12 I/O box C
13 External PCIe connector C0-C3
14-15 PCIE interface 2-3
16 BMC serial interface
17 UID button and indicator

10
No. Module name
18 BMC management interface
19-20 External PCIe connector D0-D4
21 I/O box D
22-23 PSU6-7
24 PCIE interface 1
25 BMC management interface
26 SUV connector
27-28 External PCIe connector B0-B4
29 I/O box B
30 UID indicator and button
3.5 Motherboard components
No. Module name
1Memory slot (CPU1)
2Memory slot (CPU0)
3 CPU1
4 CPU0
5 Motherboard handle 1
6 Slimline interface x12

11
Component Identication
No. Module name
7 Hard disk backplane power supply interface
8Hard disk backplane signal interface J91
9Hard disk backplane signal interface J92
10 M.2 Riser slot
11 Right-hand front panel interface
12 PCH SD card slot
13 BMC SD card slot
14 CLR_CMOS
15 Motherboard handle 0
16 UID button
17 BMC debug interface
18 BMC management interface/USB3.0 slot x2
19 10G on-board network interface x2
20 VGA interface
21 TPM interface
22 KVM interface
23 MINISAS interface 0
24 MINISAS interface 1
25 MINISAS interface 2
26 MB PDB interface
For the position of the CMOS clear jumper, see 3.5 Motherboard Components.
Item Funcon descripon Jumper funcon
CLR_CMOS CMOS clear jumper Short-circuit pin1-2, normal status;
Short-circuit pin2-3, clear CMOS.
Note:
It is required to shut down the system, as well as disconnect the power supply during CMOS
clearing. Hold for 5 seconds after short-circuiting Pin2-3, and then short-circuit Pin1 and
Pin2 (the default status) of CLR_CMOS jumper with a jumper cap, to restore to its original
status.
3.6 GPU components
The upper layer of the GPU module is GPU0-GPU7, and the lower layer is GPU8-GPU15, as

12
shown below:

13
Operations
4 Operations
4.1 Power up the server
Insert the power cord plug, then press the Power Button.
4.2 Power down the Server
Alarm! To minimize the risk of personal injury, electric shock or equipment damage,
please unplug the power cord to disconnect the power source. The Power button on the
front panel cannot disconnect the power supply thoroughly. Before the power supply is
disconnected, some power supplies and internal circuits are still alive.
Note: If you are installing a hot-pluggable device, you do not need to disconnect the power
supply.
1. Back up the server data.
2. Shut down the operating system gracefully.
3. Unplug the power cord.
The system is now in the power-off position.
4.3 Extend the server from the rack
1. Follow the rack mounting instructions, and loose the four captive screws within the ears
at both sides by using a screwdriver.
2. Pull out the server slowly out of the rack. Install the lifting handles at both sides and carry
it onto an antistatic bench.
Alarm! To minimize the risk of personal injury and equipment damage, ensure the rack
is stable enough before pulling out the components from the rack. A single server weighs
above 100kg and should be lifted by at least five people or by using auxiliary equipment.

14
4.4 Remove the access panel
Alarm! To minimize the risk of personal injury caused by overheated surfaces, do not touch
the surface until the drives and internal components cool down.
Note: For the purpose of sufficient heat dissipation, do not run the server without the
access panel, fan cover or fans. If the server supports some hot-pluggable components,
please try to open the access panel for a short period.
Removing component:
1. For the non-hot-pluggable installation or maintenance, shut down the server power
supply.
2. Pull out the server out of the rack.
3. Use a screwdriver to loosen the safety screws on the cover locker.
4. Lift the handle on the cover locker and remove the access panel.