Huawei E9000 Operating instructions
Huawei Servers
Troubleshooting
Issue 20
Date 2020-09-25
HUAWEI TECHNOLOGIES CO., LTD.
Copyright © Huawei Technologies Co., Ltd. 2020. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior
written consent of Huawei Technologies Co., Ltd.
Trademarks and Permissions
and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective
holders.
Notice
The purchased products, services and features are stipulated by the contract made between Huawei and
the customer. All or part of the products, services and features described in this document may not be
within the purchase scope or the usage scope. Unless otherwise specied in the contract, all statements,
information, and recommendations in this document are provided "AS IS" without warranties, guarantees
or representations of any kind, either express or implied.
The information in this document is subject to change without notice. Every eort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, express or implied.
Huawei Technologies Co., Ltd.
Address: Huawei Industrial Base
Bantian, Longgang
Shenzhen 518129
People's Republic of China
Website: https://e.huawei.com
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. i
About This Document
Overview
This document describes how to collect logs, diagnose faults, upgrade software,
perform preventive maintenance and common operations, and collect the
information required to for troubleshoot Huawei E9000, E6000, X6000, X8000,
X6800, rack, heterogeneous, Atlas 800 AI inference (model 3010), and Atlas 800 AI
training (model 9010) servers.
It guides you through the server troubleshooting process.
Intended Audience
This document is intended for:
● Technical support engineers
● Maintenance engineers
Symbol Conventions
The symbols that may be found in this document are dened as follows.
Symbol Description
Indicates a hazard with a high level of risk which,
if not avoided, will result in death or serious
injury.
Indicates a hazard with a medium level of risk
which, if not avoided, could result in death or
serious injury.
Indicates a hazard with a low level of risk which,
if not avoided, could result in minor or moderate
injury.
Huawei Servers
Troubleshooting About This Document
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. ii
Symbol Description
Indicates a potentially hazardous situation which,
if not avoided, could result in equipment damage,
data loss, performance deterioration, or
unanticipated results.
NOTICE is used to address practices not related to
personal injury.
Supplements the important information in the
main text.
NOTE is used to address information not related
to personal injury, equipment damage, and
environment deterioration.
Change History
Issue Date Description
20 2020-09-25 This issue is the twentieth ocial release.
Updated 5.6 Handling Faults Based on
Symptoms.
19 2020-07-16 This issue is the nineteenth ocial
release.
Added information about the Atlas 800
AI training server (model 9010).
18 2020-05-12 This issue is the eighteenth ocial
release.
Changed FusionServer G5500 to
FusionServer Pro G5500.
17 2020-04-29 This issue is the seventeenth ocial
release.
Deleted contents related to the
ServiceCD.
16 2019-09-30 This issue is the sixteenth ocial release.
Added information about Atlas 800 AI
inference server (model 3010).
15 2019-09-19 This issue is the fteenth ocial release.
Added contents related to the MM920/
MM921.
14 2019-07-05 This is the fourteenth ocial release.
Added information about SmartKit.
Huawei Servers
Troubleshooting About This Document
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. iii
Issue Date Description
13 2019-01-08 This is the thirteenth ocial release.
12 2018-07-13 This is the twelfth ocial release.
11 2018-05-18 This is the eleventh ocial release.
Added description about the FusionServer
G2500 heterogeneous server.
10 2018-03-12 This issue is the tenth ocial release.
Added information about the
FusionServer Pro G5500 server.
09 2017-12-14 This issue is the ninth ocial release.
Added description about the CX916
switch module of the E9000 server.
08 2017-08-08 This issue is the eighth ocial release.
Modied 4.4.1.1 Connecting a PC to the
Ethernet Switching Plane.
07 2017-07-20 This issue is the seventh ocial release.
Modied 4.4.2.1 Collection Method.
06 2017-04-18 This issue is the sixth ocial release.
Added description that faulty E9000
compute nodes cannot be reseated in 5.5
Checking Indicators to Locate Faults.
05 2016-10-27 This issue is the fth ocial release.
Added the quick recovery method for
E9000 switch modules in 5.6 Handling
Faults Based on Symptoms.
04 2016-07-11 This issue is the fourth ocial release.
●Modied 4.4.2.7 Using the Switch
Module CLI to Collect FC Switching
Plane Information (MX210/MX220).
● Added the quick recovery method in
5.6 Handling Faults Based on
Symptoms.
03 2016-05-10 This issue is the third ocial release.
● Deleted the "Using the Web Tools of a
Switch Module to Collect Information
About an FC Switching Plane (NX120/
NX220/MX210/MX220)" section.
●Modied 4.4.2.4 Using the V8 Switch
Module CLI to Collect Ethernet
Switching Plane Information.
● Added 9 Other Resources.
Huawei Servers
Troubleshooting About This Document
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. iv
Issue Date Description
02 2015-10-27 This issue is the second ocial release.
● Added 5.5 Checking Indicators to
Locate Faults.
● Added description about how to
collect FreeBSD and Solaris host
information in 4.2 Collecting OS
Logs.
01 2015-10-09 The issue is the rst ocial release.
Huawei Servers
Troubleshooting About This Document
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. v
Contents
About This Document................................................................................................................ ii
1 Safety Instructions.................................................................................................................. 1
2 Troubleshooting Process........................................................................................................ 5
3 Preparing for Troubleshooting............................................................................................. 7
4 Collecting Information.........................................................................................................11
4.1 Collecting Basic Information............................................................................................................................................. 11
4.2 Collecting OS Logs................................................................................................................................................................12
4.3 Collecting Hardware Logs.................................................................................................................................................. 13
4.4 Collecting Switch Module Logs (for E9000+MM910).............................................................................................. 14
4.4.1 Preparing for Log Collection.......................................................................................................................................... 14
4.4.1.1 Connecting a PC to the Ethernet Switching Plane............................................................................................. 14
4.4.1.2 Querying the Software Version of the Ethernet Switching Plane................................................................. 17
4.4.2 Collecting Switch Module Logs.....................................................................................................................................18
4.4.2.1 Collection Method......................................................................................................................................................... 18
4.4.2.2 Using SmartKit to Collect Switch Module Logs...................................................................................................20
4.4.2.3 Using the V5 Switch Module CLI to Collect Ethernet Switching Plane Information.............................. 20
4.4.2.4 Using the V8 Switch Module CLI to Collect Ethernet Switching Plane Information.............................. 23
4.4.2.5 Using the Web Tools Page of a Switch Module to Collect FC Switching Plane Information (MX510)
............................................................................................................................................................................................................ 28
4.4.2.6 Using the Switch Module CLI to Collect FC Switching Plane Information (MX510).............................. 30
4.4.2.7 Using the Switch Module CLI to Collect FC Switching Plane Information (MX210/MX220)...............32
4.5 Collecting Switch Module Logs (for E9000+MM910/MM921)............................................................................. 35
4.6 Collecting Qlogic HBA Logs.............................................................................................................................................. 36
4.7 Collecting Other Logs.......................................................................................................................................................... 36
5 Diagnosing and Rectifying Faults......................................................................................37
5.1 Fault Diagnosis Rules.......................................................................................................................................................... 37
5.2 Using Tools to Diagnose Faults........................................................................................................................................38
5.3 Handling Alarms................................................................................................................................................................... 39
5.4 Using Error Codes to Locate Faults................................................................................................................................ 40
5.5 Checking Indicators to Locate Faults............................................................................................................................. 41
5.6 Handling Faults Based on Symptoms............................................................................................................................ 77
Huawei Servers
Troubleshooting Contents
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. vi
5.6.1 Power Failures.................................................................................................................................................................... 79
5.6.2 KVM Login Faults.............................................................................................................................................................. 83
5.6.3 POST Faults......................................................................................................................................................................... 85
5.6.4 Memory Errors....................................................................................................................................................................92
5.6.5 Drive I/O Faults.................................................................................................................................................................. 94
5.6.6 Ethernet Controller Faults.............................................................................................................................................. 96
5.6.7 FC Controller Faults........................................................................................................................................................ 102
5.6.8 Switch Module Faults.................................................................................................................................................... 106
5.6.9 OS Faults............................................................................................................................................................................ 108
6 Software and Firmware Upgrade.................................................................................... 113
7 Preventive Maintenance....................................................................................................115
7.1 Inspecting the Equipment Room Environment and Cable Layout.................................................................... 115
7.1.1 Precautions........................................................................................................................................................................ 115
7.1.2 Inspecting the Equipment Room Environment.....................................................................................................116
7.1.3 Inspecting Cable Layout............................................................................................................................................... 116
7.2 Inspecting Servers...............................................................................................................................................................117
7.2.1 Precautions........................................................................................................................................................................ 117
7.2.2 Inspecting Indicators...................................................................................................................................................... 117
7.2.3 Using SmartKit to Perform Health Inspection...................................................................................................... 118
7.2.4 Checking the System Status Through iBMC.......................................................................................................... 118
7.3 Huawei Server Inspection Report..................................................................................................................................120
8 Common Operations.......................................................................................................... 125
8.1 Obtaining a Product SN................................................................................................................................................... 126
8.2 Using iMana 200 to Collect Information in Batches.............................................................................................. 134
8.3 Using iBMC to Collect Information in Batches.........................................................................................................135
8.4 Using the MM910 WebUI to Collect Information in Batches (for Versions Earlier Than U54 2.20)..... 138
8.5 Using the MM910 WebUI to Collect Information in Batches (for U54 2.20 or Later)...............................138
8.6 Using the FusionDirector WebUI to Collection Information in Batches.......................................................... 139
8.7 Using the MM510 CLI to Collect Information (FusionServer Pro G5500)...................................................... 139
8.8 Logging In to the iMana 200 WebUI...........................................................................................................................140
8.9 Logging In to the iBMC WebUI......................................................................................................................................143
8.10 Logging In to the Web Tools of the MX510........................................................................................................... 148
8.11 Logging In to the MM910 WebUI.............................................................................................................................. 149
8.12 Logging In to the FusionDirector WebUI.................................................................................................................154
8.13 Logging In to the MM510 CLI..................................................................................................................................... 157
8.14 Logging In to the RMC CLI........................................................................................................................................... 159
8.15 Logging In to a Server Over a Network Port by Using PuTTY......................................................................... 163
8.16 Logging In to a Server Over a Serial Port by Using PuTTY............................................................................... 166
8.17 Logging In to a Compute Node, Passthrough Module, or Switch Module by Using the SOL Function
of the MM910............................................................................................................................................................................. 168
8.18 Logging In to a Compute Node, Passthrough Module, or Switch Module by Using the SOL Function
of the MM920/MM921............................................................................................................................................................ 170
Huawei Servers
Troubleshooting Contents
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. vii
8.19 Using WinSCP to Transfer Files................................................................................................................................... 171
8.20 Conguring an FTP Server............................................................................................................................................ 173
8.21 Using SFTP to Transfer Files......................................................................................................................................... 174
9 Other Resources.................................................................................................................. 176
9.1 Obtaining Technical Support.......................................................................................................................................... 176
9.2 Product Information Resources..................................................................................................................................... 177
9.3 Product Conguration Resources..................................................................................................................................178
9.4 Maintenance Tools............................................................................................................................................................. 178
Huawei Servers
Troubleshooting Contents
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. viii
1 Safety Instructions
General Instructions
● Comply with all local laws and regulations when installing the hardware.
These Safety Instructions are only a supplement.
● Observe the instructions that accompany all "DANGER", "WARNING",
"CAUTION", and "NOTE" symbols in this document. Follow them in
conjunction with these Safety Instructions.
● Observe all safety instructions provided on the device labels when installing
hardware. Follow them in conjunction with these Safety Instructions.
● Operations involving high voltages or moving equipment must be performed
by authorized, qualied personnel.
● Take protective measures against radio interference before operating the
device in residential areas.
Personal Safety
● Only personnel certied or authorized by Huawei are allowed to install
equipment or its components.
● Discontinue any dangerous operations and take protective measures. Report
anything that could cause personal injury or equipment damage to a project
supervisor.
● Do not move devices or install cabinets and power cables in hazardous
weather conditions.
● The average weight carried by a person cannot exceed the maximum
acceptable weight of lift (MAWL) allowed by local safety regulations. Before
moving a device, check the maximum device weight and arrange required
personnel.
● Wear clean protective gloves, ESD clothing, a protective hat, and protective
shoes, as shown in Figure 1-1.
Huawei Servers
Troubleshooting 1 Safety Instructions
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 1
Figure 1-1 Protective clothing
● Before contacting devices, wear antistatic clothing and ESD gloves, and take
o electricity-conductive materials such as watches and jewelries, as shown in
Figure 1-2.
Figure 1-2 Conductive objects to be removed
Figure 1-3 shows how to wear an ESD wrist strap.
1. Secure the wrist strap around your wrist.
2. Fasten the strap buckle and ensure that the ESD wrist strap is snug against
the skin.
3. Insert the attached ground terminal into the jack on the grounded rack or
chassis.
Huawei Servers
Troubleshooting 1 Safety Instructions
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 2
Figure 1-3 Wearing a wrist strap
● Exercise caution when using tools that could cause personal injury.
● Use a stacker when lifting hardware above shoulder height.
● Avoid any contact with high-voltage cables.
● Ensure that the device is properly grounded before powering it on.
● Do not use a ladder alone.
● Do not look into optical ports without eye protection.
Equipment Safety
● Use dedicated power cables to ensure equipment and personal safety.
● Use power cables only for dedicated devices.
● When moving a device, hold the handles or bottom of the device. Do not hold
the handle of the installed module, such as a power module, fan module,
drive, or mainboard.
● Connect the power cables to separate power distribution units (PDUs) for
active/standby operation.
Transportation Precautions
● The logistics company engaged to transport the equipment must be reliable
and comply with international standards for transporting electronics. Ensure
that the equipment being transported is always kept upright. Take necessary
precautions to prevent collisions, corrosion, package damage, damp
conditions and pollution.
● Transport the equipment in its original packaging.
● If original packages are not used, package heavy, bulky items (such as chassis
and compute nodes) and fragile components (such as PCIe cards and optical
modules) separately.
NO TE
Use Computing Product Compatibility Checker to query the components supported by
the compute node or server.
● Power o all equipment before transportation. Do not transport hazardous
materials.
Huawei Servers
Troubleshooting 1 Safety Instructions
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 3
Weight Limits Per Person
CA UTION
To reduce the risk of personal injury, comply with local regulations with regard to
the maximum weight one person is permitted to carry.
Table 1-1 lists the maximum weight each person is permitted to carry by
standards organization.
Table 1-1 Maximum handling weight
Organization Weight (kg/lb)
European Committee for Standardization (CEN) 25/55.13
International Organization for Standardization (ISO) 25/55.13
National Institute for Occupational Safety and Health
(NIOSH)
23/50.72
Health and Safety Executive (HSE) 25/55.13
General Administration of Quality Supervision,
Inspection and Quarantine of the People's Republic of
China (AQSIQ)
● Male: 15/33.08
● Female: 10/22.05
Huawei Servers
Troubleshooting 1 Safety Instructions
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 4
2 Troubleshooting Process
Troubleshooting is a process of using appropriate methods to nd the cause of a
fault and rectify the fault. The guideline of troubleshooting is to narrow down the
scope of possible causes for a fault to reduce troubleshooting complexity, identify
the root cause, and rectify the fault.
Figure 2-1 shows the recommended troubleshooting process.
Figure 2-1 Troubleshooting owchart
Table 2-1 Troubleshooting steps
Step Description
3 Preparing for
Troubleshooting
Prepare the manuals and tools required for fault diagnosis
and rectication.
Huawei Servers
Troubleshooting 2 Troubleshooting Process
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 5
Step Description
4 Collecting
Information
Collect comprehensive information for fault diagnosis.
5 Diagnosing
and Rectifying
Faults
Locate the fault and take troubleshooting measures.
9.1 Obtaining
Technical
Support
If a fault is dicult to locate or rectify after you refer to
documents, contact Huawei technical support.
Huawei Servers
Troubleshooting 2 Troubleshooting Process
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 6
3 Preparing for Troubleshooting
Scenarios
This section describes how to prepare for troubleshooting.
Basic Knowledge and Skills
Get familiar with the following basic knowledge and skills before troubleshooting:
● Server product knowledge
● Danger signs and levels
● Server hardware architecture
● Indicators on the front and rear panels
● Systems that run on servers
● Device operating conditions
● Common hardware operations such as power-on and power-o
● Common software operations such as upgrade
● Device maintenance process
Essential Materials
Table 3-1 lists the materials that you must read before routine maintenance for
Huawei servers.
Huawei Servers
Troubleshooting 3 Preparing for Troubleshooting
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 7
Table 3-1 Essential materials for routine maintenance
Document
Type
Description How to Obtain
User Guide Describes the server
structure, specications, and
installation method. Each
Huawei server has a user
guide or maintenance and
service guide.
1. Log in to Support >
Intelligent Servers or
Support > Ascend
Computing.
2. Choose a server model to
access the product page.
3. On the Documentation
tab page, choose
Operation &
Maintenance.
4. View the required user
guide or maintenance and
service guide.
Alarm Handling Describes common alarms
reported to the server iMana
200/iBMC or management
module, and alarm handling
suggestions. Each Huawei
server has an alarm
reference.
1. Log in to Support >
Management Software >
Server Management
Software > iBMC >
Troubleshooting > Alarm
Handling.
2. View the corresponding
alarm handling manual.
Equipment
Room
Management
Regulations
Describes the regulations for
equipment room
management and routine
maintenance.
Comply with the customer's
equipment room
management regulations
during onsite maintenance.
Software Tools
Table 3-2 lists the software tools required for routine maintenance of Huawei
servers.
Table 3-2 Tools for routine maintenance
Tool Server and
Version
Description
FusionServer
Tools Toolkit
Huawei-
developed V2
and V3 servers.
For details, see
the
FusionServer
Tools 2.0 Toolkit
User Guide
.
Diagnoses and congures servers for fault
locating.
Download link: FusionServer Tools
Huawei Servers
Troubleshooting 3 Preparing for Troubleshooting
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 8
Tool Server and
Version
Description
FusionServer
Tools 2.0
SmartKit
For details, see
the
FusionServer
Tools 2.0
SmartKit User
Guide
.
Used for new site deployment and
delivery, troubleshooting, and rmware
upgrade.
Download link: FusionServer Tools
Smart
Provisioning
See the
Smart
Provisioning
User Guide
.
Used to install OSs without a physical
DVD-ROM drive, congure RAID, upgrade
rmware, and perform troubleshooting.
Download link: Smart Provisioning
PuTTY All Huawei
servers of all
versions
Third-party tool used for remote access.
You can obtain the tool from the Internet.
WinSCP All Huawei
servers of all
versions
Third-party tool used for le transfer for
iMana 200/iBMC or the management
module. You can obtain the tool from the
Internet.
WFTPD All Huawei
servers of all
versions
Third-party tool used for le transfer for
the Ethernet switching plane of a switch
module. You can obtain the tool from the
Internet.
CoreFTPServer/
mini-sftp-server
All Huawei
servers of all
versions
Third-party tools used for le transfer for
the FC switching plane of a switch
module. You can obtain the tool from the
Internet.
Hardware Tools
Table 3-3 lists the hardware tools required for routine maintenance of Huawei
servers.
Table 3-3 Hardware tools required for routine maintenance
Tool Description
Floating nut hook Used to guide oating nuts to the holes in the
mounting bars of a rack.
Screwdriver Used to tighten and loosen screws. A screwdriver can be
a at-head, Phillips, hex screwdriver.
Diagonal pliers Used to trim insulation tubes and cable ties.
Multimeter Used to measure the resistance and voltage and to
check connectivity.
Huawei Servers
Troubleshooting 3 Preparing for Troubleshooting
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 9
Tool Description
ESD wrist strap Used to prevent ESD damage when you touch or
operate devices or components.
Electrostatic
discharge (ESD)
gloves
Used to prevent ESD damage to a board or precision
instrument when you insert, remove, or hold them.
Cable tie Used to bind cables.
Ladder Used to perform operations at heights.
PC Used to access the management network port or a
service network port over the network to capture data.
(You need to prepare a network cable.)
Serial cable Used to connect the serial port on the server. The serial
port is usually a DB9 or RJ45 port.
Thermometer and
hygrometer
Used to measure the equipment room temperature and
relative humidity.
Oscilloscope Used to measure the voltage and time sequence.
Huawei Servers
Troubleshooting 3 Preparing for Troubleshooting
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 10
4 Collecting Information
About This Chapter
If a fault occurs on a server, collect logs for fault diagnosis.
Collect logs immediately upon fault occurrence to obtain the original data.
4.1 Collecting Basic Information
4.2 Collecting OS Logs
4.3 Collecting Hardware Logs
4.4 Collecting Switch Module Logs (for E9000+MM910)
4.5 Collecting Switch Module Logs (for E9000+MM910/MM921)
4.6 Collecting Qlogic HBA Logs
4.7 Collecting Other Logs
4.1 Collecting Basic Information
The customer needs to collect basic information listed in Table 4-1 before
submitting a service request.
Table 4-1 Server fault records
Server fault records
Trouble Ticket
No.
Example:
123456
Fault Report
Time
Example:
2015-10-18
20:30:00
Customer Name
Full name of your
organization
Address Example:
20 Baker
Street, New York
Customer
Contact/ASP
Name
Example:
John
Smith
Contact Info
Phone number
and email address
Huawei Servers
Troubleshooting 4 Collecting Information
Issue 20 (2020-09-25) Copyright © Huawei Technologies Co., Ltd. 11
Other manuals for E9000
1
This manual suits for next models
8
Table of contents
Other Huawei Server manuals
Huawei
Huawei V100R002C01 User manual
Huawei
Huawei E9000 User manual
Huawei
Huawei TaiShan 200 User manual
Huawei
Huawei TaiShan 200 Series User manual
Huawei
Huawei Atlas 800 User manual
Huawei
Huawei FusionModule500 User manual
Huawei
Huawei FusionServer Pro 2298 V5 Quick reference guide
Huawei
Huawei SVN2000 Series User manual
Huawei
Huawei Tecal RH5485 User manual
Huawei
Huawei iManager NetEco 6000 User manual
Huawei
Huawei TaiShan 200 User manual
Huawei
Huawei FusionServer Pro 2288X V5 User manual
Huawei
Huawei OptiX PTN Series User manual
Huawei
Huawei PDU8000 Series User manual
Huawei
Huawei FusionServer Pro 2488H V6 Quick reference guide
Huawei
Huawei RH2488 V2 Server BMC User manual
Huawei
Huawei 1288H V5 User manual
Huawei
Huawei FusionServer Pro CH121 V5 V100R005 Manual
Huawei
Huawei FusionServer Pro CH242 V5 Quick reference guide
Huawei
Huawei FusionServer Pro 1288X V5 Quick reference guide
