Huawei TaiShan Operating instructions

TaiShan Servers
Troubleshooting
Issue 12
Date 2022-08-30
HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2022. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior
written consent of Huawei Technologies Co., Ltd.
Trademarks and Permissions
and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective
holders.
Notice
The purchased products, services and features are stipulated by the contract made between Huawei and
the customer. All or part of the products, services and features described in this document may not be
within the purchase scope or the usage scope. Unless otherwise specied in the contract, all statements,
information, and recommendations in this document are provided "AS IS" without warranties, guarantees
or representations of any kind, either express or implied.
The information in this document is subject to change without notice. Every eort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, express or implied.
Huawei Technologies Co., Ltd.
Address: Huawei Industrial Base
Bantian, Longgang
Shenzhen 518129
People's Republic of China
Website: https://e.huawei.com
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. i

About This Document
Overview
This document describes how to collect logs, diagnose faults, and perform
upgrade and preventive maintenance for Huawei TaiShan servers.
You can perform routine maintenance by referring to this document. Routine
maintenance includes collecting log information, taking measures based on
alarms and fault symptoms, learning about preventive maintenance items, and
upgrading software.
Intended Audience
This document is intended for:
● Technical support engineers
● Maintenance engineers
Symbol Conventions
Symbols that may be found in this document are dened as follows.
Symbol Description
Indicates a hazard with a high level of risk which, if not
avoided, will result in death or serious injury.
Indicates a hazard with a medium level of risk which, if not
avoided, could result in death or serious injury.
Indicates a hazard with a low level of risk which, if not
avoided, could result in minor or moderate injury.
Indicates a potentially hazardous situation which, if not
avoided, could result in equipment damage, data loss,
performance deterioration, or unanticipated results.
This symbol does not indicate human body injuries.
TaiShan Servers
Troubleshooting About This Document
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. ii

Symbol Description
Supplements the important information in the main text.
NOTE is used to address information not related to personal
injury, equipment damage, and environment deterioration.
Change History
Issue Release Date Description
12 2022-08-30 This issue is the twelfth ocial release.
Added information about the TaiShan
200 server (model 2280) VF.
11 2021-11-30 This issue is the eleventh ocial release.
10 2021-06-21 This issue is the tenth ocial release.
Updated 3 Preparing for
Troubleshooting and 9.4 Maintenance
Tools.
09 2021-02-07 This issue is the ninth ocial release.
Updated 5.6.6 Ethernet Controller
Faults.
08 2020-09-25 This issue is the eighth ocial release.
Updated 5.6 Handling Faults Based on
Symptoms.
07 2020-07-21 This issue is the seventh ocial release.
Updated 8.3 Logging In to the iBMC
WebUI.
06 2020-06-19 This issue is the sixth ocial release.
Added information about the TaiShan
200 server (model 5290).
05 2020-04-29 This issue is the fth ocial release.
Updated 6 Software and Firmware
Upgrade.
04 2019-11-30 This issue is the fourth ocial release.
Added information about the TaiShan
200 server (model 1280).
TaiShan Servers
Troubleshooting About This Document
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. iii

Issue Release Date Description
03 2019-09-30 This issue is the third ocial release.
● Updated 5.5 Using Indicators to
Locate Faults.
● Added information about the TaiShan
200 server (model 2180).
02 2019-07-01 This issue is the second ocial release.
Added 5.4 Using Error Codes to Locate
Faults.
01 2018-11-01 This issue is the rst ocial release.
TaiShan Servers
Troubleshooting About This Document
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. iv

Contents
About This Document................................................................................................................ ii
1 Safety Instructions.................................................................................................................. 1
2 Troubleshooting Process........................................................................................................ 5
3 Preparing for Troubleshooting............................................................................................. 6
4 Collecting Information.........................................................................................................10
4.1 Collecting Basic Information............................................................................................................................................. 10
4.2 Collecting OS Logs................................................................................................................................................................11
4.3 Collecting Hardware Logs.................................................................................................................................................. 12
5 Diagnosing and Rectifying Faults......................................................................................13
5.1 Fault Diagnosis Rules.......................................................................................................................................................... 13
5.2 Using Tools to Diagnose Faults........................................................................................................................................14
5.3 Handling Alarms................................................................................................................................................................... 14
5.4 Using Error Codes to Locate Faults................................................................................................................................ 15
5.5 Using Indicators to Locate Faults.................................................................................................................................... 16
5.6 Handling Faults Based on Symptoms............................................................................................................................ 41
5.6.1 Power Failures.................................................................................................................................................................... 42
5.6.2 KVM Login Faults.............................................................................................................................................................. 46
5.6.3 POST Faults......................................................................................................................................................................... 49
5.6.4 Memory Faults....................................................................................................................................................................53
5.6.5 Drive I/O Faults.................................................................................................................................................................. 55
5.6.6 Ethernet Controller Faults.............................................................................................................................................. 58
5.6.7 OS Faults.............................................................................................................................................................................. 63
6 Software and Firmware Upgrade...................................................................................... 68
7 Preventive Maintenance...................................................................................................... 69
7.1 Inspecting the Equipment Room Environment and Cable Layout.......................................................................69
7.1.1 Precautions.......................................................................................................................................................................... 69
7.1.2 Inspecting the Equipment Room Environment....................................................................................................... 70
7.1.3 Inspecting Cable Layout.................................................................................................................................................. 70
7.2 Inspecting Servers................................................................................................................................................................. 71
7.2.1 Precautions.......................................................................................................................................................................... 71
TaiShan Servers
Troubleshooting Contents
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. v

7.2.2 Inspecting Indicators........................................................................................................................................................ 71
7.2.3 Using SmartKit to Perform Health Inspection.........................................................................................................71
7.2.4 Checking the System Status Through iBMC.............................................................................................................72
7.3 Huawei Server Inspection Report.................................................................................................................................... 73
8 Common Operations............................................................................................................ 78
8.1 Obtaining a Product SN......................................................................................................................................................78
8.2 Using iBMC to Collect Information in One-click Mode........................................................................................... 81
8.3 Logging In to the iBMC WebUI........................................................................................................................................ 84
8.4 Logging In to a Server Over a Network Port by Using PuTTY.............................................................................. 88
8.5 Logging In to a Server Over a Serial Port by Using PuTTY.................................................................................... 90
8.6 Using WinSCP to Transfer Files........................................................................................................................................ 92
9 Other Resources.....................................................................................................................95
9.1 Obtaining Technical Support............................................................................................................................................ 95
9.2 Product Information Resources........................................................................................................................................ 96
9.3 Product Conguration Resources.................................................................................................................................... 97
9.4 Maintenance Tools............................................................................................................................................................... 97
TaiShan Servers
Troubleshooting Contents
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. vi

1 Safety Instructions
General Instructions
● Comply with all local laws and regulations when operating devices. These
safety instructions are only a supplement.
● The "DANGER", "WARNING", "CAUTION", and "NOTE" information in this
document does not represent all the safety instructions, but supplements to
the safety instructions.
● Observe all safety instructions provided on the device labels when installing
hardware. Follow them in conjunction with these safety instructions.
● Only qualied personnel are allowed to perform special tasks, such as
performing high-voltage operations and driving a forklift.
● Take protective measures against radio interference before operating the
device in residential areas.
Personal Safety
● Only personnel certied or authorized by Huawei are allowed to install
devices.
● Discontinue any dangerous operations and take protective measures. Report
anything that could cause personal injury or equipment damage to a project
supervisor.
● Do not move devices or install cabinets and power cables in hazardous
weather conditions.
● The average weight carried by a person cannot exceed the maximum
acceptable weight of lift (MAWL) allowed by local safety regulations. Before
moving a device, check the maximum device weight and arrange required
personnel.
● Wear clean protective gloves, ESD clothing, a protective hat, and protective
shoes, as shown in Figure 1-1.
TaiShan Servers
Troubleshooting 1 Safety Instructions
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 1

Figure 1-1 Protective clothing
● Before contacting devices, wear antistatic clothing and ESD gloves, and take
o electricity-conductive materials such as watches and jewelries, as shown in
Figure 1-2.
Figure 1-2 Conductive objects to be removed
Figure 1-3 shows how to wear an ESD wrist strap.
1. Secure the wrist strap around your wrist.
2. Fasten the strap buckle and ensure that the ESD wrist strap is snug against
the skin.
3. Insert the attached ground terminal into the jack on the grounded rack or
chassis.
TaiShan Servers
Troubleshooting 1 Safety Instructions
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 2

Figure 1-3 Wearing a wrist strap
● Exercise caution when using tools that could cause personal injury.
● Use a stacker when lifting hardware above shoulder height.
● Avoid any contact with high-voltage cables.
● Ensure that the device is properly grounded before powering it on.
● Never use a ladder if you are alone and cannot get assistance.
● Do not look into optical ports without eye protection when installing, testing,
or replacing optical cables.
Equipment Safety
● Use dedicated power cables to ensure equipment and personal safety.
● Use power cables only for dedicated devices.
● When moving a device, hold the handles or bottom of the device. Do not hold
the handle of the installed module, such as a power module, fan module,
drive, or mainboard.
● Connect the power cables of the active and standby PSUs to dierent PDUs to
ensure reliable system operation.
Transportation Precautions
● Hire certied logistics companies to transport the equipment complying with
international standards for transporting electronics. Ensure that the
equipment being transported is always kept upright. Take necessary
precautions against collisions, corrosion, moisture, package damage, and
contamination.
● Transport the equipment in its original packaging.
● If original packages are not used, package heavy, bulky items (such as
chassis) and fragile components (such as PCIe cards and optical modules)
separately.
NO TE
Use the Computing Product Compatibility Checker to query the components supported
by the node or server.
● Power o all equipment before transportation. Do not transport hazardous
materials.
TaiShan Servers
Troubleshooting 1 Safety Instructions
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 3

Maximum Acceptable Weight of Lift (MAWL)
CA UTION
To reduce the risk of personal injury, comply with local regulations with regard to
the MAWL.
Table 1-1 lists the maximum weight each person is permitted to carry by
standards organization.
Table 1-1 MAWLs
Organization Weight (kg/lb)
European Committee for Standardization (CEN) 25/55.13
International Organization for Standardization (ISO) 25/55.13
National Institute for Occupational Safety and Health
(NIOSH)
23/50.72
Health and Safety Executive (HSE) 25/55.13
General Administration of Quality Supervision,
Inspection and Quarantine of the People's Republic of
China (AQSIQ)
● Men: 15/33.08
● Women: 10/22.05
TaiShan Servers
Troubleshooting 1 Safety Instructions
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 4

2 Troubleshooting Process
Troubleshooting is a process of using appropriate methods to nd the cause of a
fault and rectify the fault. The guideline of troubleshooting is to narrow down the
scope of possible causes for a fault to reduce troubleshooting complexity, identify
the root cause, and rectify the fault.
Figure 2-1 shows the recommended troubleshooting process.
Figure 2-1 Troubleshooting owchart
Table 2-1 Troubleshooting steps
Step Description
3 Preparing for
Troubleshooting
Prepare the manuals and tools required for fault diagnosis
and rectication.
4 Collecting
Information
Collect comprehensive information for fault diagnosis.
5 Diagnosing
and Rectifying
Faults
Locate the fault and take troubleshooting measures.
9.1 Obtaining
Technical
Support
If the documents cannot help you to locate or rectify the
fault, contact Huawei technical support.
TaiShan Servers
Troubleshooting 2 Troubleshooting Process
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 5

3 Preparing for Troubleshooting
Scenarios
This chapter describes how to prepare for troubleshooting.
Basic Knowledge and Skills
Get familiar with the following basic knowledge and skills before troubleshooting:
● Server product knowledge
● Danger signs and levels
● Server hardware architecture
● Indicators on the front and rear panels
● Systems that run on servers
● Device operating conditions
● Common hardware operations such as power-on and power-o
● Common software operations such as upgrade
● Device maintenance process
Essential Materials
Table 3-1 lists the materials that you must read before routine maintenance for
servers.
TaiShan Servers
Troubleshooting 3 Preparing for Troubleshooting
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 6

Table 3-1 Essential materials for routine maintenance
Document
Type
Description How to Obtain
User Guide Describes the server
structure, specications, and
installation method. Each
server has a user guide or
maintenance and service
guide.
1. Log in to the support
website and choose
Support > Kunpeng
Computing or Support >
Ascend Computing
2. Choose a server model to
access the product page.
3. On the Documentation
tab page, choose
Operation &
Maintenance.
4. View the required user
guide or maintenance and
service guide.
Alarm Handling Describes common alarms
reported to the server iBMC
or management module, and
alarm handling suggestions.
Each server has an alarm
reference.
1. Log in to the Support >
Management Software >
Server Management
Software > iBMC >
Troubleshooting > Alarm
Handling page.
2. View the corresponding
alarm handling manual.
Equipment
Room
Management
Regulations
Describes the regulations for
equipment room
management and routine
maintenance.
Comply with the customer's
equipment room
management regulations
during onsite maintenance.
Software Tools
Table 3-2 lists the software tools required for routine maintenance of servers.
TaiShan Servers
Troubleshooting 3 Preparing for Troubleshooting
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 7

Table 3-2 Software tools required for routine maintenance
Tool Server and
Version
Description
FusionServer
Tools SmartKit
For details, see
the FusionServer
Tools SmartKit
User Guide.
Used for new site deployment and
delivery, troubleshooting, and rmware
upgrade.
Download addresses:
● Enterprise users: FusionServer Tools
● Carrier users: Contact the technical
support personnel of the local Huawei
oce.
Smart
Provisioning
For details, see
the Smart
Provisioning
User Guide.
Used to install OSs without a physical
DVD-ROM drive, congure RAID, upgrade
rmware, and perform troubleshooting.
Download addresses:
● Enterprise users: Smart Provisioning
● Carrier users: Contact the technical
support personnel of the local Huawei
oce.
PuTTY All servers of all
versions
Third-party tool used for remote access.
You can obtain the tool from the Internet.
WinSCP All servers of all
versions
Third-party tool used for le transfer for
iBMC or the OS. You can obtain the tool
from the Internet.
Hardware Tools
Table 3-3 lists the hardware tools required for routine maintenance of servers.
Table 3-3 Hardware tools for routine maintenance
Tool Description
Floating nut hook Used to guide oating nuts to the holes in the
mounting bars of a rack.
Screwdriver Used to tighten and loosen screws. A screwdriver can be
a at-head, Phillips, hex screwdriver.
Diagonal pliers Used to trim insulation tubes and cable ties.
Multimeter Used to measure the resistance and voltage and to
check connectivity.
ESD wrist strap Used to prevent ESD damage when you touch or
operate devices or components.
TaiShan Servers
Troubleshooting 3 Preparing for Troubleshooting
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 8

Tool Description
ESD gloves Used to prevent ESD damage to a board or precision
instrument when you insert, remove, or hold them.
Cable tie Used to bind cables.
Ladder Used to perform operations at heights.
PC Used to access the management network port or a
service network port over the network to capture data.
(You need to prepare a network cable.)
Serial cable Used to connect the serial port on the server. The serial
ports are usually RJ45 ports.
Thermometer and
hygrometer
Used to measure the equipment room temperature and
relative humidity.
Oscilloscope Used to measure the voltage and time sequence.
TaiShan Servers
Troubleshooting 3 Preparing for Troubleshooting
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 9

4 Collecting Information
About This Chapter
If a fault occurs on a server, collect logs for fault diagnosis.
Collect logs immediately upon fault occurrence to obtain the original data.
4.1 Collecting Basic Information
4.2 Collecting OS Logs
4.3 Collecting Hardware Logs
4.1 Collecting Basic Information
The customer needs to collect basic information listed in Table 4-1 before
submitting a service request.
Table 4-1 Server fault records
Server Fault Records
Trouble Ticket
No.
Example: 123456 Fault Report
Time
Example:
2015-10-18
20:30:00
Customer Name Full name of your
organization
Address Example: 20 Baker
Street, New York
Customer
Contact/ASP
Name
Example: John
Smith
Contact Info Phone number
and email address
Device Model For example,
TaiShan 100
server (model
5280)
SN/ESN Example:
2102310XXXXX
(For details about
how to obtain the
number, see 8.1
Obtaining a
Product SN.)
TaiShan Servers
Troubleshooting 4 Collecting Information
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 10

Server Fault Records
Hardware
Conguration
If the device conguration (CPUs, DIMMs, RAID controller
cards, or NICs) is modied, you need to provide the
modied conguration. If the conguration is not modied,
enter None.
OS and Service
Software Version
Example: SLES 11 SP1 64-bit . (Consider the fault symptom
to determine whether to collect the OS and service
software versions.)
Fault Occurrence
Time
Example: 2015-10-18 20:30:00
Fault Symptom Example: The server frequently restarts during OS
installation or the server stops responding upon power-on .
Action Before
Fault Occurrence
Example: BIOS settings conguration , memory capacity
expansion , network settings modication .
Action and
Result After
Fault Occurrence
(Optional)
Example: After the power cable is disconnected and then
reconnected, the fault persists .
After the DVD-ROM is replaced, the fault persists .
...
4.2 Collecting OS Logs
Collect OS logs after an OS fault occurs.
NO TICE
● Obtain the customer's written authorization before collecting information.
● Logs collected by SmartKit may contain sensitive customer information. If
sensitive customer information is involved, obtain the customer's written
authorization before performing any maintenance operation.
Table 4-2 describes the methods for collecting logs of dierent OSs.
NO TE
For details about the OSs supported by each server, use the Computing Product
Compatibility Checker.
Table 4-2 Methods for collecting OS logs
OS Collection Method
Linux (CentOS,
Ubuntu, SLES,
Use SmartKit. For details, see section "Collecting Server
Logs" in the FusionServer Tools SmartKit User Guide.
TaiShan Servers
Troubleshooting 4 Collecting Information
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 11

OS Collection Method
NeoKylin, and
others)
Log in to the OS CLI over SSH and copy all les in the /var/
log/ directory.
Collect les prexed with syslog (for example, syslog and
syslog.0), dmesg, and kernel in /var/log/boot.log
and /var/log/ before collecting other les.
NOTE
● Log les in the /var/log/ directory are subject to the actual
situation.
● If the OS is suspended and cannot be accessed, capture a
screenshot of the problem, restart the OS, and then collect logs.
4.3 Collecting Hardware Logs
Collect hardware logs after a hardware fault occurs.
NO TICE
● Obtain the customer's written authorization before collecting information.
● Logs collected by SmartKit may contain sensitive customer information. If
sensitive customer information is involved, obtain the customer's written
authorization before performing any maintenance operation.
● Use SmartKit to collect server hardware information in batches. For details
about the supported servers and operations, see section "Using SmartKit >
Collecting Server Logs" in the FusionServer Tools SmartKit User Guide.
● Use iBMC to collect hardware logs of a single server. For details, see 8.2 Using
iBMC to Collect Information in One-click Mode.
TaiShan Servers
Troubleshooting 4 Collecting Information
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 12

5 Diagnosing and Rectifying Faults
5.1 Fault Diagnosis Rules
5.2 Using Tools to Diagnose Faults
5.3 Handling Alarms
5.4 Using Error Codes to Locate Faults
5.5 Using Indicators to Locate Faults
5.6 Handling Faults Based on Symptoms
5.1 Fault Diagnosis Rules
NO TICE
● Obtain the customer's written authorization before performing any operation.
● Before performing any operation, ensure that service data will not be lost or
has been backed up.
Observe the following fault diagnosis rules:
● Check the external components and then the internal components.
During troubleshooting, check external devices for faults (such as a power
failure and peer device failure) rst.
● Check the network and then network elements (NEs).
According to the network topology, check whether the network environment
is normal and then check whether the NEs are normal. Determine which NE is
faulty if possible.
● Check the high-speed signal alarms and then the low-speed signal alarms.
Alarm signal streams show that high-speed signal alarms often cause low-
speed signal alarms. Therefore, clear high-speed signal alarms rst.
● Analyze alarms of high severity and then analyze alarms those of low severity.
Analyze critical or major alarms rst, and then analyze minor alarms.
TaiShan Servers
Troubleshooting 5 Diagnosing and Rectifying Faults
Issue 12 (2022-08-30) Copyright © Huawei Technologies Co., Ltd. 13