HP 5920 User manual

HP 5920 & 5900 Switch Series
Troubleshooting Guide
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained
herein is subject to change without notice. The only warranties for HP products and services are
set forth in the express warranty statements accompanying such products and services. Nothing
herein should be construed as constituting an additional warranty. HP shall not be liable for
technical or editorial errors or omissions contained herein.

i
Contents
Introduction··································································································································································· 1
General guidelines····························································································································································1
Collecting log and operating information ······················································································································1
Collecting common log messages ··························································································································2
Collecting diagnostic log messages ·······················································································································2
Collecting operating statistics ·································································································································3
Contacting technical support ···········································································································································4
Troubleshooting hardware ·········································································································································· 4
Unexpected switch reboot················································································································································4
Symptom····································································································································································4
Troubleshooting flowchart ·······································································································································5
Solution······································································································································································5
Operating power supply failure ······································································································································6
Symptom····································································································································································6
Solution······································································································································································6
Newly installed power supply failure ·····························································································································6
Symptom····································································································································································6
Solution······································································································································································6
Fan tray failure ··································································································································································7
Symptom····································································································································································7
Solution······································································································································································7
Related commands····························································································································································8
Troubleshooting ACL···················································································································································· 9
ACL application failure with an error message ·············································································································9
Symptom····································································································································································9
Solution······································································································································································9
ACL application failure without an error message········································································································9
Symptom····································································································································································9
Troubleshooting flowchart ···································································································································· 10
Solution··································································································································································· 10
Related commands························································································································································· 11
Troubleshooting IRF····················································································································································12
IRF fabric setup failure ··················································································································································· 12
Symptom································································································································································· 12
Troubleshooting flowchart ···································································································································· 13
Solution··································································································································································· 13
Related commands························································································································································· 15
Troubleshooting Ethernet link aggregation···············································································································16
Link aggregation failure ················································································································································ 16
Symptom································································································································································· 16
Troubleshooting flowchart ···································································································································· 17
Solution··································································································································································· 17
Related commands························································································································································· 18
Troubleshooting ports·················································································································································20
A 10-GE SFP+ fiber port or 40-GE QSFP+ fiber port fails to come up···································································· 20
Symptom································································································································································· 20

ii
Troubleshooting flowchart ···································································································································· 21
Solution··································································································································································· 21
A 10/100/1000Base-T GE copper port or 1/10GBase-T 10-GE copper port fails to come up ························ 23
Symptom································································································································································· 23
Troubleshooting flowchart ···································································································································· 23
Solution··································································································································································· 23
Non-HP transceiver module error message················································································································· 24
Symptom································································································································································· 24
Troubleshooting flowchart ···································································································································· 24
Solution··································································································································································· 24
Transceiver module does not support digital diagnosis····························································································· 25
Symptom································································································································································· 25
Troubleshooting flowchart ···································································································································· 26
Solution··································································································································································· 26
Error frames (for example, CRC errors) on a port ······································································································ 27
Symptom································································································································································· 27
Troubleshooting flowchart ···································································································································· 27
Solution··································································································································································· 27
Failure to receive packets·············································································································································· 29
Symptom································································································································································· 29
Troubleshooting flowchart ···································································································································· 29
Solution··································································································································································· 30
Failure to send packets·················································································································································· 31
Symptom································································································································································· 31
Troubleshooting flowchart ···································································································································· 31
Solution··································································································································································· 31
Related commands························································································································································· 32
Troubleshooting system management·······················································································································34
High CPU utilization ······················································································································································ 34
Symptom································································································································································· 34
Troubleshooting flowchart ···································································································································· 34
Solution··································································································································································· 34
High memory utilization ················································································································································ 36
Symptom································································································································································· 36
Troubleshooting flowchart ···································································································································· 36
Solution··································································································································································· 36
Related commands························································································································································· 37
Troubleshooting other problems································································································································38
Layer 2 forwarding failure ············································································································································ 38
Symptom································································································································································· 38
Troubleshooting flowchart ···································································································································· 39
Solution··································································································································································· 39
Related commands ················································································································································ 42
Layer 3 forwarding failure ············································································································································ 43
Symptom································································································································································· 43
Troubleshooting flowchart ···································································································································· 44
Solution··································································································································································· 44
Related commands ················································································································································ 45
Protocol flapping···························································································································································· 45
Symptom································································································································································· 45
Troubleshooting flowchart ···································································································································· 45
Solution··································································································································································· 46
Fan failure······································································································································································· 46

iii
Symptom································································································································································· 46
Troubleshooting flowchart ···································································································································· 47
Solution··································································································································································· 47
Related commands ················································································································································ 48

1
Introduction
This document provides information about troubleshooting common software and hardware problems
with the HP 5920 & 5900 switch series.
This document is not restricted to specific software or hardware versions.
General guidelines
IMPORTANT:
To prevent a problem from causing loss of configuration, save the configuration each time you finish
configuring a feature. For configuration recovery, regularly back up the configuration to a remote server.
When you troubleshoot the switch, follow these general guidelines:
•To help identify the cause of the problem, collect system and configuration information, including:
{Symptom, time of failure, and configuration.
{Network topology information, including the network diagram, port connections, and points of
failure.
{Log messages and diagnostic information. For more information about collecting this
information, see "Collecting log and operating information."
{Physical evidence of failure:
−Photos of the hardware.
−Status of the LEDs.
{Steps you have taken, such as reconfiguration, cable swapping, and reboot.
{Output from the commands executed during the troubleshooting process.
•To ensure safety, wear an ESD-preventive wrist strap when you replace or maintain a hardware
component.
•If hardware replacement is required, use the release notes to verify the hardware and software
compatibility.
Collecting log and operating information
IMPORTANT:
By default, the information center is enabled. If the feature is disabled, you must use the info-center enable
command to enable the feature for collecting log messages.
Table 1 shows the types of files that the system uses to store operating log and status information. You can
export these files by using FTP, TFTP, or USB.
In an IRF system, these files are stored on the master device. Multiple devices will have log files if
master/subordinate switchovers have occurred. You must collect log files from all these devices. To more
easily locate log information, use a consistent rule to categorize and name files. For example, save log
files to a separate folder for each member device, and include their slot numbers in the folder names.

2
Table 1 Log and operating information
Cate
g
or
y
File name format
Content
Common log logfile.log Command execution and operational log messages.
Diagnostic log diagfile.log
Diagnostic log messages about device operation, including the
following items:
•Parameter settings in effect when an error occurs.
•Information about a card startup error.
•Handshaking information between member devices when a
communication error occurs.
Operating
statistics file-basename.gz
Current operation statistics for feature modules, including the
following items:
•Device status.
•CPU status.
•Memory status.
•Configuration status.
•Software entries.
•Hardware entries.
Collecting common log messages
1. Save common log messages from the log buffer to a log file.
By default, the log file is saved in the logfile directory of the Flash memory on each member device.
<Sysname> logfile save
The contents in the log file buffer have been saved to the file
flash:/logfile/logfile.log
2. Identify the log file on each member device:
# Display the log file on the master device.
<Sysname> dir flash:/logfile/
Directory of flash:/logfile
0 -rw- 21863 Jul 11 2013 16:00:37 logfile.log
524288 KB total (107944 KB free)
# Display the log file on each subordinate device:
<Sysname> dir slot2#flash:/logfile/
Directory of slot2#flash:/logfile
0 -rw- 21863 Jul 11 2013 16:00:37 logfile.log
524288 KB total (107944 KB free
3. Transfer the files to the desired destination by using FTP, TFTP, or USB. (Details not shown.)
Collecting diagnostic log messages
1. Save diagnostic log messages from the diagnostic log file buffer to a diagnostic log file.

3
By default, the diagnostic log file is saved in the diagfile directory of the Flash memory on each
member device.
<Sysname> diagnostic-logfile save
The contents in the diagnostic log file buffer have been saved to the file
flash:/diagfile/diagfile.log
2. Identify the diagnostic log file on each member device:
# Display the diagnostic log file on the master device.
<Sysname> dir flash:/diagfile/
Directory of flash:/diagfile
0 -rw- 161321 Jul 11 2013 16:16:00 diagfile.log
524288 KB total (107944 KB free)
# Display the diagnostic log file on each subordinate device:
<Sysname> dir slot2#flash:/diagfile/
<Sysname> dir slot2#flash:/diagfile/
Directory of slot2#flash:/diagfile
0 -rw- 161321 Jul 11 2013 16:16:00 diagfile.log
524288 KB total (107944 KB free)
3. Transfer the files to the desired destination by using FTP, TFTP, or USB. (Details not shown.)
Collecting operating statistics
You can collect operating statistics by saving the statistics to a file or displaying the statistics on the
screen.
When you collect operating statistics, follow these guidelines:
•Log in to the device through a network port or management port instead of the console port, if
possible. Network and management ports are faster than the console port.
•Do not execute commands during operating statistics collection.
•HP recommends saving operating statistics to a file to retain the information.
NOTE:
The amount of time to collect statistics increases along with the number of IRF member devices.
To collect operating statistics:
1. Disable pausing between screens of output if you want to display operating statistics on the screen.
Skip this step if you are saving statistics to a file.
<Sysname> screen-length disable
2. Collect operating statistics for multiple feature modules.
<Sysname> display diagnostic-information
Save or display diagnostic information (Y=save, N=display)? [Y/N] :
3. At the prompt, choose to save or display operating statistics:
# To save operating statistics, enter yat the prompt and then specify the destination file path.
Save or display diagnostic information (Y=save, N=display)? [Y/N] :y

4
Please input the file name(*.tar.gz)[flash:/diag.tar.gz] :flash:/diag.tar.gz
Diagnostic information is outputting to flash:/diag.tar.gz.
Please wait...
Save successfully.
<Sysname> dir flash:/
Directory of flash:
…
6 -rw- 898180 Jun 26 2013 09:23:51 diag.tar.gz
524288 KB total (107944 KB free
# To display operating statistics on the monitor terminal, enter nat the prompt.
Save or display diagnostic information (Y=save, N=display)? [Y/N] :N
===============================================
===============display clock===============
07:46:55 UTC Sat 10/10/2014
=================================================
===============display version===============
……
Contacting technical support
If you cannot resolve a problem after using the troubleshooting procedures in this document, contact HP
Support. When you contact an authorized HP support representative, be prepared to provide the
following information:
•Information described in "General guidelines."
•Product serial numbers.
•Technical support registration numbers (if applicable).
This information will help the support engineer assist you as quickly as possible.
The following is the contact of HP Support for networking products:
http://www.hp.com/networking/support
Troubleshooting hardware
This section provides troubleshooting information for common hardware problems.
NOTE:
This section describes how to troubleshoot unexpected switch reboot, power supply failure, and fan tray
failure. To troubleshoot ports, see "Troubleshooting ports."
Unexpected switch reboot
Symptom

5
The switch reboots unexpectedly when it is operating.
Troubleshooting flowchart
Figure 1 Troubleshooting unexpected switch reboot
Solution
To resolve the problem:
1. Verify that you can access the CLI after the switch reboots.
{If you can access the CLI, execute the display diagnostic-information command to collect log
messages.
{If you cannot access the CLI, go to step 2.
2. Verify that the system software image on the switch is correct.
Log in to the switch through the console port and restart the switch. If the system reports that a CRC
error has occurred or that no system software image is available, reload the system software
image. The system software image is automatically set to the current system software image during
the Boot ROM image running process.
3. If the problem persists, contact HP Support.
The switch reboots unexpectedly
Can the CLI be
accessed?
Collect log messages Reload the system
software image
Yes
No
Is the fault
rectified?
End
Contact HP Support
Yes No
Yes
No
Is the system software
image correct?

6
Operating power supply failure
Symptom
A trap or log is generated indicating that an operating power supply is faulty.
Solution
To resolve the problem:
1. Execute the display power command to display power supply information.
<Sysname> display power
Slot 1
Input Power : 203(W)
Power 1
State : Normal
Type : AC
Power 2
State : Absent
If the power supply is in Absent state, go to step 2. If the power supply is in Fault state, go to step 3.
2. Remove and reinstall the power supply to make sure the power supply is installed securely. Then,
execute the display power command to verify that the power supply has changed to Normal state.
If the power supply remains in Absent state, replace the power supply.
3. When the power supply is in Fault state, do the following:
a. Verify that the power supply is connected to the power source securely. If it has been
disconnected from the power source (The switch is being powered up with another power
supply.), connect the power source to it.
b. Determine whether the power supply is in high temperature. If dust accumulation on the power
supply causes the high temperature, remove the dust. Then remove and reinstall the power
supply. Execute the display power command to verify that the power supply has changed to
Normal state. If the power supply remains in Fault state, go to step c.
c. Install the power supply into an empty power supply slot. Then execute the display power
command to verify that the power supply has changed to Normal state in the new slot. If the
power supply remains in Fault state, replace the power supply.
4. If the problem persists, contact HP Support.
Newly installed power supply failure
Symptom
A trap or log is generated indicating that a newly installed power supply is faulty.
Solution
To resolve the problem:

7
1. Execute the display power command to display power supply information.
<Sysname> display power
Slot 1
Input Power : 203(W)
Power 1
State : Normal
Type : AC
Power 2
State : Absent
If the power supply is in Absent state, go to step 2. If the power supply is in Fault state, go to step 3.
2. When the power supply is in Absent state, do the following:
a. Remove and reinstall the power supply to make sure the power supply is installed securely.
Then execute the display power command to verify that the power supply has changed to
Normal state. If the power supply remains in Absent state, go to step b.
b. Remove and install the power supply into an empty power supply slot. Then execute the display
power command to verify that the power supply has changed to Normal state in the new slot.
If the power supply remains in Absent state, go to step 4.
3. Remove and install the power supply into an idle power supply slot. Then execute the display
power command to verify that the power supply has changed to Normal state in the new slot. If the
power supply remains in Fault state, go to step 4.
4. If the problem persists, contact HP Support.
Fan tray failure
Symptom
A trap or log is generated indicating that an operating fan tray or a newly installed fan tray is faulty.
Solution
To resolve the problem:
1. Execute the display fan command to display the operating states of the fan tray.
<Sysname> display fan
Slot 1
FAN 1
State : FanDirectionFault
Wind Direction :Port-to-Power
Prefer Wind Direction :Power-to-Port
FAN 2
State : FanDirectionFault
Wind Direction :Port-to-Power
Prefer Wind Direction :Power-to-Port
{If the Wind Direction value is different from the Prefer Wind Direction value, replace the fan tray
with one that has an airflow direction as required, or execute the fan prefer-direction command
to configure the preferred airflow direction as required.

8
{If the fan tray is in Absent state, go to step 2.
{If the fan tray is in Fault state, go to step 3.
2. Remove and reinstall the fan tray to make sure the fan tray is installed securely. Then execute the
display fan command to verify that the fan tray has changed to Normal state. If the fan tray
remains in Absent state, replace the fan tray.
3. Execute the display environment command to display temperature information. If the temperature
continues to rise, put your hand at the air outlet to feel if air is being expelled out of the air outlet.
If no air is being expelled out of the air outlet, remove and reinstall the fan tray. Then execute the
display fan command to verify that the fan tray has changed to Normal state. If the fan tray
remains in Fault state, replace the fan tray.
You must make sure the switch operating temperature is below 60°C (140°F) while you replace
the fan tray. If a new fan tray is not readily available, power off the switch to avoid damage
caused by high temperature.
4. If the problem persists, contact HP Support.
Related commands
This section lists the commands that you might use for troubleshooting the hardware.
Command Descri
p
tion
dir Displays information about files and directories.
display boot-loader Displays current configuration files and system software
images.
display environment Displays temperature information.
display fan Displays the operating states of the fan tray.
display logbuffer Displays the state of the log buffer and the log information in
the log buffer.
display power Displays power supply information.

9
Troubleshooting ACL
This section provides troubleshooting information for common problems with ACLs.
ACL application failure with an error message
Symptom
The system fails to apply a packet filter or an ACL-based QoS policy to the hardware. It also displays the
"Reason: Not enough hardware resource" message.
Solution
To resolve the problem:
1. Execute the display qos-acl resource command, and then check the Remaining field for ACL
resources insufficiency.
If this field displays 0, the ACL hardware resources are exhausted.
2. To free hardware resources, delete unnecessary ACLs.
3. If the problem persists, contact HP Support.
ACL application failure without an error message
Symptom
The system applies a packet filter or an ACL-based QoS policy to the hardware. However, the ACL does
not take effect.

10
Troubleshooting flowchart
Figure 2 Troubleshooting ACL application failure
Solution
Choose a solution depending on the module that uses the ACL.
ACL used in a QoS policy
To resolve the problem when the ACL is used in a QoS policy:
1. Verify that the QoS policy is configured correctly:
a. Use one of the following commands to check the QoS policy for configuration errors,
depending on the policy application destination:
Destination Command
Interface display qos policy interface
VLAN display qos vlan-policy
Global display qos policy global
Control plane display qos policy control-plane slot
slot-number
b. If the QoS policy does not contain a class-behavior association, associate the traffic behavior
with the traffic class.
c. If the QoS policy contains a class-behavior association, execute the display traffic classifier
user-defined command and the display traffic behavior user-defined command to check for
traffic class and behavior configuration errors, respectively.
−If they are configured incorrectly, reconfigure them.
QoS policy/packet filter
configured correctly?
ACL
configured correctly?
Reconfigure QoS
policy/packet filter Resolved?
Failure to apply an ACL
No
No
Yes
Contact the support
Resolved?
End
Reconfigure ACL Yes
No
Yes
No
Yes

11
−If they are configured correctly, go to step 2.
2. Verify that the ACL is configured correctly.
Execute the display acl command to check whether the ACL is configured correctly.
{If the ACL is configured incorrectly, reconfigure it.
{If the ACL is configured correctly, go to step 3.
3. If the problem persists, contact HP Support.
ACL used in a packet filter
To resolve the problem when the ACL is used in a packet filter:
1. Verify that the packet filter is configured correctly.
Execute the display packet-filter command to check whether the packet filter is configured
correctly.
{If there are any configuration errors, reconfigure the packet filter.
{If there is no configuration error, go to step 2.
2. Verify that the ACL is configured correctly.
Execute the display acl command to check whether the ACL is configured correctly.
{If the ACL is configured incorrectly, reconfigure it.
{If the ACL is configured correctly, go to step 3.
3. If the problem persists, contact HP Support.
Related commands
This section lists the commands that you might use for troubleshooting ACLs.
Command Descri
p
tion
display acl Displays configuration and match statistics for ACLs.
display diagnostic-information Displays operating statistics for multiple feature modules in
the system.
display packet-filter Displays whether an ACL has been successfully applied to an
interface for packet filtering.
display qos-acl resource Displays QoS and ACL resource usage.
display qos policy control-plane Displays information about the QoS policies applied to the
specified control plane.
display qos policy global Displays information about global QoS policies.
display qos policy interface Displays information about the QoS policies applied to an
interface or to all interfaces.
display qos policy user-defined Displays user-defined QoS policies.
display qos vlan-policy Displays information about QoS policies applied to VLANs.
display traffic classifier user-defined Displays traffic class configuration.
display traffic behavior user-defined Displays traffic behavior configuration.

12
Troubleshooting IRF
This section provides troubleshooting information for common problems with IRF.
IRF fabric setup failure
Symptom
An IRF fabric cannot be set up.

13
Troubleshooting flowchart
Figure 3 Troubleshooting IRF fabric setup failure
Solution
To resolve the problem:
1. Verify that the number of member devices does not exceed the upper limit.
If you are adding a new member device to an existing IRF fabric or merging IRF fabrics, use the
display irf command to identify the number of member devices in the IRF fabrics. If the total number
of member devices exceeds the upper limit, the IRF setup will fail.
The upper limit varies by software version:
{Release 22xx—Four.

14
{Release 2307 and later versions—Nine.
2. Verify that the member ID of each member device is unique:
a. Execute the display irf command to view the member ID of each member device.
b. Assign a unique member ID to each member.
−If you are adding a new member device to an existing IRF fabric, assign the member device
a member ID that is not being used in the IRF fabric.
−If you are merging IRF fabrics, make sure each member device in these IRF fabrics has a
unique member ID.
3. Verify that the IRF port bindings and physical IRF link connections are correct:
IMPORTANT:
W
hen you connect two neighboring IRF members, you must connect the physical interfaces of IRF-port
1 on one member to the physical interfaces of IRF-port 2 on the other.
a. Execute the display irf configuration command on each member device, and check the
IRF-Port1 and IRF-Port2 fields for IRF port bindings.
b. Verify that all or none of the ports in a port group are used as IRF physical interfaces.
The following are port grouping rules:
−The 10-GE ports or SFP+ ports are grouped by port number in order, starting from 1. Each
port group contains four ports.
−The 10-GE breakout interfaces split from a QSFP+ port belong to the same port group.
c. Verify that the physical IRF connections are consistent with the IRF port bindings.
d. If there are binding errors or connection inconsistencies, reconfigure the IRF port bindings or
reconnect the IRF physical interfaces.
4. Verify that the IRF links are 10 Gbps or 40 Gbps:
a. Verify that the transceiver modules or DAC cables for IRF connection are labeled with 10Gbps,
SFP+, 40Gbps, or QSFP+.
b. Replace the transceiver module or DAC cable if it is not for 10 Gbps or 40 Gbps connection.
5. (Optional.) Verify that transceiver modules at the two ends of an IRF link are the same type.
If the transceiver modules are not the same type, replace them to be the same type.
Skip this step if you are using SFP+ or QSFP+ DAC cables to connect IRF member devices.
6. Verify that all IRF links are up:
a. Execute the display irf topology command, and then check the Link field.
−If the Link field for each IRF port does not display DOWN, go to step 7.
−If the Link field for an IRF port displays DOWN, go to step b.
b. Execute the display irf link command, and then check the Status field of the physical interfaces
bound to the IRF port.
−If the field displays DOWN or ADM (administratively down) for all the physical interfaces,
remove the link failures or use the undo shutdown command to bring up the administratively
down physical interfaces.
−If the field displays UP for at least one of the physical interfaces, go to step c.
c. Save the configuration, and then execute the irf-port-configuration active command in system
view to activate the IRF port configuration.

15
IMPORTANT:
A
ctivatin
g
IRF port confi
g
uration requires a reboot. To prevent confi
g
uration loss, you must save
the configuration before you execute the irf-port-configuration active command.
7. Verify that all member devices use the same software version:
a. Execute the display version command to identify the software version of each member device.
b. Upgrade the software of all member devices to the same version.
NOTE:
Typically, the irf auto-update enable command can automatically synchronize a member device with
the software version of the master device. However, the synchronization might fail when the gap
between the software versions is too large. For more information, see the release notes.
8. If the problem persists, contact HP Support.
Related commands
This section lists the commands that you might use for troubleshooting IRF.
Command Descri
p
tion
display interface
Displays interface information.
Use this command to verify that each IRF port has a minimum of one
physical interface in up state.
display irf Displays IRF fabric information, including the member ID, role, priority,
bridge MAC address, and description of each IRF member.
display irf configuration
Displays basic IRF settings, including the current member ID, new
member ID, and physical interfaces bound to the IRF ports on each IRF
member device. The new member IDs take effect at reboot.
display irf topology Displays the IRF fabric topology, including the member IDs, IRF port state,
and adjacencies of IRF ports.
display version Displays system version information.
irf-port-configuration active Activates IRF configuration on IRF ports.
undo shutdown Brings up an Ethernet interface or subinterface.

16
Troubleshooting Ethernet link aggregation
This section provides troubleshooting information for common problems with Ethernet link aggregation.
Link aggregation failure
Symptom
Some member ports fail to be placed in Selected state, and link aggregation does not operate correctly.