Cisco Catalyst 4000 series Instruction Manual

Hardware Troubleshoot for Catalyst
4000/4912/2980G/2948G Series Switches
Document ID: 18935
Interactive: This document offers customized analysis of your Cisco
device.
Contents
Introduction
Prerequisites
Requirements
Components Used
Conventions
Preparation for Troubleshooting Hardware on Catalyst Switches
Online Troubleshooting Tools
Catalyst 4000 Family Troubleshooting Procedures
General Problem Solving Model
General Problem Solving Flow Chart
Common Problems
Symptom Description
System/Supervisor/Module Problems and Steps to Resolve Them
Supervisor Crashes and Steps to Resolve Them
Misleading Problems
show Command Descriptions
Related Information
Introduction
This document provides troubleshooting procedures on how to diagnose hardware problems on Catalyst 4000
family switches. The Catalyst 4000 family includes the 4003 and 4006 modular chassis and the 2948G,
2980G, and 4912G fixed models. The naming conventions for the Catalyst 4000 and Catalyst 2900 can be
very confusing. Refer to Understanding Catalyst 2900 and Catalyst 4000 Naming Conventions for more
information on how to help clarify these issues.
The goal is to help Cisco customers identify and fix some basic hardware issues, or to perform more extensive
troubleshooting before you contact Cisco Technical Support. An orderly troubleshooting process with the
collection of specific diagnostics ensures that information necessary to the resolution of the problem is not
lost. If you refine the scope of the problem, this saves valuable time in the search for a solution.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
Catalyst 4000 Command Reference• How LAN Switches Work•

Components Used
This document is not restricted to specific software and hardware versions.
Conventions
Refer to Cisco Technical Tips Conventions for more information on document conventions.
Preparation for Troubleshooting Hardware on Catalyst
Switches
Many hardware problems encountered during field installations or during normal operation can be prevented
by a thorough product overview ahead of time. For those customers not already familiar with general system
and power requirements, proper installation procedure, switch management and software considerations for
these switches, Cisco recommends that you read documents in Cisco Catalyst 4000 Series Switches
Troubleshooting TechNotes.
This document covers this important information:
Which supervisor is supported in which chassis?• How do I back up my configuration?• Which software version is General Deployment (GD) for the Catalyst 4000 Family?•
This document assumes familiarity with the Catalyst 4000 Command Reference. You should also have a prior
understanding of switching fundamentals, or have read How LAN Switches Work. Additional online
documentation is referenced throughout this document in order to assist in troubleshooting.
Online Troubleshooting Tools
Cisco has a variety of troubleshooting tools and resources in order to help you interpret switch output,
determine hardware software compatibility, track bugs, and search field notices. These tools and resources are
referenced throughout this document:
Output Interpreter (registered customers only) Paste in the output of a command and get the
interpretation with relevant errors, warnings, and status information.
•
Error Message Decoder (registered customers only) Paste in system error messages and discover
their meaning.
•
Bug Toolkit (registered customers only) Search for bugs.• Troubleshooting AssistantThis provides step−by−step instructions to many common network issues.•
Catalyst 4000 Family Troubleshooting Procedures
This section discusses troubleshooting procedures, symptoms, show commands, and diagnostics for the
Catalyst 4000 family. This section assumes you have read the companion guide to this document, as described
in the Introduction of this document, and that you understand your switch and its capabilities.
Note: If the switch is connected to the network, do not reset or reseat modules as a first troubleshooting
step! In addition to the downtime that users experience, the internal buffer, which logs system messages are
erased and potentially useful information in regards to hardware or software errors are lost. If the switch is
offline, you have more freedom to monitor LED status, pull cables, reseat modules, or reset the switch as
necessary. Troubleshooting LED status is discussed in more detail later in this document.

Hidden Commands
Some commands presented in this document are known as hidden, which means that they cannot be parsed
with a "?", and you cannot Tab in order to complete. When a hidden command is suggested in this document,
simply gather the output and send it to the TAC engineer, if you open a case. It is possible that this output is
useful in solving your case. These commands are undocumented, and therefore the TAC engineer is not
required to explain the output to the customer.
If you want to troubleshoot any problem, this requires a method or set of procedures which, if followed
correctly, produces a solution. Begin by understanding general problem solving for LAN networks.
General Problem Solving Model
If you want to troubleshoot any problem, this requires a method or set of procedures which, if followed
correctly, produces a solution. Begin by understanding general problem solving for LAN networks. Hardware
failures in LAN networks are characterized by certain symptoms. These symptoms can be general such as the
inability to Telnet between switches, more specific such as link flapping, or perhaps the switch is resetting
itself. Each symptom can be traced to one or more causes if you use specific troubleshooting techniques. A
systematic approach works best. Define the specific symptoms, identify all potential problems that could be
causing the symptoms, and then eliminate each potential problem, from most likely to least likely, until the
symptoms disappear.
General Problem Solving Flow Chart
This diagram outlines the steps that detail the problem−solving process:
Complete these steps:
Define the problem.
It is important to first identify the problem being experienced. This allows you to identify what kinds
of causes can result in these symptoms. In order to help determine the problem, ask yourself these
questions:
What is the primary symptom?♦ Is the problem specific to this switch or does it affect other switches on the network as well?♦ Is this a problem with one or more ports on a specific module? What type of ports: 10/100,
Multimode Fiber (MMF), Singlemode Fiber (SMF), GigabitEthernet, and so forth?
♦
1.

What device is connected to the switch ports that experiences the problem?♦ When did this problem first occur and has it occurred more than once?♦ What happened at the time the problem was first noticed? Is there anything unique about
traffic conditions at that time of day? For example, was this a peak time for traffic?
♦
Did you run any particular commands at the time or make any configuration changes?♦
Gather the facts.
Gather diagnostics and show commands output from the switch to isolate the scope of the problem. If
physical access to the equipment is possible, locate and list any modules with red or yellow LEDs,
disconnected cables, or loose connections.
2.
Consider the possible causes.
Consider possible problems based on the information you gathered. With certain data, you are able,
for example, to eliminate hardware as a problem, so that you can focus on software problems. At
every opportunity, try to narrow the number of potential problems so that you can create an effective
plan of action.
3.
Create and implement an action plan.
Create an action plan based on the potential problems. Focus on only one potential problem at a time.
If you alter more than one variable simultaneously, you can solve the problem, but the identification
of the specific change that eliminated the symptom becomes far more difficult and does not help you
solve the same problem if it occurs in the future.
4.
Observe the results.
Be sure to gather and analyze the results each time a variable is changed to determine if the problem
has been fixed.
5.
Repeat the process.
Repeat testing for possible causes until the problem is resolved.
6.
Common Problems
As described in the Problem Solving Model, the first step in resolving a problem is to identify the symptom.
Refer to Catalyst Troubleshooting Tips for more information on some common problems associated with all
Catalyst switches that can be resolved.
Most hardware problems with LAN networks fall into these categories and each category has various
symptoms related to it:
Connectivity Problems• System/Supervisor/Module Problems• Supervisor Crashes•
Connectivity Problems
These problems can occur when communication with the supervisor, module, or hosts connected to the
module is intermittent or has been lost.
System/Supervisor/Module Problems
These problems can occur when system status LEDs indicate a problem, the supervisor or modules are not
recognized or show faulty, or when users experience poor performance.

Supervisor Crashes
These problems can occur when the switch has reset, continually resets, or is down completely.
Symptom Description
This section discusses symptoms, troubleshooting procedures, and commands for Catalyst 4000 family
switches. This section assumes you are able to identify your switch chassis, supervisor engine, modules, and
feature cards, and that you understand the system specifications, cabling, power, and software requirements as
described for Cisco Catalyst 4500 Series Switches Install and Upgrade Guides.
If you have not determined what your primary symptom is, see the General Problem Solving Model section of
this document and apply the steps to your problem.
Connectivity Problems and Steps to Resolve Them
This section covers common connectivity issues that the customer can encounter with the Catalyst 4000.
These commands are supported by the Output Interpreter tool for CatOS and can be used to assist in
troubleshooting switch port problems:
show version• show module• show system• show port• show mac• show counters• show cdp neighbors detail•
If you have the output of the supported commands from your Cisco device, you can use the Output Interpreter
(registered customers only) to display potential issues and fixes. In order to use Output Interpreter (registered
customers only) , you must be a registered user, be logged in, and have JavaScript enabled.
Can not console/Telnet into the supervisor
Both of these problems are covered in the Catalyst Troubleshooting Tips document that is mentioned earlier.
Not able to console
Verify that the power switch is in the ON position (|) and the system OK LED is ON.1. Connect the cable directly to the console port and not through a patch panel.2. Verify that the correct cabling and hardware is used to connect to your particular supervisor
engine. Refer to the Connecting a Terminal to the Console Port on Catalyst Switches
document for more information.
3.
•
Not able to Telnet
Complete the steps in thedetailed procedure described in Catalyst Troubleshooting Tips. If it
is determined that the sc0 management interface is not configured or not configured correctly,
refer to Configuring an IP Address on Catalyst Switches for more information.
1.
Attempt to Telnet from a PC directly connected to the switch in the same VLAN as the sc0
interface in order to eliminate any routing issues.
2.
Gain console access to the switch and make sure the supervisor is not in boot> or rommon>.
If the switch is in one of these modes, you need to complete the steps in the recovery
procedures. Refer to Recovering Catalyst 4000 and Catalyst 5000 Switches from Corrupted or
3.
•

Missing Software, or an Upgrade Failure, or from ROMMON Mode for more information on
recovery.
Receiving the "Failed to allocate session block" error message
If you receive the Failed to allocate session block error message while you access the switch
on the Telnet, the problem occurs because the switch cannot allocate the required memory for the Telnet
application. The available free memory is low because of some process that uses more memory or because of
a memory leakage in the switch.
In order to avoid the error, issue the show proc mem command and verify the process that uses more memory
in the switch. In order to resolve the problem, either add more memory to the system or disable some features
in order to free some of the existing memory.
If there is memory leak in the switch, reset the switch in order to release all the process in the memory. If the
error message still appears even after you reboot, upgrade the software version of the switch.
Can not connect to a remote host, router, or another switch
Complete these steps:
Verify that the port LED status is green. If the link LED is solid orange, it has been disabled by the
software. If it is blinking orange after supervisor bootup and module initialization, this is a hardware
failure. If there is no link LED, check and swap the cables. Verify operation of the end device and
NIC.
Refer to Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues for more information
on NIC troubleshooting.
1.
What type of media is involved? Fiber? Gigabit Interface Converter (GBIC)? Gigabit Ethernet?
10/100 BaseTX? If this a physical layer issue, refer to the Physical Layer Troubleshooting section of
Troubleshooting Switch Port Problems for more information.
2.
Issue the show port <mod/port> command in order to verify that the status is connected, which
means that the port is operational. If any other status is displayed, see the Port status shows not
connected, faulty, disabled, inactive, or errdisable section for troubleshooting steps.
If the end device is a Cisco router or switch, and Cisco Discovery Protocol (CDP) is enabled, issue the
show cdp neighbor detail command in order to identify the device, remote interface type, and remote
IP address.
Note: A status of connected does not mean that the ports are free of errors. If there are errors on the
ports, proceed to the Seeing errors on the ports section of this document.
3.
Swap the cables. Move the cable to a different port. Eliminate patch panels. Patch panels are a
common source of connectivity failures, so attempt to connect directly to the end device. Verify the
operation of the end device.
4.
Capture the output of the show config , show module , and show test 0 commands.
Issue the show module command in order to verify that the status is ok for that module and
not disabled or faulty.
If the status is disabled, issue the set module enable <mod> command.◊ If the status is faulty, establish a console connection to capture bootup Power On Self
Test (POST) diagnostics and any system error messages. Issue the reset <mod>
command in order to reset the module. Issue the show test 0 command in order to
determine if this module passed all of it's diagnostic tests on bootup.
◊
a.
5.

Remove the module and inspect for bent pins. Reseat the module, firmly press down
the ejector levers, and tighten the captive installation screws. If the output of the
show module command status is still faulty, try the module in another slot. Slot 2
accepts line cards or a supervisor engine. If necessary, power off/on the switch. If the
status is still faulty, the module has failed.
◊
Issue the show test 0 command in order to verify that the port has passed its last diagnostic
test on bootup. If F is indicted for that port, proceed as in step a.
b.
Verify whether this device is on the same or a different VLAN. Remember that this is a Layer 2 (L2)
device and a router is required to route between VLANs.
6.
If you connect to another switch, ask yourself these questions:
What type of port is this? A trunk port?♦ If it is a trunk port, what trunk encapsulations does it support?♦ Is the port capable of EtherChannel?♦
Issue the show port capabilities command for a quick look at port capabilities. Refer to LAN
Technical Tips for more information on how to troubleshoot issues with trunking or EtherChannel.
7.
Port status shows not connected, faulty, disabled, inactive, or errdisable
Possible Port Status
Status Description and Work Around
connected
Port is operational and connected to end device. A
status of connected does not mean the ports are
error−free. If there are errors on the ports, proceed
to the Seeing errors on the ports section of this
document.
notconnect Nothing is connected to the port. Check or swap
cables. Verify the operation of the end device.
faulty
Possible hardware failure. Issue the show test
command in order to verify. If Fdisplays for a
port, proceed as in step 5 of the Can not connect to
a remote host on the switch section of this
document.
disabled
Manually disabled. Issue the set port enable
<mod/port> command in order to enable the port.
If the port status does not change to enable, issue
the show module command in order to determine
if the module is disabled.
inactive Port belongs to a VLAN that does not exist. Issue
the set vlan <vlan> command in order to add a
VLAN.
errdisable Port had been shut down due to errors. Refer to the
Recovering From errDisable Port State on the
CatOS Platforms document for more information.
Seeing errors on the ports
Complaints of poor performance by users can sometimes translate to errors on switch ports. Output from the
port error counters command help you troubleshoot connectivity problems.

Verify port status and troubleshoot accordingly. Refer to the Port status shows not connected, faulty,
disabled, inactive, or errdisable section of this document.
1.
Capture the output of the show port <mod/port> , show mac <mod/port> , and show counters
<mod/port> commands.
These are common causes for data link errors on ports:
speed/duplex misconfiguration♦ network congestion♦ NICs or drivers
Refer to Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues for more
information.
♦
cabling♦ bad port♦
The show port <mod/port> command can show Late−Coll, Align−Err, FCS−Err, Xmit−Err, and
Rcv−Err errors. Refer to the the Show Port for CatOS and Show Interfaces for Cisco IOS section of
Troubleshooting Switch Port Problems for more informaiton on these errors and possible causes.
The show mac <mod/port> command shows the number of unicast, multicast, and broadcast frames
transmitted. Issue this command in order to verify if frames are received and transmitted.
In−Discards show frames that do not need to be switched. This is normal if the port was connected to
a hub and two devices exchanged data. Lrn−Discards indicate that Content Addressable Memory
(CAM) entries are being discarded. In−Lost counter displays the sum of all error packets received on
the port. The Out−Lost counter indicates egress port buffer overflows. Refer to the Show Mac for
CatOS and Show Interfaces Counters for Cisco IOS section of Troubleshooting Switch Port Problems
for more information on these errors and possible causes.
The show counters <mod/port> command is useful in particular for troubleshooting port problems.
For example, this counter results if you issue the command:
5 badTxCRC = 0
If badTxCRC were incrementing, this can be bad hardware corrupting packets. Capture the output of
the show counters <mod/port> command and open a case with the Cisco Technical Support.
2.
Issue the clear counters command in order to reset the output of the show port <mod/port>,show
mac <mod/port>, and show counters <mod/port> commands. View the command outputs several
times in order to see if errors are incrementing.
If you have not been able to track down any reason for intermittent connectivity loss on the switch in
the previous steps mentioned, capture the output of the show nvramenv 1 command, as well as the
other commands in the previous steps, and open a case with the Cisco Technical Support.
3.
Refer to these documents for more information on how to troubleshoot the other causes of port errors:
Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues♦ Configuring and Troubleshooting Ethernet 10/100Mb Half/Full Duplex Auto−Negotiation♦
4.
Experiencing poor performance
Poor performance is often perceived to be a hardware problem, when in fact it can be attributed most often to
connectivity problems. See the Seeing errors on the ports section for troubleshooting steps.

Getting continuous %PAGP−5 left/joined bridge messages
Complete these steps:
Capture the show port <mod/port>,show mac <mod/port>, and show spantree summary
command output.
System messages similar to these messages are informational, although if the errors continue to
repeat, the link can be flapping.
2002 Jan 19 14:59:05 %PAGP−5−PORTFROMSTP:Port 2/11 left bridge port 2/11
2002 Jan 19 14:59:23 %PAGP−5−PORTTOSTP:Port 2/11 joined bridge port 2/11
1.
If these messages occur repeatedly on certain ports, refer to these document for possible causes:
Common CatOS Error Messages on Catalyst 4000 Series Switches♦ Common CatOS Error Messages on Catalyst 5000/5500 Series Switches♦ Common CatOS Error Messages on Catalyst 6000/6500 Series Switches♦
2.
If you also see errors on the port in the show port <mod/port> and show mac<mod/port> command
output, see the Seeing errors on the ports section for troubleshooting steps.
3.
Issue the show spantree summary command in order to verify how many ports are in each VLAN, if
any ports on the switch are blocking, and which VLANs are being blocked. Since Spanning−Tree
Protocol (STP) loops can cause link flaps or actually bring down a switch or network, with the
appearance of a hardware failure, this is vital information to capture, whether troubleshooting
hardware or software. Refer to LAN Technical Tips for more information on how to troubleshoot
STP.
4.
Can not autonegotiate or speed/duplex mismatch
Complete these steps:
Make sure you have speed and duplex configured identically on both sides of the link. Catalyst 4000
switchports are set to auto by default. When both sides of a 100 BaseTX link autonegotiate correctly,
the show port <mod/port> command output is as follows:
Duplex Speed
−−−−−−− −−−−−−−
a−full a−100
Hardcode both sides. Remember when hardcoding the port, the port speed must be set first and then
the duplex setting must be set. Issue the show port <mod/port> command. The switch output is as
follows:
Duplex Speed
−−−−−−− −−−−−−−
full 100
Note: Even though the switch has been hard coded, the connecting device must still be hardcoded to
eliminate problems.
1.
If there is an autonegotiation problem caused by a speed/duplex mismatch or NIC incompatibility,
errors show up on the ports. Refer to these documents for more information:
Configuring and Troubleshooting Ethernet 10/100Mb Half/Full Duplex Auto−Negotiation♦ Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues♦
2.

System/Supervisor/Module Problems and Steps to Resolve Them
System, supervisor, and module problems occur when either system status LEDs indicate a problem, the
supervisor or modules are not recognized or show faulty, or when users are experiencing poor performance.
The following commands are supported by the Output Interpreter and can be used to assist in troubleshooting
system, supervisor, and module problems: show version, show module, or show system.
If you have the output of the supported commands from your Cisco device, you can use the Output Interpreter
(registered customers only) in order to display potential issues and fixes. In order to use Output Interpreter
(registered customers only) , you must be a registered user, be logged in, and have JavaScript enabled.
Having problems upgrading software
Complete these steps:
Most customer problems that have to do with software upgrades are the result of not understanding
the copy tftp procedure, the boot process, or the Flash system for the supervisor.
Refer to Working with System Software Images for more information, specifically, on the copy tftp
procedure for your supervisor.
Refer to Using the Flash File System for more information on the Flash file system for your
supervisor.
Refer to Recovering Catalyst Switches Running CatOS from Booting Failures for more information
on rommon recovery information.
Capture the show version,show flash, or dir bootflash command output, which depends on the type
of supervisor you have. Verify that you have enough DRAM and Flash for the image to which you
attempt to upgrade, and then perform the copy tftp procedure.
1.
Set the boot environment variable and the config−register. Refer to Modifying the Switch Boot
Configuration for more information on these settings.
Cat4000−c> (enable) set boot ?
auto−config Set auto config file
config−register Set configuration register
sync Set sync parameters
system Set BOOT environment variable
Cisco recommends that you set the boot environment variable and config−register in this way:
Verify the image that you want to boot, currently installed in Flash. Issue the dir bootflash:
command.
Cat4000−c> (enable) dir bootflash:
−#− −length− −−−−−date/time−−−−−− name
1 4106492 Aug 17 2001 16:22:52 cat4000.6−3−1.bin
2 3554592 Nov 28 2001 10:38:33 cat4000.5−5−11.bin
3 4199168 Dec 07 2001 10:30:01 cat4000−k9.6−3−3.bin
4 3651336 DEC 11 2001 12:26:20 cat4000.5−5−8.bin
216540 bytes available (15512100 bytes used)
a.
Set the boot environment variable for the image in Flash from which you want to boot.
Cat4000−c> (enable) set boot system flash bootflash:cat4000.6−3−1.bin
BOOT variable = bootflash:cat4000.6−3−1.bin,1;
b.
2.

Set the config−register to boot from Flash.
Cat4000−c> (enable) set boot config−register 0x2102
Configuration register is 0x2102
ignore−config: disabled
auto−config: non−recurring
console baud: 9600
boot: image specified by the boot system commands
c.
If you end in rommon or boot mode during the upgrade, refer to Recovering Catalyst Switches
Running CatOS from Booting Failures for more information.
3.
Use the Bug Toolkit to track down bugs, or refer to Release Notes for Catalyst 4000 Family Software
Release 5.x for Caveats.
4.
Supervisor is not online or is stuck in boot or rommon
The most common causes for a Catalyst 4000 family supervisor not to be recognized is when it is stuck in
boot or rommon mode due to a missing or corrupt image. In these modes, you are not able to Telnet to the
supervisor and must have a console session open.
If the supervisor is stuck in either boot or rommon mode, complete the troubleshooting steps in
Recovering Catalyst Switches Running CatOS from Booting Failures.
1.
If the supervisor is not in either boot or rommon mode but is still not online, complete the
troubleshooting steps for the Supervisor Engine in the System component LEDs are orange/red
section of this document.
2.
System component LEDs are orange/red or supervisor not online
Complete these steps:
If you observe orange or red LEDs on startup, wait until the system boots up completely before
concluding that there is a problem. The system status LED on the supervisor will stay orange until
bootup is complete, then turn green if bootup is successful. One cause of an orange sys−status LED is
a fan failure.
Next, the supervisor initializes the switching modules, which operate differently depending on the
module; some flash on and off, and others stay orange until initialization is complete. At this point,
the link (port) LEDs turns off altogether until a signal is detected.
1.
Understand the Catalyst 4000 family components and what the LEDs tell you. As a starting place,
refer to Troubleshooting the Installation for more information:
Look at the front panel LEDs for your supervisor. Refer to these documents for more information:
Installing the Modules♦ Catalyst 4500 E−Series Module Installation Note♦
Look at the front panel LEDs for your switching module. Refer to Catalyst 4500 E−Series Module
Installation Note for more information:
2.
Capture the show version, show system, show module, and show test 0 command output.
Power supplyincludes the power supplies and power supply fans. The PS1, PS2, and PS3, for the
Catalyst 4006, status LEDs should be green. If one or both are red, this can indicate a power supply
failure.
When you issue the show system command, determine if PS1 or PS2 status is faulty.a.
3.

Note: The Catalyst 4006 requires two power supplies installed to operate the switch and the
third is for redundancy. Refer to Module Overview for more information.
Inspect the power supplies. Make sure there is power applied to both units. If a redundant
power supply is installed but has no power, the show system command output shows that the
power supply status and sys−status is faulty.
b.
Reseat the power supply. Try a different circuit or swap power cords. If the status is still red,
or the show system command output shows faulty, this is a power supply failure. Refer to
Removal and Replacement Procedures for more information.
Fan assemblyWhenever system power is on, the system fan assembly should operate. You
should be able to hear the fan assembly to determine if it operates.
c.
Inspect the fan assembly and power supplies to verify if power is being applied to the system.a. Issue the show system command to determine if the fan−status is faulty.b. Reseat the fan assembly and tighten the captive installation screws. If necessary, reset the
switch. If the show system command output still shows faulty, this is a fan failure. Refer
to Removal and Replacement Procedures for more information.
Supervisor engineThe supervisor engine contains the system operating software. Check the
supervisor engine if you have trouble with the system software. The status LED on the
supervisor engine indicates whether the supervisor engine has passed all diagnostic tests.
Have a console session open and determine whether the supervisor is in boot or rommon
mode. If this is the case, see the Supervisor is not online or stuck in rommon section for
troubleshooting steps.
c.
Issue the show system command in order to determine if the sys−status is faulty. Issue the
show test 0 command in order to determine if the supervisor has passed all diagnostic tests as
of the last bootup of the switch. Note any F for fail results.
a.
Inspect the fan assembly and power supplies for any problems.b. Have a console session open and capture bootup POST diagnostics and system error
messages. Reset the switch and issue the show test 0 command in order to determine if the
diagnostic test on bootup has been passed.
c.
Remove the supervisor and inspect for bent pins. Reseat the supervisor, firmly press down the
ejector levers, and tighten the captive installation screws. Wait for the supervisor to initialize.
If the show system command sys−status is still faulty, the supervisor has failed.
Switching modulesThe status LEDs on each switching module indicate whether the
switching module has been initialized correctly. The supervisor engine must be operating
properly before the switching module initializes. If a switching module is improperly installed
in the switch, it does not function.
d.
If a link (port) LED is solid orange or is blinking orange after the supervisor bootup and
module initialization, see the Can not connect to a remote host, router, or another switch
section.
a.
Capture the show version and show module command output. Determine if the software
version you are running supports this module. Refer to the Module Overview section of
Catalyst 4500 E−Series Module Installation Note for more information.
b.
Determine if the status is disable. This indicates that the module was administratively
disabled. The status LED is orange in this case. Issue the set module enable <mod>
command.
c.
View the output of the show module command in order to determine if status is faulty for
that module. View the output of the show test 0 command in order to determine if this
module passed all its diagnostic tests as of the last bootup of the switch. Note any F for fail
results.
d.
Have a console session open and capture bootup POST diagnostics and any system error
messages. Issue the reset <mod> command in order to reset the module. Issue the show test
0command in order to determine if this module has passed all of its diagnostic tests on
e.

bootup. Note any F for fail results.
Remove the module and inspect for bent pins. Reseat the module, firmly press down the
ejector levers, and tighten the captive installation screws. If the show module status is still
faulty, try the module in another slot. If necessary, power off/on the switch. If status is still
faulty, the module has failed.
f.
Switching module is not recognized
The most common cause for a switching module or a line card to not be recognized is due to the wrong
version of software.
Determine that this is a problem with just one module and not all modules. If all modules are affected,
complete the steps in the System component LEDs are orange/red or supervisor not online section.
Capture the output the show version , show module , and show test 0 commands.
1.
Issue the show version command in order to check the model number of the module you have
problems with and the software version you use. Determine the total DRAM and total Flash. Refer to
the Module Overview section of Catalyst 4500 E−Series Module Installation Note in order to
determine if the hardware is compatible with the software.
If an upgrade is required, always check the release notes first. Refer to the to Catalyst 4500 Family
Release Notes and choose the version to which you need to upgrade. For example, choose the Release
Notes for Catalyst 4000 Family Software Release 5.x and search on any information in regards to
your hardware.
Refer to Managing Software Images and Working with Configuration Files on Catalyst Switches for
more information.
2.
If the supervisor is not stuck in boot or rommon and you have determined that the module is
supported by the current version of software, complete the steps for troubleshooting the Switching
Module in the System component LEDs are orange/red or supervisor not online section.
3.
Module status is showing faulty or not ok
Complete these steps:
Capture the show module and show test 0 command output.1. For any status other than ok in the output of these two commands, complete the troubleshooting steps
for the Switching Module in the System component LEDs are orange/red or supervisor not online
section.
2.
Experiencing poor performance
Poor performance is often perceived to be a hardware issue, but this is usually not the case. When customers
describe to Cisco Technical Support that users on a particular switch experience slow performance, this often
turns out to be related to connectivity problems, software misconfiguration, or problems elsewhere on the
network.
Identify whether performance issues occur for users connected to all switching modules, one module
in particular, or just users on one or more ports. Capture the show module and show test 0 command
output. Make sure that the supervisor and modules have an ok status. If there is a faulty status,
complete the troubleshooting steps for the Switching Module in the System component LEDs are
orange/red or supervisor not online section.
1.
Capture the show port <mod/port> , show Mac<mod/port> , and show counters <mod/port>
command output. If you see incrementing errors on port counters, troubleshoot this performance issue
as a connectivity issue. See the Seeing errors on the ports section for troubleshooting steps.
2.

Capture the show config and show logging buffer 1023 command output. The show config
command shows only the non−default configuration changes. Ideally, every time you make a change,
you should have backed−up the configuration to use as a comparison. Issue the show config
command in order to possibly associate a configuration change with the behavior you experience.
If you see any system messages other than informational messages that can indicate a hardware or
some other problem, issue the show logging buffer 1023 command in order to capture these
messages. This command displays the last 1023 system messages with timestamps, by default. Also,
refer to Messages and Recovery Procedures well as Common CatOS Error Messages on Catalyst 4000
Series Switches in order to see if you can rule out any harmless system messages from those that can
indicate a problem. Use the Error Message Decoder tool in order to help decipher the output of any
messages.
3.
Many performance related problems are related to network traffic conditions. Capture the show
system command output in order to see if this is a network traffic problem.
The show system command can be used to check the current backplane utilization, which is typically
less than ten percent. If you believe that you are having performance related issues on a particular
switch, look at the Peak field, which is the peak backplane utilization on the switch since it was last
booted, and note the timestamp indicated by Peak−Time. Keep in mind that spikes in traffic
percentage on the backplane can be a STP loop or broadcast storm. Refer to Spanning Tree Protocol
Problems and Related Design Considerations for more information.
4.
Capture the show proc cpu command output. This command helps identify a process that can cause
high CPU utilization on the supervisor. This is an excerpt of show proc cpu command output:
Cat4000−c> (enable) show proc cpu
CPU utilization for five seconds: 11.62%
one minute: 12.00%
five minutes: 12.00%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
−−− −−−−−−−−−−− −−−−−−−−−− −−−−−−−− −−−−−−− −−−−−−− −−−−−−− −−− −−−−−−−−−−−−−−−
1 20176816 0 0 88.38% 88.00% 88.00% −2 Kernel and Idle
When you view the output of this command, remember that the CPU utilization is the first thing
shown. Do not confuse the Kernel and Idle amount as CPU utilization. The Kernel and Idle is the
percentage of CPU that was idle for that time frame. Therefore, in the past five minutes, only 11.62
percent of the CPU was utilized, which is within typical boundaries.
Refer to Understanding CPU Utilization on Catalyst 4000, 2948G, 2980G, and 4912G Switches for
more information and a complete understanding of how the CPU is utilized on Catalyst 4000, 2948G,
2980G, and 4912G switches.
Complete these steps in order to get a baseline of your switch and help identify which process can
cause a problem:
Issue the show proc cpu command during a time of normal activity for your network. Save
the results.
♦
Run this command again if you experience any performance related issues.♦ Compare the two outputs. Is there a process you can identify that is unusually high in
comparison?
♦
Run the command multiple times. Is there a significant increase or decrease in CPU
utilization or spikes? Or, does CPU utilization remain consistently high?
♦
The answer is most likely not a hardware problem, but points elsewhere.
5.
One performance related issue that results from misconfiguration is when the inband channel, which
is used for any control traffic terminating on the switch such as ping, Telnet, VLAN Trunk Protocol
(VTP), STP, CDP, and so forth, is not put in a separate VLAN from user data.
6.

It is always recommended to keep the management or sc0 interface of the switch in a separate VLAN
from the user data. Otherwise, any broadcast or multicast storm can flood the inband channel to the
Network Management Processor (NMP), which needs to be free to handle the protocols just
mentioned.
If you have not been able to track down any reason for performance issues on the switch in the
previous steps mentioned, capture the output of these commands, as well as the other commands in
the previous steps, and open a case with the Cisco Technical Support:
show nvramenv 1 (hidden)♦ show interposition 1 (hidden)♦
These are hidden commands, which means they cannot be parsed with a "?" and you cannot Tab to
complete. Type the command out in its entirety. It is possible that this output is not useful in the
resolution your case. These commands are undocumented, and therefore the TAC engineer is not
required to explain the output to the customer.
Although fairly rare, memory leaks do occur and can cause what seem naturally to be poor
performance and other symptoms. If you have not been able to track down any reason for
performance issues on the switch in the previous steps mentioned, capture the output of the show
mbuf total (hidden) command, as well as the other commands in the previous steps, and open a case
with the Cisco Technical Support.
There are two things to consider when you look at the output of this command in order to help
determine if you have a memory leak issue:
Look at the output and if the free mbufs or clusters values decrease but never increase, this
can indicate a possible memory leak.
♦
Look at the output, and if the lowest free memory has ever approached zero or was at zero,
this indicates the switch either runs low on or has ran out of memory.
♦
Both of these issues indicate a memory issue that obviously affects the protocols/processes that
require this memory.
Cat4000−c> (enable) show mbuf total
mbufs 9280 clusters 3660
free mbufs 9256 clfree 3659
lowest free mbufs 9235 lowest clfree 3638
These are hidden commands, which means they cannot be parsed with a "?" and you cannot Tab to
complete. Type the command out in its entirety. It is possible that this output is not useful in the
resolution your case. These commands are undocumented, and therefore the TAC engineer is not
required to explain the output to the customer.
7.
Getting system error messages
As mentioned in the Introduction of this document, Cisco has a suite of online diagnostic tools to help you
determine hw/sw compatibility, interpret output, and decode errors. One of these tools is the Error Message
Decoder. This tool requires a login and can be used to decipher the output of system messages you are
concerned about.
System messages have timestamps by default, which can help in isolating a timeframe for your
problem. by Issue the show time command in order to make sure your system clock is set correctly.
Also, verify that your connecting devices are set so that the logs match.
1.
Capture the output of any system messages with the show logging buffer 1023 command. Many
system messages are informational in nature while others can indicate a problem. Refer to these
documents for more information:
2.

Messages and Recovery Procedures♦ Common CatOS Error Messages on Catalyst 4000 Series Switches♦
Supervisor Crashes and Steps to Resolve Them
Supervisor crashes occur when the switch has reset, continually resets, or is down completely.
These commands are supported by the Output Interpreter and can be used to assist in troubleshooting
supervisor crashes: show version or show system.
If you have the output of the supported commands from your Cisco device, you can use Output Interpreter
(registered customers only) in order to display potential issues and fixes. In order to use Output Interpreter
(registered customers only) , you must be a registered user, be logged in, and have JavaScript enabled.
Getting system error messages
System error messages can be useful if you experience a switch reset. See the Getting system error messages
section for more information.
Switch has reset or is continually resetting
If the switch has reset or crashed due to a reason related to hardware or software, it is important to capture the
output of certain show commands as quickly as possible.
Capture the show log,show version,show test 0,and show logging buffer 1023 command output.
The show log command output has a number of important indications of problems that can be related
to a crash.
It keeps track of the last ten system resets with timestamps that show when the reboot
occurred. This is a snapshot of the Reboot History output:
Reboot History: Jan 23 2002 11:14:16 0, Jan 22 2002 14:57:21 0
DEC 24 2001 13:56:38 0, DEC 24 2001 13:52:30 0
DEC 11 2001 12:31:59 0, DEC 07 2001 13:26:48 0
DEC 07 2001 10:42:19 0, DEC 07 2001 10:36:16 0
Nov 28 2001 11:03:10 0, Oct 26 2001 16:04:26 0
The Reboot History only indicates that the switch was reset. It can have been reset manually
by the user or due to a crash. But, the most recent manual reset of the switch is recorded
further down in the output.
Last software reset by user: Jan 23 2002 11:14:16 0
Notice that the timestamp of the last manual reset, 1/23/2002,11:13:13, matches the most
recent entry in the Reboot History.
♦
It shows if there have been any exceptions. Exceptions are CPU dumps that occur
immediately after a crash. For example:
MCP Exceptions/Hang: 0
In this case, there were no exceptions recorded. If there were an exception, it includes a
timestamp that can be matched with the Reboot History, and also includes a HEX dump or
stack, which can be decoded by a TAC engineer in order to determine whether this was a
software forced exception or due to hardware.
♦
1.

The show version command provides software version information to use for a bug search. For
example, if you identify an exception in the show log command output, use the Bug Toolkit in order
to search for bugs on the Catalyst 4000 and the exception. Also, the show version command gives
you a quick snapshot of how long the switch has been up. For example:
Uptime is 28 days, 11 hours, 42 minutes
The show test 0 command output indicates an F status on the supervisor or module if any of the
diagnostics failed. An improperly seated module can cause the switch to crash. If the supervisor or
module shows failed, proceed with the troubleshooting steps in the System component LEDs are
orange/red or supervisor not online section of this document.
The show logging buffer 1023 command displays all system messages, which includes possible error
messages that can relate to the crash. See the Getting system error messages section for
troubleshooting suggestions.
Issue the show commands and troubleshooting procedures in the preceding steps first. If these steps
fail, capture the show tech−support command output. This command displays output for all these
commands continuously, which means the output continues to scroll until complete or until the
display is ended with the Ctrl + C keystrokes:
sh version, sh flash, sh microcode, sh system, sh module, sh port, sh Mac, sh trunk, sh vlan, sh
vtp domain, sh spantree active, sh spantree summary, sh test, sh arp, sh ip route, sh cdp
neighbor detail, sh netstat stats, show memory buffers, show out−of−band stats, sh inband stats,
show cam static, sh cam count dynamic, sh cam system, sh config, sh log, sh proc, sh proc mem,
sh proc CPU, ps, Ps −c
Often, the output from all these commands is not necessary to resolve a specific problem, so TAC
engineers cannot ask for it. But, it is beneficial to have this output should other show commands or
troubleshooting steps fail to resolve the problem.
2.
Should all the previous troubleshooting steps fail to diagnose the problem, capture these hidden
commands, as well as the other commands in the previous steps, and open a case with the Cisco
Technical Support:
ps−c (capture multiple times)♦ show mbuf all (hidden)♦ show nvramenv 1 (hidden)♦ show interposition 1 (hidden)♦
These are hidden commands, which means they cannot be parsed with a "?" and you cannot Tab to
complete. Type the command out in its entirety. This output can or cannot be useful in the resolution
of your case. These commands are undocumented, and therefore the TAC engineer is not required to
explain the output to the customer.
3.
Misleading Problems
There are many misleading problems that are thought to be caused by faulty hardware. This section lists a few
issues that are often confused as a hardware failure.
One common customer issue is for the system LED to show faulty when additional power supplies
are added, but not plugged in. When this happens, both the ps#−status and sys−status shows faulty.
This is because the switch senses an additional power supply is installed but is not active. Since this
can also mean that the additional power supply has actually failed, an onsite inspection is required.
•
A common misconception when you view output of the show proc cpu command is that the Kernel
and Idle percentage is interpreted to be the CPU utilization for that time period. The Kernel and Idle is
the percentage of CPU that was idle for that time frame.
•

show Command Descriptions
These table breaks down what show commands are used to help troubleshoot the different symptom types.
Connectivity
Problems System/Supervisor/Module
Problems Supervisor Resets/Crashes
show version
show config
show module
show system
show port
capabilities show
port <mod/port>
show
Mac<mod/port>
show counters
<mod/port>
clear counters
show cdp
neighbors detail
show spantree
summary
show version show module
show flash show config
show test 0 show system
show time show logging
buffer 1023 show proc CPU
or Ps −c show mbuf all
(hidden) show nvramenv 1
(hidden) show interposition
1(hidden)
show log show
logging buffer
1023 show
version show
test 0 show
system show
tech support
ps−c (multiple
times) (hidden)
show mbuf all
(hidden) show
nvramenv 1
(hidden) show
interposition 1
(hidden)
Capture these show commands that depends on your symptom(s).
Notice that many of the commands in each previous symptom category overlap. This is because the same
symptom can occur in different levels of severity; one can cause a performance issue and the other can cause a
crash.
Notice also that some of the commands seem meant more for software troubleshooting or configuration
problems. For instance, the show spantree summary command shows which VLANs run STP, how many
ports are in each VLAN, if any ports on the switch are blocking, and for which VLANs that they are blocking.
Since STP loops can actually bring down a switch or network that gives the appearance of a hardware failure,
this is vital information to capture, whether troubleshooting hardware or software.
show version
This command verifies the version of software you are running. This command also has information about the
size of Flash and DRAM. This is useful information should you need to upgrade. If an upgrade is required,
always check the release notes first. Refer to the to Catalyst 4500 Family Release Notes and choose the
version you need to upgrade. For example, choose the Release Notes for Catalyst 4000 Family Software
Release 5.x and search on any information in regards to your hardware.
Refer to Table 2−82: show version Command Output Fields for more information.
Cat4000−c> (enable) show version
WS−C4006 Software, Version NmpSW: 6.3(1)
Copyright (c) 1995−2001 by Cisco Systems, Inc.
NMP S/W compiled on Jul 24 2001, 12:55:29
GSP S/W compiled on Jul 24 2001, 10:36:29
System Bootstrap Version: 5.4(1)
Hardware Version: 2.0 Model: WS−C4006 Serial #: JAB04380209

Mod Port Model Serial # Versions
−−− −−−− −−−−−−−−−− −−−−−−−−−−−−−−−−−−−− −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
1 2 WS−X4013 JAB04380209 Hw : 2.0
Gsp: 6.3(1.0)
Nmp: 6.3(1)
2 34 WS−X4232−L3 JAB045004AA Hw : 1.5
3 24 WS−X4424−GB−RJ45 JAB0514071N Hw : 0.7
5 6 WS−X4306 JAB02400048 Hw : 0.2
DRAM FLASH NVRAM
Module Total Used Free Total Used Free Total Used Free
−−−−−− −−−−−−− −−−−−−− −−−−−−− −−−−−−− −−−−−−− −−−−−−− −−−−− −−−−− −−−−−
1 65536K 33235K 32301K 16384K 16173K 211K 480K 180K 300K
Uptime is 28 days, 11 hours, 42 minutes
show module
This command displays information about the modules installed in the switch. In particular, note the status of
the module. If the status is faulty, this can be a hardware failure.
Cat4000−c> (enable) show module
Mod Slot Ports Module−Type Model Sub Status
−−− −−−− −−−−− −−−−−−−−−−−−−−−−−−−−−−−−− −−−−−−−−−−−−−−−−−−− −−− −−−−−−−−
1 1 2 1000BaseX Supervisor WS−X4013 no OK
2 2 34 Router Switch Card WS−X4232−L3 no OK
3 3 24 10/100/1000 Ethernet WS−X4424−GB−RJ45 no disable
5 5 6 1000BaseX Ethernet WS−X4306 no OK
Mod Module−Name Serial−Num
−−− −−−−−−−−−−−−−−−−−−−− −−−−−−−−−−−−−−−−−−−−
1 JAB04380209
2 JAB045004AA
3 JAB0514071N
5 JAB02400048
Mod MAC−Address(es) Hw Fw SW
−−− −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− −−−−−− −−−−−−−−−− −−−−−−−−−−−−−−−−−
1 00−02−b9−83−ac−00 to 00−02−b9−83−af−ff 2.0 5.4(1) 6.3(1)
2 00−02−16−f6−64−5c to 00−02−16−f6−64−7d 1.5 12.0(7)W5( 12.0(14)W5(20)
3 00−30−85−0e−2c−18 to 00−30−85−0e−2c−2f 0.7
5 00−10−7b−f6−9c−e4 to 00−10−7b−f6−9c−e9 0.2
Cat4000−c> (enable)
Refer to Table 2−35: show module Command Output Fields for more information.
show flash
This command displays the the contents of the Flash file system. Flash file systems differ between Catalyst
supervisors. Some supervisors use the show flash command to display the contents, while others use the dir
bootflash: command. When you copy an image to the SupIIIG, for example, you use the download command
and the Flash is completely erased in the process of installing the image. With other sups, you can use the
copy tftp flash command in order to add one or more images.
Many problems, both hardware and software related, can be avoided if you understand the Flash system for
your supervisor.
Refer to the show flash or dir bootflash: command for more information.
Cat4000−c> sh flash
−#− ED −−type−− −−crc−−− −seek−− nlen −length− −−−−−date/time−−−−−− name

1 .. ffffffff 4e88958b 42a97c 17 4106492 Aug 17 2001 16:22:52 cat4000.6−3n
2 .. ffffffff b965ace8 78e71c 18 3554592 Nov 28 2001 10:38:33 cat4000.5−5n
3 .. ffffffff 70a608c8 b8fa9c 20 4199168 DEC 07 2001 10:30:01 cat4000−k9.n
4 .. ffffffff e873ea40 f0b224 17 3651336 DEC 11 2001 12:26:20 cat4000.5−5n
216540 bytes available (15512100 bytes used)
Cat4000−c>
show config
This command displays the non−default system configuration. This is useful to capture every time you make a
configuration change as a way to possibly associate changes to hardware or software problems. Notice there is
is a timestamp for each output. Compare the output to the show config all command output, which shows the
entire system configuration and can be quite lengthy. Refer to the show config command for more
information.
Cat4000−c> (enable) show config
This command shows non−default configurations only.
Use 'show config all' to show both default and non−default configurations.
.............
..................
....................
..
begin
!
# ***** NON−DEFAULT CONFIGURATION *****
!
!
#time: Tue Jan 22 2002, 11:20:05
!
#version 6.3(1)
!
!
#system web interface version(s)
!
#test
!
#system
set system name Cat4000−c
!
#frame distribution method
set port channel all distribution Mac both
!
#vtp
set vtp domain blah
!
#ip
set interface sc0 1 172.16.84.200/255.255.255.0 172.16.84.255
set interface sl0 down
set interface me1 1.1.1.1 255.255.255.0 1.1.1.255
set ip route 0.0.0.0/0.0.0.0 172.16.84.1
!
#syslog
set logging level cops 2 default
!
#set boot command
set boot config−register 0x2102
clear boot system all
!