Cisco CRS-1 - Carrier Routing System Router User manual

Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Cisco IOS XR Troubleshooting Guide for
the Cisco CRS-1 Router
Cisco_IOS_XR_Software Release_3.9
October, 2010
Text Part Number: OL-21483-02

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at
www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1005R)
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display
output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in
illustrative content is unintentional and coincidental.
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
© 2010 Cisco Systems, Inc. All rights reserved.

i
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
OL-21483-02
CONTENTS
Preface vii
Changes to This Document vii
Obtaining Documentation and Submitting a Service Request viii
CHAPTER 1General Troubleshooting 1-1
Troubleshooting Techniques and Approaches 1-1
Documenting the Network 1-2
Verifying and Troubleshooting CLI Access 1-2
General CLI Access Information 1-2
User Access Privileges 1-3
CLI Access Through a Console Port 1-3
CLI Access Through a Terminal Server 1-3
CLI Access Through the Management Ethernet Interface 1-4
Basic Cisco IOS XR Verification and Troubleshooting Tools 1-7
man Command 1-7
describe Command 1-9
show platform Command 1-10
top Command 1-10
show context Command 1-11
show users Command 1-13
Verifying the System 1-13
Validating and Troubleshooting Cisco IOS XR Software Package Installation 1-27
Verifying the Software Version 1-27
Validating the Installation 1-30
install verify Command 1-30
show install active Command 1-32
show install committed Command 1-33
Validating and Troubleshooting Cisco IOS XR Software Configuration 1-35
Local and Global Configurations 1-35
Collecting Configuration Information 1-37
Verifying the Running Configuration 1-38
Using the show configuration failed Command 1-47
Startup Failed Configuration 1-47
Commit Configuration Failed 1-48

Contents
ii
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
OL-21483-02
ASIC Errors 1-49
Trace Commands 1-56
Packets 1-57
Logging Archive for Harddisk 1-60
SNMP Polling Awareness of SystemOwner, LR Owner, MIB Location 1-60
Error File Locations and Data Collection Scripts 1-60
Error File Locations 1-61
harddisk: 1-61
Default disk location 1-62
Sysmgr Collection Scripts 1-62
Wdsysmon Collection Scripts 1-63
Shutdown Collection Scripts 1-66
ASIC error Collection Scripts 1-66
Monitoring 1-66
monitor interface Command 1-67
monitor controller Command 1-68
monitor processes Command 1-69
monitor threads Command 1-70
Gathering Information for Technical Support 1-70
CHAPTER 2Troubleshooting Booting 2-73
Booting Tips 2-73
Verifying Successful Bootup 2-74
Verifying and Troubleshooting the Console Connection 2-74
Verifying and Troubleshooting Route Processor and Shelf Controller Cards 2-75
Troubleshooting RP and SC Cards Not Booting 2-75
Troubleshooting RP and SC Cards Resetting While Booting 2-78
Troubleshooting Blocked FCC Shelf Controller or LCC Route Process Minimum Boot Image
Requests 2-79
CHAPTER 3Troubleshooting Forwarding 3-81
Troubleshooting IPv4 CEF Information 3-81
Troubleshooting Adjacency Information 3-85
Troubleshooting Transient Traffic Drop in Forwarding 3-88
Troubleshooting Control Plane Information 3-90
Troubleshooting the Interface Manager 3-95
Interface Manager Control Process 3-96
Troubleshooting the Trace Logs for the Interface Manager 3-100

Contents
iii
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
OL-21483-02
Troubleshooting the Client for the Interface Manager 3-101
Troubleshooting the Rules for the Interface Manager 3-101
Troubleshooting the Control Chain and Interface Information 3-103
Troubleshooting the Registrations for the Interface Manager 3-105
Troubleshooting the Interface Manager Distributor 3-106
Interface Manager Distributor Overview 3-106
Troubleshooting the Trace Logs for the Interface Manager Distributor 3-107
Troubleshooting the Clients for the Interface Manager Distributor 3-107
Troubleshooting the Interface Information for the Interface Manager Distributor 3-108
Troubleshooting the Global Registrations for the Interface Manager Distributor 3-109
Troubleshooting the Rules for the Interface Manager Distributor 3-110
CHAPTER 4Troubleshooting Router Switch Fabric 4-111
CRS-1 Switch Fabric Overview 4-111
Understanding the Flags Field 4-113
Using the Online Diagnostics Tools 4-115
Verifying and Troubleshooting the Fabric Plane State 4-115
Verifying and Troubleshooting Up Fabric Planes 4-121
Troubleshooting Down Fabric Planes 4-126
Guidelines for Maintenance of Fabric Links 4-130
CHAPTER 5Troubleshooting Interfaces 5-131
Verifying and Troubleshooting Configured Interfaces 5-131
Verifying and Troubleshooting Pluggable Optical Interfaces 5-139
Troubleshooting Common Issues with Bundle Interfaces 5-144
CHAPTER 6Troubleshooting the Control Plane Ethernet Network 6-147
Cisco CRS-1 Control Plane Ethernet Network Overview 6-147
Using the Online Diagnostics Tools 6-148
Troubleshooting Booting the System Control Plane Ethernet Network 6-149
Examples 6-151
Troubleshooting the Multishelf System Router Topology 6-153
Examples 6-156
Troubleshooting the Catalyst Switch 6-158
Restrictions 6-158
Troubleshooting the CRS-1, 4-slot, 8-slot, or 16-slot System Router Topology 6-159
Examples 6-162

Contents
iv
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
OL-21483-02
CHAPTER 7Collecting System Information 7-165
Capturing Logs 7-165
Using ping and traceroute 7-166
Using Debug Commands 7-166
Using Diagnostic Commands 7-166
Online Diagnostics 7-166
Transient Condition when Standby RP Becomes Active 7-168
Offline Diagnostics—FDIAG RUNNING State 7-169
Additional Reference for Diagnostic Commands 7-169
Commands Used to Display Process and Thread Details 7-169
CHAPTER 8Process Monitoring and Troubleshooting 8-171
System Manager 8-172
Watchdog System Monitor 8-172
Deadlock detections 8-172
Hang detection 8-172
Core Dumps 8-173
follow Command 8-173
show processes Commands 8-175
show processes boot Command 8-175
show processes startup Command 8-176
show processes failover Command 8-177
show processes blocked Command 8-178
Example: 8-179
Redundancy and Process Restartability 8-179
Process States 8-180
Synchronous Message Passing 8-181
Blocked Processes and Process States 8-181
Process Monitoring 8-182
Process Monitoring Commands 8-183
Monitoring CPU Usage and Using Syslog Messages 8-183
Troubleshooting High CPU Utilization and Process Timeouts 8-185
General Guidelines for Troubleshooting CPU Utilization Problems 8-185
Using show process and top processes Commands 8-186
Troubleshooting a Process Block 8-188
Troubleshooting a Process Crash on Line Cards 8-192
Troubleshooting a Memory Leak 8-193
Troubleshooting a Hardware Failure 8-194

Contents
v
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
OL-21483-02
Troubleshooting SNMP Timeouts 8-194
Troubleshooting Communication Among Multiple Processes 8-194
Troubleshooting a Process Restart 8-195
CHAPTER 9Troubleshooting Memory 9-197
Watchdog System Monitor 9-197
Memory Monitoring 9-197
Configuring and Displaying Memory Thresholds 9-198
Setting Timeout for Persistent CPU Hogs 9-201
Memory Usage Analyzer 9-202
Troubleshooting Global Memory 9-202
Troubleshooting Process Memory 9-203
Identifying Process Memory Problems 9-203
Resolving Process Memory Problems 9-207
CHAPTER 10 Troubleshooting Upgrading and Downgrading Software 10-209
Validating and Troubleshooting ROM Monitor Software Installation 10-209
Verifying and Troubleshooting the ROM Monitor Version 10-210
Troubleshooting Upgrading and Downgrading ROM Monitor Software on Cisco CRS-1 Routers 10-213
Troubleshooting Upgrading and Downgrading Cisco IOS XR Software Packages 10-214
CHAPTER 11 Troubleshooting the Statistics Infrastructure 11-215
Debugging Statistics Infrastructure 11-215
debug statsd api Commands 11-215
debug statsd manager Commands 11-216
debug statsd server errors Command 11-216
Trace Commands for the Statistics Infrastructure 11-217
show statsd manager trace Command 11-217
show statsd server trace Command 11-218
show stats lib trace 11-218
Show Commands for the Statistics Infrastructure 11-218
show statsd manager info Command 11-219
show statsd collectors Command 11-220
show statsd registrations Command 11-222
show statsd requests Command 11-222
Diagnosing Problems with Statistics Values 11-223
Errors When Retrieving Data from the Statistics Manager EDM 11-224
Timeouts and Delays When Retrieving Data 11-224

Contents
vi
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
OL-21483-02
Displaying Incorrect Rates for the show interfaces Command 11-225
Displaying Incorrect Data for the show Commands 11-226
CHAPTER 12 Multiprotocol Label Switching Checklist 12-229
Multiprotocol Label Switching Recommendations 12-229
Troubleshooting L3VPN MPLS Traffic Loss 12-230
CHAPTER 13 Troubleshooting Load Balancing 13-231
About Load Balancing with Cisco Express Forwarding 13-231
Load Balancing Function 13-231
Source Information for Load Balancing 13-231
Layer 2 Load Balancing 13-232
Terminology 13-233
Troubleshooting Layer 3 or Layer 4 Load Balancing 13-233
Verifying the Routing Table Entries for Parallel Links 13-234
Configuring Layer 4 Load Balancing 13-236
Verifying the CEF Database and Measuring Flows 13-236
Troubleshooting Layer 2 Load Balancing 13-237
Verifying the Bundle Status, IGP Route, and CEF Database 13-237
Configuring Layer 4 Load Balancing 13-238
Viewing the Expected Paths and Measuring the Flows 13-238
Configuration Examples for Troubleshooting Load Balancing 13-239
Troubleshooting Layer 3 or Layer 4 Load Balancing Example 13-240
Verifying Parallel Links in the Routing Table 13-240
Verifying IPv4/IPv6 Unicast Load Balancing 13-243
Troubleshooting Layer 2 Load Balancing Example 13-246
Verifying the Bundle Status 13-246
Checking the IGP Routing Table and CEF Database 13-247
Verifying Bundle Load Balancing 13-247
I NDEX

vii
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
OL-21483-02
Preface
This guide describes how to troubleshooting a router using the Cisco IOS XR software.
This preface contains the following sections:
• Changes to This Document, page vii
• Obtaining Documentation and Submitting a Service Request, page viii
Changes to This Document
Table 1 lists the technical changes made to this document since it was first issued.
Table 1 Changes to This Document
Revision Date Change Summary
OL-21483-02 October,
2010
Added information about the following topics:
• Chapter 1—Added links to information on SNMP timeouts and the MIBs download site.
• Chapter 1—Added a list of commands to use to gather information prior to contacting TAC.
• Chapter 4—Added a drawing of the fabric architecture.
• Chapter 4—Added information about using online diagnostics to help troubleshoot fabric.
• Chapter 4—Added the CLI command show controllers fabric link health.
• Chapter 4—Added information about using the correct fiber cleaning kit and procedure.
• Chapter 4—Added a section with guidelines on maintenance of fabric links.
Continued, next page

viii
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router
OL-21483-02
Preface
Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, submitting a service request, and gathering additional
information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and
revised Cisco technical documentation, at:
http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html
Subscribe to the What’s New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed
and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free
service and Cisco currently supports RSS version 2.0.
OL-21483-02 October,
2010
(Continued from previous page)
• Chapter 5—Added a section on troubleshooting the line card optical interfaces
(pluggable-optics).
• Chapter 6—Added information on using online diagnostics to help troubleshoot the Ethernet
control plane.
• Chapter 7—Added CLI commands for displaying online diagnostics.
• Chapter 7—Added a note about online diagnostics behavior after an RP failover.
• Chapter 8—Added details on SNMP timeouts.
• Chapter 8—Added information on normal and abnormal causes of high CPU utilization.
• Chapter 10—Added clarifications regarding software and firmware upgrades.
OL-21483-01 December,
2009
Initial release of this document.
Table 1 Changes to This Document (continued)
Revision Date Change Summary