Alcatel-Lucent OmniSwitch 6860 User manual

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
OmniSwitch 6860/6900/10K
Troubleshooting Guide
Alcatel-Lucent Enterprise
26801 West Agoura Road, Calabasas, CA 91301
[phone]
www.alcatel-lucent.com
Copyright 1995-2015 Alcatel-Lucent
ALL RIGHTS RESERVED

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 2 of 148
Table of Contents
1. About This Guide............................................................................................................................................4
1.1. Supported Platforms ..................................................................................................................................4
1.2. Who Should Read this Manual? ................................................................................................................4
1.3. When Should I Read this Manual? ............................................................................................................4
1.4. What is in this Manual?.............................................................................................................................4
1.5. What is Not in this Manual? ......................................................................................................................5
1.6. How is the Information Organized?...........................................................................................................5
1.7. Related Documentation..............................................................................................................................5
1.8. Before Calling Alcatel-Lucent’s Technical Assistance Center..................................................................6
2. Troubleshooting the Switch System..............................................................................................................8
2.1. Introduction................................................................................................................................................8
2.2. Troubleshooting the System on OS6900/OS10K Switches.......................................................................8
2.3. Troubleshooting System on OS6860/E Switches ......................................................................................9
2.4. Advanced Troubleshooting......................................................................................................................12
2.5. Memory Leak...........................................................................................................................................14
2.6. Packet Driver...........................................................................................................................................14
2.7. The second column of output provides changes since the previous execution of the command.Logs....15
3. Troubleshooting Virtual Chassis.................................................................................................................16
3.1. Introduction..............................................................................................................................................16
3.2. Basic Troubleshooting.............................................................................................................................18
3.3. Advanced Troubleshooting......................................................................................................................19
3.4. Scenarios :................................................................................................................................................22
4. Troubleshooting Switched Ethernet Connectivity.....................................................................................27
4.1. Verify Physical Layer Connectivity.........................................................................................................27
4.2. Verify Current Running Configuration....................................................................................................27
4.3. Verify Source Learning ...........................................................................................................................28
4.4. Verify Switch Health...............................................................................................................................28
4.5. Verify ARP..............................................................................................................................................28
5. Troubleshooting Source Learning/Layer 2 Forwarding...........................................................................29
5.1. Basic Troubleshooting.............................................................................................................................29
5.2. Advanced Troubleshooting Scenarios .....................................................................................................31
MAC Addresses Not Aging Out.....................................................................................................................31
MAC Address Flapping..................................................................................................................................33
5.3. bShell Troubleshooting............................................................................................................................34
6. Troubleshooting ARP...................................................................................................................................37
6.1. Basic Troubleshooting.............................................................................................................................37
6.2. Advanced ARP Troubleshooting.............................................................................................................39
7. Troubleshooting Spanning Tree..................................................................................................................42
7.1. Basic troubleshooting ..............................................................................................................................42
7.2. Advanced Troubleshooting......................................................................................................................44
8. Troubleshooting Link Aggregation.............................................................................................................48
8.1. Basic Troubleshooting.............................................................................................................................49
8.2. Advanced Troubleshooting......................................................................................................................50
9. Troubleshooting BOOTP/DHCP/UDP Relay.............................................................................................53
9.1. Troubleshooting DHCP ...........................................................................................................................53
9.2. Troubleshooting UDP Relay....................................................................................................................56
9.3. Basic troubleshooting ..............................................................................................................................59
9.4. Advanced troubleshooting.......................................................................................................................61
9.5. Troubleshooting in Maintenance Shell....................................................................................................62
10. Troubleshooting QoS..................................................................................................................................63
10.1. Introduction............................................................................................................................................63
10.2. Basic Troubleshooting...........................................................................................................................75

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 3 of 148
10.3. Advanced troubleshooting.....................................................................................................................76
10.4. Troubleshooting in the Maintenance Shell............................................................................................79
10.5. Troubleshooting in bShell......................................................................................................................79
11. Troubleshooting RIP ..................................................................................................................................82
12. Troubleshooting OSPF...............................................................................................................................85
12.1. Supported debug variables.....................................................................................................................85
12.2. Planned and unplanned Virtual Chassis takeover..................................................................................85
12.3. Minimum working configuration...........................................................................................................87
12.4. Basic troubleshooting ............................................................................................................................87
12.5. Advanced troubleshooting.....................................................................................................................89
13. Troubleshooting BGP.................................................................................................................................93
13.1. BGP process...........................................................................................................................................93
13.2. Advance Troubleshooting......................................................................................................................97
14. Troubleshooting IP Multicast Switching (IPMS).....................................................................................99
14.1. Introduction............................................................................................................................................99
14.2. Basic troubleshooting ..........................................................................................................................106
14.3. Advanced troubleshooting...................................................................................................................109
14.4. Troubleshooting in the Maintenance Shell..........................................................................................110
14.5. Troubleshooting in bShell....................................................................................................................111
15. Troubleshooting IP Multicast Routing (IPMR).....................................................................................113
15.1. Introduction..........................................................................................................................................113
15.2. Minimum working configuration.........................................................................................................113
15.3. Basic Troubleshooting.........................................................................................................................114
15.4. Advanced Troubleshooting..................................................................................................................115
15.5. Troubleshooting in bShell....................................................................................................................115
16. Troubleshooting 802.1X ...........................................................................................................................116
17. Troubleshooting Universal Network Profiles (UNP).............................................................................118
17.1. Troubleshooting in bShell....................................................................................................................118
18. Troubleshooting SNMP............................................................................................................................120
18.1. Troubleshooting SNMP on OmniSwitch OS6900/OS10K/OS6860 series..........................................120
18.2. SNMP Security....................................................................................................................................124
18.3. SNMP Statistics...................................................................................................................................125
19. Troubleshooting Power Over Ethernet...................................................................................................127
19.1. Troubleshooting PoE on OmniSwitch on OS6860 and OS6860E.......................................................127
20. Troubleshooting Ethernet Ring Protection (ERP).................................................................................130
20.1. Troubleshooting ERP on OmniSwitch.................................................................................................130
21. Troubleshooting Shortest Path Bridging (SPB).....................................................................................136
21.1. Troubleshooting SPB on OmniSwitch OS6900/OS10K/OS6860 series .............................................136
21.2. SPB debug information........................................................................................................................138
21.3. Advanced Troubleshooting Scenarios .................................................................................................139
21.4. bShell Troubleshooting........................................................................................................................140
22. Troubleshooting sFlow.............................................................................................................................141
22.1. sFLOW Debug.....................................................................................................................................142
23. Troubleshooting Port Mirroring and Port Monitoring.........................................................................143
23.1. Troubleshooting Port Mirroring...........................................................................................................143
23.2. Troubleshooting port monitoring.........................................................................................................144
24. Troubleshooting IPV6 ..............................................................................................................................146
24.1. IPv6 Routing........................................................................................................................................146
24.2. Troubleshooting DHCPv6 Relay.........................................................................................................147
24.3. Troubleshooting a 6to4 Tunnel............................................................................................................148

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 4 of 148
1. About This Guide
The OmniSwitch troubleshooting guide describes how to use Command Line Interface (CLI) and low level
shell commands available on the OmniSwitch Family to troubleshoot switch and network problems. Reading
the OmniSwitch User Guides prior to reading this guide is highly recommended.
This document is for internal Alcatel-Lucent employees only. Distribution to clients, users, and
partners should ONLY be done with the consent of Technical Support.
1.1. Supported Platforms
This information in this guide applies to the following products:
• OmniSwitch 6860
• OmniSwitch 6860E
• OmniSwitch 6900
• OmniSwitch 10K
1.2. Who Should Read this Manual?
The principal audience for this user guide is Service and Support personnel who need to troubleshoot switch
problems in a live network. In addition, network administrators and IT support personnel who need to
configure and maintain switches and routers can use this guide to troubleshoot a problem upon advice from
Alcatel-Lucent Service and Support personnel. However, this guide is not intended for novice or first-time
users of Alcatel-Lucent OmniSwitches. Misuse or failure to follow procedures in this guide correctly can cause
lengthy network down time and/or permanent damage to hardware. Caution must be followed on distribution
of this document.
1.3. When Should I Read this Manual?
Always read the appropriate section or sections of this guide before you log into a switch to troubleshoot
problems. Once you are familiar with the commands and procedures in the appropriate sections you can use
this document as reference material when you troubleshoot a problem.
1.4. What is in this Manual?
The principal sections (i.e., the chapters numbered numerically) use CLI and Dshell commands to analyze and
troubleshoot switch problems. Each section documents a specific switch feature (e.g., hardware, server load
balancing, routing).
Note. Dshell commands should only be used by Alcatel-Lucent personnel or under the
direction of Alcatel-Lucent. Misuse or failure to follow procedures that use Dshell
commands in this guide correctly can cause lengthy network down time and/or
permanent damage to hardware.

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 5 of 148
1.5. What is Not in this Manual?
This guide is intended for troubleshooting switches in live networks. It does not provide step-by-step
instructions on how to set up particular features on the switch or a comprehensive reference to all CLI
commands available in the OmniSwitch. For detailed syntax on non debug CLI commands and comprehensive
information on how to configure particular software features in the switch, consult the user guides, which are
listed in “Related Documentation” on page 4.
1.6. How is the Information Organized?
Each chapter in this guide includes troubleshooting guidelines related to a single software feature, such as
server load balancing or link aggregation.
1.7. Related Documentation
The following are the titles and descriptions of all the Release 8 and later OmniSwitch user guides:
• OmniSwitch 6860/6860E Hardware Users Guide
Complete technical specifications and procedures for OmniSwitch 6860, power supplies, fans, and Network
Interface (NI) modules.
• OmniSwitch 6900 Hardware Users Guide
Complete technical specifications and procedures for OmniSwitch 6900, power supplies, fans, and Network
Interface (NI) modules.
• OmniSwitch 10K Hardware Users Guide
Complete technical specifications and procedures for OmniSwitch 10K, power supplies, fans, and Network
Interface (NI) modules.
• OmniSwitch AOS Release(7/8) CLI Reference Guide
Complete reference to all CLI commands supported on the OmniSwitch. Includes syntax definitions, default
values, examples, usage guidelines and CLI-to-MIB variable mappings.
• OmniSwitch AOS Release(7/8) Switch Management Guide
Includes procedures for readying an individual switch for integration into a network. Topics include the
software directory architecture, image rollback protections, authenticated switch access, managing switch files,
system configuration, using SNMP, and using web management software (WebView).
• OmniSwitch AOS Release (7/8) Network Configuration Guide
Includes network configuration procedures and descriptive information on all the major software features and
protocols included in the base software package. Chapters cover Layer 2 information (Ethernet and VLAN
configuration), Layer 3 information (RIP and static routes), security options (authenticated VLANs), Quality
of Service (QoS), and link aggregation.
• OmniSwitch AOS Release (7/8) Series Advanced Routing Configuration Guide
Includes network configuration procedures and descriptive information on the software features and protocols
included in the advanced routing software package (OSPF, IS-IS,BGP, DVMRP, PIM-SM).
OmniSwitch AOS Release 7 Data Center Switching Guide
Includes configuration information for data center networks using virtualization technologies (SPBM and
UNP), Data Center Bridging protocols (PFC, ETC, and DCBX), and FCoE/FC gateway functionality.
• OmniSwitch AOS Release (7/8) Tranceiver Guide
This OmniSwitch Transceivers Guide provides specifications and compatibility information for the supported
OmniSwitch transceivers for all OmniSwitch AOS 8 Release Products.
• Technical Knowledge Center, Field Notices
Includes information published by Alcatel Lucent Enterprise’s Service and Support group.
• Release Notes
Includes critical Open Problem Reports, feature exceptions, and other important information on the features
supported in the current release and any limitations to their support.

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 6 of 148
These user guides can be provided by contacting support or downloaded at Alcatel-Lucent Enterprise support
website.
Telephone: [phone]
Email: [e-mail]
Support Web Site: http://service.esd.alcatel-lucent.com
1.8. Before Calling Alcatel-Lucent’s Technical Assistance Center
Before calling Alcatel-Lucent’s Technical Assistance Center (TAC), make sure that you have read through the
appropriate section (or sections) and have completed the actions suggested for your system’s problem.
Additionally, do the following and document the results so that the Alcatel-Lucent TAC can better assist you:
• Have a network diagram ready. Make sure that relevant information is listed, such as all IP addresses and
their associated network masks.
• Have any information that you gathered while troubleshooting the issue to this point available to provide to
the TAC engineer.
• If the problem appears to be with only a few-fewer than four-switches, capture the output from the “show
tech-support” CLI command on these switches. (See Appendix C, “Technical Support Commands,” for more
information on show tech-support CLI commands.)
When calling Alcatel-Lucent TAC in order to troubleshoot or report a problem following information can be
helpful to get a quick resolution:
boot.cfg file (vcboot.cfg,vcsetup.cfg in case of Virtual Chassis)
tech_support *.log & *.tar files, created by using:
• show tech-support
• show tech-support layer2
• show tech-support layer3
• show tech-support eng
swlog, swlog.0, swlog.1 up to swlog.6 located in /flash/ and swlog_chassisX, swlog_chassisX.0,
swlog_chassisX.1 up to swlog_chassisX.6 located in /flash/chassisX (where X is the active chassis number;
e.g. 127.10.1.65)
command.log file if present
PMD files (Post Mortem Dump) if present (*pmd*) located in /flash/pmd and/or /flash/niX/pmd (where X is
the active NI number; e.g. /flash/ni1/pmd)
Captures of the following commands:
• ls –lR (case sensitive)
• show transceivers
• show configuration status
• show log swlog
• show command-log (if command-log is enabled)
• show user
• show snmp statistics
• show mac-learning
• boardinfo
Console output if captured during the issue
SNMP Traps received by the NMS during the issue and any other

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 7 of 148
Health graphs captured during that time
Dial-in or remote access can also provide effective problem resolution.
If a Virtual Chassis fail over to the secondary chassis happened because of this failure then include this
information from both of the chassis.
6860-> debug show virtual-chassis connection
Address Address
Chas MAC-Address Local IP Remote IP Status
-----+------------------+-----------------+-----------------+-------------
1 e8:e7:32:b3:34:51 127.10.2.65 127.10.1.65 Connected
3 e8:e7:32:b3:36:9b 127.10.2.65 127.10.3.65 Connected
4 e8:e7:32:b3:37:17 127.10.2.65 127.10.4.65 Connected
In SuperUser Mode the chassis are mounted as follows:
127.10.1.65:/flash/ on /mnt/chassis1_CMMA
127.10.2.65:/flash/ on /mnt/chassis2_CMMA
127.10.3.65:/flash/ on /mnt/chassis3_CMMA
127.10.4.65:/flash/ on /mnt/chassis4_CMMA
6860-> su
Entering maintenance shell. Type 'exit' when you are done.
SHASTA #-> ls /mnt/chassis2_CMMA/
811_555 swlog_chassis2.1
bootflash swlog_chassis2.2
capManCmmTrace swlog_chassis2.3
capManNiTrace swlog_chassis2.4
certified swlog_chassis2.5
diags swlog_chassis2.6
eeprom swlog_chassis4
externalCPU swlog_chassis4.0
foss swlog_chassis4.1
fpga_name swlog_chassis4.2
hwinfo swlog_chassis4.3
issu system
lost+found tech_support.log
network u-boot.8.1.1.R01.462.tar.gz
pmd u-boot.8.1.1.R01.70.tar.gz
switch u-boot_copy
swlog vcboot.cfg.13.err
swlog_chassis1 vcboot.cfg.14.err
swlog_chassis2 working
swlog_chassis2.0
SHASTA #->

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 8 of 148
2. Troubleshooting the Switch System
In order to troubleshoot the system, a basic understanding of the operation of Chassis Management
Modules (CMMs) and their interaction with Network Interface (NI) modules is required. Some concepts are
covered in this chapter:
Understanding of the “Diagnosing Switch Problems” chapter in the appropriate OmniSwitch Switch
Management Guide.
Understanding of the “Using Switch Logging” from the appropriate OmniSwitch Network
Configuration Guide is highly recommended.
Summary of the commands in this chapter is listed here:
_________________________________________________________
show module status
show powersupply
show health
show health all cpu
show health configuration
show swlog
show log swlog
top
top -b -n 1 -m | head
debug qos internal "chassis 1 slot 1 list 1 verbose"
cat /proc/pktdrv
_________________________________________________________
2.1. Introduction
The CMM is the Management Module of the switch. All of the critical operations of the switch including the
monitoring is the responsibility of the CMM. CMM not only provides monitoring but also CMM synchronizes
all of the NI for different operations.
2.2. Troubleshooting the System on OS6900/OS10K Switches
1. To troubleshoot system problems, the first step is to check the condition of all switch modules.
-> show module status
Operational Firmware
Slot Status Admin-Status Rev MAC
------+-------------+------------+---------+------------------
CMM-B UP POWER ON 2.0 e8:e7:32:9b:e2:6e
SLOT-1 UP POWER ON 2.12 00:e0:b1:e4:c5:79
SLOT-2 UP POWER ON 0.3 e8:e7:32:a5:d9:30
SLOT-5 UP POWER ON 0.7 00:e0:b1:e4:ae:a1
-> show powersupply
Total Power Input PS
Slot PS Power Used Voltage Type Status Location
---------+---------+--------+---------+--------+--------+-----------
1 1200 16 120 AC PWRSAVE Internal

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 9 of 148
2 1200 588 119 AC UP Internal
3 1200 368 123 AC UP Internal
4 1200 24 123 AC UP Internal
2. Check the CPU and memory status.
-> show health
CMM Current 1 Min 1 Hr 1 Day
Resources Avg Avg Avg
----------------------+---------+-------+-------+-------
CPU 1 1 1 1
Memory 22 22 22 22
Use the command “show health all cpu”, to examine the CPU on all modules.
-> show health all cpu
CPU Current 1 Min 1 Hr 1 Day
Avg Avg Avg
-------------------+----------+--------+-------+--------
Slot 1 11 11 11 10
Slot 2 8 8 7 7
Slot 5 5 5 5 5
3. Use the command “show log swlog” to check the output of the switch’s log files.
-> show log swlog
/flash/swlog_CMMB.7 not found!
/flash/swlog_CMMB.6 not found!
Displaying file contents for '/flash/swlog_CMMB.5'
Apr 28 20:26:48 (none) syslog.info syslogd started: BusyBox v1.19.3
Apr 28 20:26:48 (none) user.notice kernel: klogd started: BusyBox v1.19.3 (2013-03-18 16:33:36
PDT)
…..
2.3. Troubleshooting System on OS6860(E) Switches
If the switch is having problems the first place to look for is the CMM. All tasks are supervised by the CMM.
Any inconsistencies between the CMM and the NI can cause problems.
1 The first step for troubleshooting problems with the switch is to look at the overall health of the switch.
Verify that all of the modules in the chassis are up and operational, using the command:
LAB-6860> show module status
Operational
Chassis/Slot Status Admin-Status MAC
--------------+-------------+------------+------------------
1/CMM-A UP POWER ON e8:e7:32:ae:78:11
1/SLOT-1 UP POWER ON e8:e7:32:ae:78:18
The operational status can be DOWN while the power status is ON, indicating a possible software issue. For
the CMM, the base chassis MAC address is displayed. For NI modules, the MAC address for the
corresponding NI is displayed.
2 Verify the power supply (or supplies).
Check the power supply status, using the command:
sno-lab-r1-6860> show powersupply

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 10 of 148
Total PS
Chassis/PS Power Type Status Location
-----------+---------+--------+--------+-----------
1/1 150 AC UP Internal
Total 150
sno-lab-r1-6860> show powersupply 1
Module in slot PS-1
Model Name: PS-150AC,
Module Type: 0x6040102,
Description: AC-PS,
Part Number: 903400-90,
Hardware Revision: A04,
Serial Number: 1335000462,
Manufacture Date: Sep 2 2013,
Operational Status: UP,
Power Provision: 150W
Make sure that all the known good power supplies are operational.
3 Verify the CPU utilization.
The CPU utilization of CMM can be viewed by using the command:
LAB-6860> show health
CMM Current 1 Min 1 Hr 1 Day
Resources Avg Avg Avg
----------------------+---------+-------+-------+-------
CPU 7 7 7 7
Memory 61 61 56 56
The above command shows the memory, CPU statistics for current, 1 minimum average, 1 hour average and 1
hour maximum.
Check the threshold in health configuration, using command:
sno-lab-r1-6860> show health configuration
Rx Threshold = 80,
TxRx Threshold = 80,
CPU Threshold = 80,
Memory Threshold = 80,
Sampling Interval (Secs) = 10
All the values should be within the threshold. Any value above the threshold indicates abnormal behavior. The
1 hour average might be high if the switch was booted whithin the last hour but should normalize during the
first hour of operation.
If none of the values are above the threshold, the next step is to attempt to isolate the problem to a particular
NI, using command:
6860-> show health slot 1/1
Slot 1/ 1 Current 1 Min 1 Hr 1 Day
Resources Avg Avg Avg
----------------------+---------+-------+-------+-------
CPU 8 7 7 6
Memory 61 61 59 57
Receive 0 0 0 0
Receive/Transmit 0 0 0 0

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 11 of 148
6860-> show health port 1/1/1
Port 1/ 1/ 1 Current 1 Min 1 Hr 1 Day
Resources Avg Avg Avg
----------------------+---------+-------+-------+-------
Receive 0 0 0 0
Receive/Transmit 0 0 0 0
The above commands may help to narrow the problem to a particular NI or to the CMM. For more detail see,
the section “Monitoring Switch Health” in the chapter titled “Diagnosing Switch Problems” in the appropriate
OmniSwitch Network Configuration Guide.
4 Check the switch log.
One of the most important things to check is the switch log. The switch log contains the log events based on
the settings of the log levels and applications configured to generate log events messages. Default settings of
the switch log can be view using the command:
sno-lab-r1-6860> show swlog
Operational Status : Running,
File Size per file : 12500K bytes,
Log Device : console flash socket,
Log Device : ipaddr 192.168.2.131 remote command-log,
Syslog FacilityID : local7(23),
Hash Table entries age limit : 60 seconds,
Switch Log Preamble : Enabled,
Switch Log Debug : Disabled,
Switch Log Duplicate Detection : Disabled,
Console Display Level : info
By default, the chassis is set to log to flash and console. This can be changed and specific SYSLOG servers
can be used to log the messages, please refer to the Switch Management Guide for further details. The default
application trace level is ‘info’. Any error messages or informational messages would be logged in the switch
log.
The switch log should be viewed to see if any error messages were generated by the switch. The command to
use is:
sno-lab-r1-6860> show log swlog
/flash/swlog_chassis1.7 not found!
/flash/swlog_chassis1.6 not found!
/flash/swlog_chassis1.5 not found!
/flash/swlog_chassis1.4 not found!
/flash/swlog_chassis1.3 not found!
/flash/swlog_chassis1.2 not found!
Displaying file contents for '/flash/swlog_chassis1.1'
Jan 1 00:00:32 OS6860 syslogd started: BusyBox v1.19.3
Jan 1 00:00:32 OS6860 kernel: klogd started: BusyBox v1.19.3 (2014-05-14 02:47:33 PDT)
Jan 1 00:00:32 OS6860 kernel: [ 0.000000] Booting Linux on physical CPU 0
Jan 1 00:00:32 OS6860 kernel: [ 0.000000] Linux version 3.6.5
…….
If the log messages do not show enough information then they can be changed for specific applications to a
higher log level or for all the applications running in the switch. For setting up different log levels in switch
log, please refer to the “Using Switch Logging” chapter in the appropriate OmniSwitch Network Configuration
Guide.

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 12 of 148
2.4. Advanced Troubleshooting
Troubleshooting High CPU utilization
1. First, identify which CPU is excessively high.
CLI shell:
6860->show health
6860->show health all cpu
Maintenance shell( AOS7 & 8 are Linux based, typing su in CLI will lead to Linux BASH shell):
Top
In the maintenance shell the “top” command is used to continuously monitor the tasks consuming
CPU.
->top
Mem: 1172224K used, 849676K free, 0K shrd, 548K buff, 742380K cached
CPU: 0.0% usr 4.5% sys 0.0% nic 95.4% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 0.23 0.19 0.15 2/237 3857
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
3857 3854 root R 2940 0.1 0 4.5 top
2090 813 root S 301m 15.2 0 0.0 /bin/ipnid
2022 813 root S 299m 15.1 1 0.0 /bin/bcmd -p
2081 813 root S 296m 14.9 1 0.0 /bin/slNi
2105 813 root S 277m 14.0 0 0.0 /bin/ipmsni
2102 813 root S 277m 14.0 0 0.0 /bin/qosnid
1749 813 root S 276m 14.0 0 0.0 /bin/qoscmmd
2069 813 root S 268m 13.6 0 0.0 /bin/stpNi
2063 813 root S 266m 13.4 0 0.0 /bin/lacpNi
2. The most common causes for high CPU utilization:
i. An abnormal process
A process goes into an infinite loop. This is probably a software issue.
A process is doing extensive calculations. It is possible that the network is not well
scaled.
AOS is under a DoS attack.
ii. Abnormal traffic
Too many messages exchanged between AOS subsystems. Examples include: extensive
logging, MAC learning, malfunctioning bus, HW interrupts.
Too many frames or packets are trapped to CPU
Proc
Task
Proc
Task
aaaCmm
AAA
mvrpNi
MVRP NI
agCmm
Access Guardian CMM
ntpd
NetworkTime Protocol
agNi
Access Guardian NI
ofcmmd
Open Flow CMM
appMonCmm
app-mon
ofnid
Open Flow NI
appMonNiStub
app-mon
pmCmm
Port mapping CMM
bcd
Broadcom driver
pmNi
Port mapping NI
bcmd
BCM
pmmCmm
Port Mirroring & Monitoring
CMM
bfd
BFD CMM
pmmnid
Port Mirroring & Monitoring Ni

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 13 of 148
3. Identify the process causing high CPU usage
Use the commands “top” and “ps” in the maintenance shell to find the process(es) consuming the
most CPU and use knowledge of the network to determine whether or not the consumption is
abnormal.
Each process presents a task running in CPU. The meaning of process can be found in above figure.
4. Resolve the process causing high CPU
bfdni
BFD NI
portmgrcmm
Port manager CMM
capmanc
Capability Manager CMM
portmgrni
Port manager Ni
capmani
Capability Manager NI
qmrCmm
Quarantine Manager
dhcpsrv
DHCP
qosCmmd
QOS
dpiCmm
DPI
radCli
RADIUS client
dpiNi
DPI
remcfgMgr
Remote Config
eoamCmm
OAM
rmon
Remote Monitoring
eoamNi
OAM
saaCmm
Service Assurance Agent
erpCmm
ERP
sesmgrCmm
Session Manager
erpNi
ERP
sipCmm
SIP Snooping CMM
evbCmm
Edge Virtual Bridging
CMM
sipNi
SIP Snooping Ni
evbNi
Edge Virtual Bridging NI
slCmm
SRC-LEARNING CMM
flashMgr
flash
slNi
SRC-LEARNING Ni
havlanCmm
vlan
slbcmmd
SLB
hmonCmm
Health CMM
stpCmm
STP CMM
hmonNi
Health NI
stpNi
STP NI
ipcmmd
IP
svcCmm
Service Manager (useb by SPB,
MPLS/VPLS)
ipmscmm
IPMS CMM
tacClientCmm
TACACS
ipmsni
IPMS NI
trapmgr
TRAP
ipnid
ARP
udldCmm
UDLD CMM
iprm
IP Routing
udldNi
UDLD NI
ipsec6d
IPSEC
udpRelayCmmd
UDP Relay
isis
isis
udpRelayNi
DHCP relay/snooping
isisVc
Virtual-Chassis
vcmCmm
Virtual Chassis Manager CMM
lacpNi
LACP
vcmNi
Virtual Chassis Manager Ni
lagCmm
Link Agg
vcspCmm
Virtual Chassis Split Protection
CMM
ldapClientCmm
LDAP
vcspNi
Virtual Chassis Split Protection Ni
lldpCmm
LLDP CMM
vfcm
Virtual Flow Controller CMM
lldpNi
LLDP NI
vfcn
Virtual Flow Controller Ni
loamNi
Link OAM
vmCmm
Vlan Manager CMM
lpCmm
Learned Port Security
CMM
vmNi
Vlan Manager Ni
lpNi
Learned Port Security NI
vrrp
VRRP
mcipcd
Multi-Chassis IPC
vstkCmm
VLAN Stacking (Q-in-Q) CMM
mipgwd
Gateway software
vstkNi
VLAN Stacking (Q-in-Q) Ni
mvrpCMM
MVRP CMM
webMgtd
WEBVIEW

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 14 of 148
Find the specific chapter in this troubleshooting guide for the task causing high CPU. It is possible
that the task needs to be restarted. Before restarting the task, please contact Alcatel-Lucent
customer support.
5. Identify abnormal traffic
It's worthwhile to verify how many packets are trapped by the CPU due to FFP rules (packets may be
trapped to the CPU due to other reasons—see chapter Packet Driver for more details). The following
commands will be helpful to locate abnormal traffic by showing the type and number of packets.
 debug qos internal "chassis 1 slot 1 list 1 verbose"
Entry U Slice CIDU CIDL MIDU MIDL TCAM Count[+] Green[+] Red[+] NotGreen[+]
List 1: 41 entries set up
HgMcastARP(16) 0 14 - 3642 - - 3684 18656[18656] 0[0] 0[0] 0[0]
HgMcastARP(16) 0 15 - - - - 3940 18656[0] 0[0] 0[0] 0[0]
McastARP(17) 0 14 3591 - - - 3685 0[0] 0[0] 0[0] 0[0]
McastARP(17) 0 15 - - - - 3941 0[0] 0[0] 0[0] 0[0]
ISIS_BPDU1(22) 0 14 - 3584 -- 3639 0[0] 0[0] 0[0] 0[0]
ISIS_BPDU1(22) 0 15 - - - - 3895 0[0] 0[0] 0[0] 0[0]
ISIS_BPDU2(23) 0 14 3585- - - 3640 0[0] 0[0] 0[0] 0[0]
2.5. Memory Leak
The following output shows memory utilization by process and can be helpful in determing which task is
consuming memory, or if memory for a specific task is gradually increasing in the case of a memory leak.
SHASTA #-> top -b -n 1 -m | head
Mem total:2021900 anon:306672 map:639136 free:846800
slab:35708 buf:572 cache:743544 dirty:4 write:0
Swap total:0 free:0
PID VSZ^VSZRW RSS (SHR) DIRTY (SHR) STACK COMMAND
2022 161m 135m 81244 10004 81228 9992 132 /bin/bcmd -p
1856 92640 83368 8108 7296 8108 7296 132 /bin/dhcpv6srv
1868 91796 83240 8108 7276 8108 7276 132 /bin/dhcpsrv
2102 40620 22688 29864 10196 29844 10180 132 /bin/qosnid
2081 38956 20208 19468 10452 19444 10432 132 /bin/slNi
2105 32660 15884 22692 9888 22652 9856 132 /bin/ipmsni
2.6. Packet Driver
The packet driver is responsible for filtering and classifying all frames and packets trapped to the CPU.
SHASTA #-> cat /proc/pktdrv
Board Type : 0x6062202
Tb : 4fa13fbd8f93d
Wakeups : 26647195 26647194
Last Task Wakeup : 0 0
Last Task Ran : 4fa13fbd8ae6c 8903c213fe
Task Latency : 3b fffb067507e965cd
Task Max Latency : fffffffffffffffc fffb067507e9658e
TX Chain Ints : 24887887 24887884
TX Desc Ints : 0 0
TX Timeout Ints : 0 0
…..
Buffer States
Invalid : 1
Free : 1947
Classify : 0

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 15 of 148
RX Dma : 1755
TX Dma : 0
……..
Task Info
Ipv4: [flags=1] [id 1] [ring size 100] [head 18] [tail 18]
Arp : [flags=1] [id 2] [ring size 256] [head 145] [tail 145]
Ip4t: [flags=1] [id 3] [ring size 100] [head 0] [tail 0]
………..
Classify Queues
Ipv4: Tx : 7645246 7645246
Ipv4: Rx Q Full Drops : 135 135
Ipv4: Rx : 7640318 7640318
………..
2.7. The second column of output provides changes since the
previous execution of the command.Logs
Besides the swlog, the kernel log in /var/log/.

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 16 of 148
3. Troubleshooting Virtual Chassis
3.1. Introduction
A Virtual Chassis is a group of switches managed through a single management IP address that operates as a
single bridge and router. It provides both node level and link level redundancy for layer 2 and layer 3 services
and protocols acting as a single device. The use of a virtual chassis provides node level redundancy without the
need to use redundancy protocols such as STP and VRRP between the edge and the aggregation/core layer.
Some of the key benefits provided by Virtual Chassis are:
A single, simplified configuration to maintain
Optimized bandwidth usage between the access layer and core
Active-Active multi-homed link aggregation
Provides predictable and consistent convergence with redundant links to the two switches
Allows for exclusion of spanning-tree and other redundancy protocols like VRRP between the access
layer and the core
A Virtual Chassis appears as single router or bridge with support for all protocols
A Virtual Chassis can be upgraded using ISSU to minimize network impact
Summary of the commands in this chapter is listed here:
___________________________________________________________
show virtual-chassis topology
show virtual-chassis consistency
show virtual-chassis vf-link
show virtual-chassis vf-link member-port
show virtual-chassis chassis-reset-list
show running-directory
show interfaces status
show log swlog | grep vcmCmm
debug show virtual-chassis topology
debug show virtual-chassis status
cat vc_debug.txt
____________________________________________________________
Master Chassis Election :
The learning window is 30 seconds after VFL comes up
Master chassis election is based on:
Highest chassis priority
-> virtual-chassis configured-chassis-priority 100
The highest number configured-chassis-priority will become the Master chassis.
Without setting this value the lowest chassis identifier becomes the key value used to determine which switch
will become the Master.

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 17 of 148
Longest Chassis uptime
Lowest Chassis ID
Lowest Chassis MAC Address
In OS6860/OS6860E auto VC, once the Master is elected the chassis connecting the lower VFL port of the
master is assigned Chassis 2 and the value increments for each additional chassis up to chassis 8.
Virtual Chassis - Boot-Up
The Master chassis contains the vcboot.cfg file that details the configuration for the entire virtual chassis. All
the switches (i.e. the one that will eventually become the Master and the ones that will become Slaves) contain
a vcsetup.cfg file that allows them to establish an initial connection over a VFL to all the other neighboring
switches.
1. Upon boot-up, a switch will read its local vcsetup.cfg file and attempt to connect to the other neighbor
switches
2. Upon connection, the switches will exchange the parameters configured in their local vcsetup.cfg files
3. As a result of this exchange, they will discover the topology, elect a Master based on criteria described
in the next section, start periodic health checks over the VFL, and synchronize their configuration as
defined within the vcboot.cfg configuration file
4. All Slaves, if they do not have a local copy of vcboot.cfg, or if their local copy does not match the
copy found on the Master, will download the vcboot.cfg from the Master chassis and reboot using this
copy of vcboot.cfg as its configuration file
Startup Error Mode
If a switch is unable to successfully come up in virtual chassis mode, it enters a special fallback mode called
start up error mode. A switch moves to start up error mode if either one of the following conditions occur:
The vcsetup.cfg and vcboot.cfg configuration files are present in the running directory, but no valid
advanced license is installed on the switch.
The vcsetup.cfg file is corrupted or edited in such a way that it is unable to read a valid chassis identifier
in the appropriate range.
A switch start up error mode will keep all of its front-panel user ports, including the virtual-fabric links
member ports disabled. This mode can be identified on the switch by using the show virtual-chassis topology
command. The chassis role will display Inconsistent, whereas the chassis status will show either one of the
following values:
Invalid-Chassis-Id: The chassis is not operational in virtual chassis mode because no valid chassis
identifier has been found in the configuration. Typically this means that the vcsetup.cfg file is corrupted,
empty or contains an invalid (e.g. out of range) chassis identifier.

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 18 of 148
Invalid-License: The chassis is not operational in virtual chassis mode because no valid advanced license
has been found.
3.2. Basic Troubleshooting
This command is used to provide a detailed status of the virtual chassis topology.
OS6860-> show virtual-chassis topology
Local Chassis: 1
Oper Config Oper
Chas Role Status Chas ID Pri Group MAC-Address
-----+------------+-------------------+--------+-----+------+------------------
1 Master Running 1 100 0 e8:e7:32:b3:3c:3b
2 Slave Running 2 100 0 e8:e7:32:b3:49:11
This command is used to provide a detailed status of the parameters taken into account to determine the
consistency of a group of switches participating in the virtual chassis topology.
OS6860-> show virtual-chassis consistency
Legend: * - denotes mandatory consistency which will affect chassis status
licenses-info - A: Advanced; B: Data Center;
Config Oper Oper Config
Chas Chas Chas Hello Control Control
Chas* ID Status Type* Group* Interv Vlan* Vlan License*
------+------+---------+-------+------+-------+--------+--------+----------
1 1 OK OS6860 0 15 4094 4094
2 2 OK OS6860 0 15 4094 4094
A more detailed version of this output is available after adding the chassis-id option
OS6860-> show virtual-chassis chassis-id 1 consistency
Legend: * - denotes mandatory consistency which will affect chassis status
licenses-info - A: Advanced; B: Data Center;
Given Master
Consistency Chassis Chassis Status
---------------------+-----------+-----------+---------
Chassis-ID* 1 1 OK
Config-Chassis-ID 1 1 OK
Chassis-Type* OS6860 OS6860 OK
License* OK
Chassis-Group* 0 0 OK
Hello-Interval 15 15 OK
Oper-Control-Vlan* 4094 4094 OK
Config-Control-Vlan 4094 4094 OK
Displays a summary of the configured and operational parameters related to the virtual fabric links on the
virtual chassis topology
OS6860-> show virtual-chassis vf-link
Primary Config Active Def Speed
Chassis/VFLink ID Oper Port Port Port Vlan Type
-------------------+----------+---------+-------+-------+---------+-----------
1/0 Up 1/1/30 2 2 1 21G
2/0 Up 2/1/30 2 2 1 21G
And per link

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 19 of 148
OS6860-> show virtual-chassis vf-link member-port
Chassis/VFLink ID Chassis/Slot/Port Oper Is Primary
-------------------+------------------+----------+-------------
1/0 1/1/29 Up No
1/0 1/1/30 Up Yes
2/0 2/1/29 Up No
2/0 2/1/30 Up Yes
This command displays the list of all chassis that must be reset along with a specified chassis in order to
prevent a virtual chassis topology split
OS6860-> show virtual-chassis chassis-reset-list
Chas Chassis reset list
-----+---------------------
1 1,
2 2,
Display Virtual Chassis logs from SWLOG
-> show log swlog | grep vcmCmm
3.3. Advanced Troubleshooting
The below command lets the user know whether the unit has reached the ready state or not.
OS6860-> debug show virtual-chassis topology
Local Chassis: 1
Oper Config Oper System
Chas Role Status Chas ID Pri Group MAC-Address Ready
-----+------------+-------------------+--------+-----+------+------------------+-------
1 Master Running 1 100 0 e8:e7:32:b3:3c:3b Yes
2 Slave Running 2 100 0 e8:e7:32:b3:49:11 Yes
Warning:
"RCD Operational Status" is "Up" even in case there is a duplicated chassis group ID in the same
network!
The below command lets the user know on which state the VC has failed :
OS6860-> debug show virtual-chassis status
ID Level Parameter Value Timestamp Status
----+------+-----------------------------+----------------+-----------+---------
0 L0 Chassis Identifier 1 01:03:57 OK
1 L0 Designated NI Module 1 01:03:57 OK
2 L0 Designated NI Module (@L5) 1 00:40:45 OK
3 L0 License Configured Yes 01:03:57 OK
4 L0 License Configured (@L5) Yes 00:40:45 OK
5 L0 VFL Links Configured 2 01:03:57 OK
6 L0 VFL Links Configured (@L5) 2 00:40:45 OK
7 L0 VFL Ports Configured 2 01:03:57 NOK_08
8 L0 VFL Ports Configured (@L5) 2 00:40:45 OK
11 L0 Chassis Ready Received Yes 00:40:38 OK
12 L1 VFL Intf Oper Status Down 01:03:57 NOK_09
13 L1 VFL Intf Oper Status (@L5) Down 00:40:45 NOK_09
14 L2 VFL LACP Status Down 01:03:57 NOK_14
15 L2 VFL LACP Status (@L5) Down 00:40:45 NOK_14
16 L2 VFL LACP Up -> Down 3 00:46:05 INFO_04

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 20 of 148
17 L2 VFL LACP Down -> Up 4 00:45:56 INFO_03
18 L3 VCM Protocol Role (@L5) Slave 00:40:45 OK
19 L3 VCM Protocol Role Master 01:03:57 OK
20 L3 VCM Protocol Status (@L5) Running 00:40:45 OK
21 L3 VCM Protocol Status Running 01:03:57 OK
24 L4 VCM Connection Up 01:03:57 OK
25 L4 VCM Connection (@L5) Up 00:40:45 OK
26 L5 VCM Synchronization Multi-node 01:03:57 OK
27 L6 Chassis Sup Connection Up 00:46:00 OK
28 L6 Remote Flash Mounted Yes 00:46:26 OK
29 L6 Image and Config Checked Yes 00:41:01 OK
30 L6 VC Takeover Sent Yes 00:42:55 OK
31 L7 VC Takeover Acknowledged Yes 00:42:58 OK
32 L8 System Ready Received Yes 00:41:06 OK
33 L8 RCD Operational Status N/A N/A N/A
34 L8 RCD IP Address N/A N/A N/A
Error/Information Codes Detected:
--------------------------------------------------------------------------------
NOK_08
There are no virtual-fabric member ports configured on this switch.
If there are multiple virtual-fabric links configured, we must have
at least one member port configured or assigned to each of the
virtual-fabric links.
Troubleshooting Tips:
-> show virtual-chassis vf-link member-port | grep "<local-chassis-id>/"
NOK_09
There are no virtual-fabric member interfaces operationally up.
If there are multiple virtual-fabric links configured, we must have
at least one member port interface up on each virtual-fabric link.
Troubleshooting Tips:
-> show virtual-chassis vf-link member-port | grep "<local-chassis-id>/"
-> show interfaces port <chassis>/<slot>/<port> status
NOK_09
There are no virtual-fabric member interfaces operationally up.
If there are multiple virtual-fabric links configured, we must have
at least one member port interface up on each virtual-fabric link.
Troubleshooting Tips:
-> show virtual-chassis vf-link member-port | grep "<local-chassis-id>/"
-> show interfaces port <chassis>/<slot>/<port> status
NOK_14
The virtual-fabric links configured on this switch are not operationally up.
If there are multiple links configured, all of them must be operationally up
in order for this parameter to be reported as OK.
Troubleshooting Tips:
-> show virtual-chassis vf-link | grep "<local-chassis-id>/"
NOK_14
The virtual-fabric links configured on this switch are not operationally up.
If there are multiple links configured, all of them must be operationally up
in order for this parameter to be reported as OK.
Troubleshooting Tips:
-> show virtual-chassis vf-link | grep "<local-chassis-id>/"
INFO_04
This parameter provides the counter of how many times the
virtual-fabric link operational status has transitioned from
up to down since the switch was started. Depending on the specific
operations you are performing on the system, this counter may
increase. Under normal conditions this counter should ideally
be one unit smaller than the counter for the opposite transitions
from operational down to up
INFO_03
This parameter provides the counter of how many times the
virtual-fabric link operational status has transitioned from
down to up since the switch was started. Depending on the specific
operations you are performing on the system, this counter may