Extreme Networks ExtremeWare Version 7.8 User manual

Extreme Networks, Inc.
3585 Monroe Street
Santa Clara, California 95051
[phone]
http://www.extremenetworks.com
Advanced System
Diagnostics and
Troubleshooting Guide
ExtremeWare Software Version 7.8
Published: May 2008
Part number: 100279-00 Rev 01

2
©2005 Extreme Networks, Inc. All rights reserved. Extreme Networks, ExtremeWare, Alpine, and BlackDiamond are
registered trademarks of Extreme Networks, Inc. in the United States and certain other jurisdictions. ExtremeWare Vista,
ExtremeWorks, ExtremeAssist, ExtremeAssist1, ExtremeAssist2, PartnerAssist, EPICenter, Extreme Standby Router
Protocol, ESRP, SmartTraps, Summit, Summit1i, Summit5i, Summit7i,Summit48i, Summit48si, SummitPx, Summit 200,
Summit Virtual Chassis, SummitLink, SummitGbX, SummitRPS and the Extreme Networks logo are trademarks of
Extreme Networks, Inc., which may be registered or pending registration in certain jurisdictions. The Extreme
Turbodrive logo is a service mark of Extreme Networks, which may be registered or pending registration in certain
jurisdictions. Specifications are subject to change without notice.
NetWare and Novell are registered trademarks of Novell, Inc. Solaris is a trademark of Sun Microsystems, Inc. F5,
BIG/ip, and 3DNS are registered trademarks of F5 Networks, Inc. see/IT is a trademark of F5 Networks, Inc.
“Data Fellows”, the triangle symbol, and Data Fellows product names and symbols/logos are
trademarks of Data Fellows.
F-Secure SSH is a registered trademark of Data Fellows.
All other registered trademarks, trademarks and service marks are property of their respective owners.

Advanced System Diagnostics and Troubleshooting Guide 3
Contents
Preface
Introduction 9
Terminology 9
Conventions 9
Related Publications 10
Chapter 1 Introduction
Introduction 11
Diagnostics: A Brief Historical Perspective 12
Overview of the ExtremeWare Diagnostics Suite 12
Supported Hardware 13
Applicable ExtremeWare Versions 13
Chapter 2“ i ” Series Switch Hardware Architecture
Diagnostics Support 15
The BlackDiamond Systems 16
BlackDiamond 6800 Series Hardware Architecture Differences 16
The BlackDiamond Backplane 17
BlackDiamond I/O Modules 18
Management Switch Modules 19
BlackDiamond MSM Redundancy 20
Causes of MSM Failover and System Behavior 20
Alpine Systems 22
Summit “i” Series Systems 23
Chapter 3 Packet Errors and Packet Error Detection
Overview 25

4Advanced System Diagnostics and Troubleshooting Guide
Contents
Definition of Terms 26
Standard Ethernet Detection for Packet Errors on the Wire 27
Extreme Networks’ Complementary Detection of Packet Errors Between Wires 27
Hardware System Detection Mechanisms 28
Software System Detection Mechanisms 29
Failure Modes 30
Transient Failures 30
Systematic Failures 30
Soft-State Failures 30
Permanent Failures 31
Responding to Reported Failures 31
Health Check Messages 33
Alert Messages 33
Checksum Error Messages: 34
Corrective Behavior Messages 35
Chapter 4 Software Exception Handling
Overview of Software Exception Handling Features 37
System Watchdog Behavior 37
System Software Exception Recovery Behavior 38
Redundant MSM Behavior 38
Configuring System Recovery Actions 40
Related Commands 40
Configuring System Recovery Actions on “i” Series Switches 40
Configuring System Recovery Actions on “e” Series Switches 41
Usage Notes 41
Configuring Reboot Loop Protection 43
Related Commands 43
Configuring Reboot Loop Protection 43
Dumping the “i” Series Switch System Memory 45
Related Commands 45
Configuring an Automatic System Dump During System Recovery 45
Initiating a Manual System Dump 46
Example Log for a Software Exception 47
Chapter 5 Diagnostics
Diagnostic Test Functionality 49
How Diagnostic Tests are Run 49
How the Test Affects the Switch 50
System Health Checks: A Diagnostics Suite 52
Diagnostic Suite Components 52
The Role of Memory Scanning and Memory Mapping 53
Modes of Operation 54

Advanced System Diagnostics and Troubleshooting Guide 5
Contents
The Role of Processes to Monitor System Operation 55
Power On Self Test (POST) 56
Related Commands 56
Configuring the Boot-Up Diagnostics 56
Runtime (On-Demand) System Diagnostics 57
Runtime Diagnostics on “i” Series Systems 57
Related Commands 58
Running the Diagnostics on BlackDiamond Systems 58
Runtime Diagnostics on “i” Series Alpine and Summit Systems 58
System Impact of Running the Diagnostics on “i” Series Switches 59
Runtime Diagnostics on “e” Series Systems 59
Related Commands 60
Running the Diagnostics on Summit “e” Switches 60
System Impact of Running the Diagnostics on Summit “e” Series Switches 60
Automatic Packet Memory Scan (via sys-health-check) 60
Memory Scanning and Memory Mapping Behavior 61
Limited Operation Mode 64
Effects of Running Memory Scanning on “i” Series Switches 64
Summit, Alpine, or BlackDiamond with a Single MSM 64
BlackDiamond System with Two MSMs 64
Interpreting Memory Scanning Results 66
Per-Slot Packet Memory Scan on BlackDiamond Switches 67
Related Commands 67
Configuring the Packet Memory Scan Recovery Mode 67
System Impact of Per-Slot Packet Memory Scanning 68
Network Impact of Per-Slot Packet Memory Scanning 68
System (CPU and Backplane) Health Check 70
Health Check Packet Types 70
Backplane Health Check States 70
Related Commands 71
Health Check Functionality 71
Alarm-Level Response Action 71
Auto-Recovery Response Action 71
Backplane Health Check 72
Viewing Backplane Health Check Results—show log Command 72
Viewing Backplane Health Check Diagnostic Results—show diagnostics Command 73
Analyzing the Results 78
CPU Health Check 79
Viewing CPU Health Check Results—show log Command 79
Viewing CPU Health Check Diagnostic Results—show diagnostics Command 80
Analyzing the CPU Health Check Results 80
Transceiver Diagnostics 82
Usage Guidelines 82
Related Commands 82
Configuring the Transceiver Diagnostics 82

6Advanced System Diagnostics and Troubleshooting Guide
Contents
System Impacts of the Transceiver Diagnostics 83
Network Impact of the Transceiver Diagnostics 83
Viewing Diagnostics Results 84
Example Log Messages for Transceiver Diagnostic Failures 84
Examples, show diagnostics Command 85
Example—show switch Command 86
Transceiver Diagnostic Result Analysis 87
FDB Scan 88
Usage Guidelines 88
Related Commands 89
Enabling FDB Scanning 89
Disabling FDB Scanning 89
Configuring the FDB Scan Diagnostics 90
System Impact of the FDB Scan Diagnostic 90
Network Impact of the FDB Scan Diagnostic 90
Viewing Diagnostics Results 91
Example Log Messages for FDB Scan Diagnostic Failures 91
Example FDB Scan Results from the show diagnostics Command 91
Example Output from the show switch command 92
Example Output from the show fdb remap Command 92
Chapter 6 Additional Diagnostics Tools
Temperature Logging for Modular Switches 93
Related Commands 93
System Impacts of Temperature Logging 94
Network Impact of Temperature Logging 94
Syslog Servers 94
Related Commands 94
Enabling Logging to Remote Syslog Server Targets 94
Disabling Logging to Remote Syslog Server Targets 95
Adding a Syslog Server 95
Deleting a Remote Syslog Server 95
System Impact of the Syslog Server Facility 95
Network Impact of the Syslog Server Facility 96
Cable Diagnostics 96
Cable Diagnostics for the High-Density Gigabit Ethernet I/O Modules in “i” Series
Switches 96
Related Commands 96
Running Cable Diagnostics 97
Viewing and Interpreting CDM Test Data 98
Cable Diagnostics for “e” Series Switches 101
Chapter 7 Troubleshooting Guidelines
Contacting Extreme Technical Support 103
Americas TAC 103

Advanced System Diagnostics and Troubleshooting Guide 7
Contents
Asia TAC 104
EMEA TAC 104
Japan TAC 104
What Information Should You Collect? 105
Analyzing Data 105
Diagnostic Troubleshooting 106
Extreme Networks’ Recommendations 107
Using Memory Scanning to Screen I/O Modules 109
Appendix A Limited Operation Mode and Minimal Operation Mode
Limited Operation Mode 111
Triggering Limited Operation Mode 112
Bringing a Switch Out of Limited Operation Mode 112
Minimal Operation Mode 112
Triggering Minimal Operation Mode 112
Bringing a Switch Out of Minimal Operation Mode 113
Appendix B Reference Documents
General Information 115
Other Documentation Resources 115
Index
Index of Commands

8Advanced System Diagnostics and Troubleshooting Guide
Contents

Advanced System Diagnostics and Troubleshooting Guide 9
Preface
This Preface provides an overview of this guide, describes guide conventions, and lists other
publications that might be useful.
Introduction
This guide describes how to use the ExtremeWare hardware diagnostics suite to test and validate the
operating integrity of Extreme Networks switches. The tools in the diagnostic suite are used to detect,
isolate, and treat faults in a system.
This guide is intended for use by network designers, planners, and operations staff.
Terminology
When features, functionality, or operation is specific to a modular or stand-alone switch family, the
family name is used. Explanations about features and operations that are the same across all product
families simply refer to the product as the “switch.”
Conventions
Table 1 and Table 2 list conventions that are used throughout this guide.
Table 1: Notice Icons
Icon Notice Type Alerts you to...
Note Important features or instructions.
Caution Risk of personal injury, system damage, or loss of data.
Warning Risk of severe personal injury.

10 Advanced System Diagnostics and Troubleshooting Guide
Preface
Related Publications
The publications related to this one are:
•ExtremeWare Software User Guide, Software Version 7.7.
•ExtremeWare Software Command Reference, Software Version 7.7.
•ExtremeWare Error Message Decoder.
Documentation for Extreme Networks products is available on the World Wide Web at the following
location:
http://www.extremenetworks.com/services/documentation/Default.asp
Table 2: Text Conventions
Convention Description
Screen displays This typeface indicates command syntax, or represents information as it appears on the
screen.
The words “enter”
and “type” When you see the word “enter” in this guide, you must type something, and then press the Return or Enter key. Do not press the Return or Enter key when an instruction
simply says “type.”
[Key] names Key names are written with brackets, such as [Return] or [Esc].
If you must press two or more keys simultaneously, the key names are linked with a
plus sign (+). Example:
Press [Ctrl]+[Alt]+[Del].
Letter in bold type Letters within a command that appear in bold type indicate the keyboard shortcut for a
command. When entering the command, you can use just the bolded letters instead of
the entire word.
Words in italicized type Italics emphasize a point or denote new terms at the place where they are defined in
the text.

Advanced System Diagnostics and Troubleshooting Guide 11
1Introduction
This guide describes how to use the ExtremeWare hardware diagnostics suite to test and validate the
operating integrity of Extreme Networks switches. The tools in the diagnostic suite are used to detect,
isolate, and treat faults in a system.
This chapter contains the following sections:
•Diagnostics: A Brief Historical Perspective on page 12
•Overview of the ExtremeWare Diagnostics Suite on page 12
•Supported Hardware on page 13
•Applicable ExtremeWare Versions on page 13
Introduction
The purpose of this guide is to provide information and guidelines to assist you in implementing the
diagnostic suite within ExtremeWare. The Extreme Networks diagnostic software is intended to identify
possible faulty hardware or software error conditions and—depending on how the various diagnostics
features are configured—take the appropriate preconfigured action. The action might be to enable the
switch to write informative error messages to the switch log, attempt to recover itself and continue
operating, or simply remove the suspect system component from service.
It is important to note that while each diagnostic test—by itself—is not complicated, system
configuration, as a whole, must be evaluated to ensure ongoing, expected behavior both with the switch
and across the network itself. For example, in implementing the diagnostic suite, you must take into
consideration these operational variables:
•Redundancy mechanisms implemented
•Levels of redundancy within the network
•Acceptable outage windows (scheduled and unscheduled)

12 Advanced System Diagnostics and Troubleshooting Guide
Introduction
Diagnostics: A Brief Historical Perspective
Diagnostic utility programs were created to aid in troubleshooting system problems by detecting and
reporting faults so that operators or administrators could go fix the problem. While this approach does
help, it has some key limitations:
•It is, at its base, reactive, meaning a failure must occur before the diagnostic test can be used to look
for a cause for the failure.
•It can be time consuming, because the ability to troubleshoot a failure successfully based on the
information provided by the diagnostics test depends greatly on the types of information reported
by the test and the level of detail in the information.
Because users of mission-critical networks and network applications are becoming increasingly
dependent on around-the-clock network access and highest performance levels, any downtime or
service degradation is disruptive and costly. Time lost to an unexpected failure, compounded by more
time lost while someone attempts to isolate and fix the failure, has become increasingly less acceptable.
The process of improving diagnostic tests to minimize failures and their impact is a kind of feedback
system: What you learn through the use of the diagnostics improves your understanding of hardware
failure modes; what you learn from an improved understanding of hardware failure modes improves
your understanding of the diagnostics.
The goal of the current generation of ExtremeWare diagnostics is to help users achieve the highest
levels of network availability and performance by providing a suite of diagnostic tests that moves away
from a reactive stance—wherein a problem occurs and then you attempt to determine what caused the
problem—to a proactive state—wherein the system hardware, software, and diagnostics work together
to reduce the total number of failures and downtime through:
•More accurate reporting of errors (fewer false notifications; more information about actual errors)
•Early detection of conditions that lead to a failure (so that corrective action can be taken before the
failure occurs)
•Automatic detection and correction of packet memory errors in the system’s control and data planes
Administrators will now find a greatly reduced MTTR (mean time to repair) due to fast and accurate
fault identification. Multiple modules will no longer need to be removed and tested; faulty components
will usually be identified directly. Over time, there should be a significant reduction in the number of
problems found.
NOTE
In spite of the improved ExtremeWare hardware diagnostics, some network events might still occur,
because software is incapable of detecting and preventing every kind of failure.
Overview of the ExtremeWare Diagnostics Suite
The ExtremeWare diagnostic suite includes the following types of tools for use in detecting, isolating,
and treating faults in a switch. Each of these diagnostic types is summarized below, but is described in
greater detail in later sections of this guide.
•Power-on self test (POST)—A sequence of hardware tests that run automatically each time the switch
is booted, to validate basic system integrity. The POST runs in either of two modes: normal (more
thorough, but longer-running test sequence) or FastPOST (faster-running basic test sequence).

Supported Hardware
Advanced System Diagnostics and Troubleshooting Guide 13
•On-demand system hardware diagnostics—Run on demand through user CLI commands; runs in
either of two modes: normal (faster-running basic test sequence) or extended (more thorough, but
longer-running test sequence).
The extended diagnostics include the packet memory scan, which checks the packet memory area of
the switch fabric for defects and maps out defective blocks. This test can be run by itself, as part of
the slot-based extended diagnostics, or can be invoked from within the system health checks.
•Switch-wide communication-path packet error health checks—(Apply only to “i” series Summit,
Alpine, and BlackDiamond switches.) An integrated diagnostic subsystem—the system health check
feature—that consists of a number of different test types, operating proactively in the background to
detect and respond to packet error problems in module memory or on communication paths.
The system health checks for “i” series switches include the following kinds of tests:
—Backplane health checks
—CPU health checks
—Switch fabric checksum validation
—Dynamic memory scanning and memory mapping
—Transceiver diagnostics
—Forwarding database (FDB) scan
NOTE
One component of the system health check feature is a bus monitoring and data integrity verification
subsystem that monitors the operation of all data and control busses within the switch. This low-level
subsystem—composed of software and hardware components—passes the results of its tests to
another system health check subsystem, an intelligent layer that is responsible for interpreting the
test results and reporting them to the user. Since the inception of the use of this intelligent
interpretation and reporting layer in the system health check feature, the operation of this
interpretation and reporting subsystem has undergone significant changes. At its current level of
maturity, it represents an intelligent, integral component of the overall system health check system.
Supported Hardware
The ExtremeWare diagnostic suite applies only to Extreme Networks switch products based on the
“inferno” series chipset. Equipment based on this chipset are referred to as being “inferno” series or “i”
series products: the BlackDiamond family of core chassis switches (6804, 6808, and 6816), the Alpine
systems (3802, 3804, 3808), and the Summit “i”-series stackables (Summit1i, Summit5i, Summit7i,
Summit 48i, and Summit48Si).
A subset of the ExtremeWare diagnostic suite—runtime, slot-based diagnostics—applies only to Extreme
Networks Summit “e” series switch products. The Summit “e” series switches include the following
models: Summit 200-24, Summit 200-48, Summit 300-24, Summit 300-48, Summit 400-24p,
Summit 400-24t, and Summit 400-48t.
Applicable ExtremeWare Versions
The information in this guide is based on the features and feature attributes found in
ExtremeWare Version 7.4 or later.

14 Advanced System Diagnostics and Troubleshooting Guide
Introduction

Advanced System Diagnostics and Troubleshooting Guide 15
2“ i ” Series Switch Hardware Architecture
This chapter provides a brief summary of the “i” series switch hardware features most relevant to
understanding the use of the Extreme Networks diagnostic suite.
This chapter contains the following sections:
•Diagnostics Support on page 15
•The BlackDiamond Systems on page 16
•Alpine Systems on page 22
•Summit “i” Series Systems on page 23
Diagnostics Support
The ExtremeWare diagnostic suite applies only to Extreme Networks switch products based on the
“inferno” series chipset. Equipment based on this chipset are referred to as being “inferno” series or “i”
series products: the BlackDiamond family of core chassis switches (6804, 6808, and 6816), the Alpine
systems (3802, 3804, 3808), and the Summit “i”-series stackables (Summit1i, Summit5i, Summit7i,
Summit 48i, and Summit48Si).
NOTE
These switches and the switch modules use naming conventions ending with an “i” to identify them as
“inferno” series or “i” series products. For the most current list of products supporting the “i” chipset,
such as the MSM-3 and other “3”-series modules, such as the G16X3, consult your release notes.
Unless otherwise specified, a feature requiring the “i” chipset requires the use of both an “i”
chipset-based management module, such as the MSM64i, and an “i” chipset-based I/O module, such as
the G8Xi.

16 Advanced System Diagnostics and Troubleshooting Guide
“i” Series Switch Hardware Architecture
The BlackDiamond Systems
In the context of the advanced system diagnostics suite, the BlackDiamond family of core chassis
switches share the same fundamental hardware architecture: a multislot modular chassis containing a
passive backplane that supports redundant load-sharing, hot-swappable switch fabric modules. On
BlackDiamond systems, each I/O module and MSM represents an individual switch containing its own
switching fabric and packet memory.
Figure 1: BlackDiamond 6800 Series architecture, general block diagram
BlackDiamond 6800 Series Hardware Architecture Differences
In the context of understanding the ExtremeWare diagnostics and their use in troubleshooting system
problems, these are the key hardware distinctions between the BlackDiamond 6816, 6808, and 6804.
•BlackDiamond 6816—Modular chassis with passive backplane; sixteen chassis slots for I/O modules;
four chassis slots for MSMs.
•BlackDiamond 6808—Modular chassis with passive backplane; eight chassis slots for I/O modules;
two chassis slots for MSMs.
•BlackDiamond 6804—Modular chassis with passive backplane; four chassis slots for I/O modules;
two chassis slots for MSMs.
CPU
Subassembly
EXT MAC
To PHY
To MSM-ATo MSM-B
To BackplaneTo Backplane To BackplaneTo Backplane
To PHY
EXT MAC
INT MACINT MAC
NVRAM
CPLD
UART
PCMCIA
FLASH
SRAM
MGMT
OTP
PQM
Packet
Memory
FDB
VPST
AFQM
ASIC
(Quake)
Control Bus Control Bus
Control Bus Control Bus
I/O Module (G8Xi) Master MSM (MSM-A)
Master MSM Daughter Card
Control Bus
Data/Packet
Bus (PBus)
Transceiver
(To all other slots)
SE
ASIC
(Twister)
PBus PBus
BackplaneBackplane
DN_033A
MACMAC
MACMAC
OTP
PQM
Packet
Memory
FDB
VPST
AFQM
ASIC
(Quake)
SE
ASIC
(Twister)
PBus PBus

The BlackDiamond Systems
Advanced System Diagnostics and Troubleshooting Guide 17
The BlackDiamond Backplane
The BlackDiamond backplane is a passive backplane, meaning that all the active components such as
CPUs, ASICs, and memory have been moved onto plug-in modules, such as the I/O modules and
MSMs.
Figure 2: BlackDiamond passive backplane architecture (BlackDiamond 6808 shown)
The BlackDiamond backplane provides inter-slot electrical connections for both network data traffic and
a separate control bus for switch fabric management. Data traffic is carried on four AUI links between
each MSM and each I/O slot on BlackDiamond 6804 and BlackDiamond 6808 systems, and on two AUI
links between each MSM and each I/O slot on BlackDiamond 6816 systems. Device management occurs
on a 32-bit PCI bus connecting MSMs and I/O modules. The number of backplane slots for I/O
modules and MSMs determines the BlackDiamond system type (6804, 6808, 6816).
The chief advantages of a passive backplane are:
•The absence of active components yields a much lower possibility of backplane failure.
•You can remove and replace system modules faster, making upgrades and repairs easier, faster, and
cheaper.
NOTE
One disadvantage of a passive backplane is that a problem on one switch module might cause other
switch modules to fail. More information on this possibility is covered in later chapters of this guide.
Switch
Module
64 Gbps
Switch
Fabric
Eight Load-Shared
Gigabit Links
Fault-Tolerant
Switch Fabric
and System
Management
A
1
2
3
8
B
64 Gbps
Switch
Fabric
Switch
Module
Switch
Module
Switch
Module
DN_001
A

18 Advanced System Diagnostics and Troubleshooting Guide
“i” Series Switch Hardware Architecture
BlackDiamond I/O Modules
Each BlackDiamond I/O module has a built-in switching fabric (see Figure 3) giving the module the
capability to switch local traffic on the same module. Traffic that is destined for other modules in the
chassis travels across the backplane to the MSMs, where it is switched and sent to its destination I/O
module.
Figure 3: BlackDiamond I/O module architecture (G8Xi 32 Gb fabric shown)
Each BlackDiamond I/O module has eight load-shared Gigabit Ethernet links to both MSMs through
the backplane. The load sharing algorithm distributes traffic across different channels through the
backplane’s Gigabit Ethernet links, providing bi-directional communication.
Each BlackDiamond I/O module is equipped with the following kinds of hardware components:
•PHY: An industry-standard ASIC responsible for physical layer (layer 1) signal, clocking, etc.
•MAC: The MAC handles the standard MAC layer functions as well as some other functions to
prepare a packet for transmission to the switch fabric or to the external network, including 802.1p
and DiffServ examination, VLAN insertion, MAC substitution, TTL decrement, 802.1p and DiffServ
replacement, etc.
Each I/O module has both external MACs and internal MACs. External MACs handle the interface to
the external ports; internal MACs handle the interface to the BlackDiamond backplane. Each MSM
provides four Gigabit Ethernet links to each I/O module.
•PBUS: The packet data bus that transfers packets between the MAC and the packet memory.
•Switch engine (distributed packet processor) ASIC (Twister) and its associated memories: packet
RAM and FDB RAM. The SE ASIC implements a high-speed, parallel data transfer bus for
transferring packets from MACs to packet memory and back.
•Address filtering and queue management ASIC (Quake) and its associated memories: OTP RAM, PQ
RAM, and VPST RAM.
When a data packet is received by the PHY, the PHY passes the packet to the MAC. The MAC handles
the layer 2 tasks, such as tagging and the MAC address, then transfers the packet across the PBUS to the
AFQM
ASIC
OTP RAM
4x GbE to
MSM-A
4x GbE to
MSM-B
PHY
PHY
MAC
MAC
MAC
MAC
PQ RAM
VPST RAM
SRAM (Packet Mem + FDB)
DN_026C
SE
ASIC
SE
ASIC
SE
ASIC
SE
ASIC
PBUS

The BlackDiamond Systems
Advanced System Diagnostics and Troubleshooting Guide 19
packet memory for temporary storage. Based on the information in memory, such as the FDB, the
address filtering and queue management ASIC makes a forwarding decision. If the next hop is a local
port (on the same module), the packet is forwarded to the external MAC and PHY for the exit port. If
the packet is destined for another module (as either slow path traffic or fast path traffic), the packet is
transferred to the internal MAC and then on to the MSM (CPU).
All I/O modules share the management bus on the backplane, and use it to communicate to each other
and to the MSMs.
Management Switch Modules
As its name indicates, the Management Switch Fabric Module (MSM) serves a dual role in the system: it
is equipped to act as the internal switch fabric for data that is being transferred between I/O modules
in the chassis, and to handle the upper-layer processing and system management functions for the
switch. (See Figure 4.)
Figure 4: BlackDiamond MSM architecture (MSM64 i shown)
An MSM is equipped with the following hardware components: CPU subsystem (dual CPUs, DRAM,
NVRAM, flash memory, and PCMCIA slot) and switch fabric subsystem (Quake ASICs, OTP, PQ, and
VPST RAM, packet memory and FDB SRAM, Twister ASICs, PBUS, and MAC ASICs).
AFQM
ASIC
256 MB
DRAM
OTP RAM
MAC
MAC
MAC
MAC
MAC
MAC
MAC
CPU
CPU
PCMCI
Slot
MAC
PQ RAM
VPST RAM
Flash
NVRAM AFQM
ASIC
OTP RAM
PQ RAM
VPST RAM
SRAM (Packet Mem + FDB)
DN_027C
SE
ASIC
SE
ASIC
SE
ASIC
SE
ASIC
SE
ASIC
SE
ASIC
SE
ASIC
SE
ASIC
PBUS
CPU
Subsystem

20 Advanced System Diagnostics and Troubleshooting Guide
“i” Series Switch Hardware Architecture
BlackDiamond MSM Redundancy
The CPU subsystems on a pair of BlackDiamond MSMs operate in a master-slave relationship.
(See Figure 5.)
Figure 5: BlackDiamond MSM redundancy scheme
The master MSM CPU subsystem actively manages the switch and the task of switching packets in the
CPU control (or management) path. The slave MSM CPU subsystem is in standby mode, but is checked
periodically by the master MSM CPU (via EDP) to determine whether it is still available.
The master MSM also guarantees that management operations occur in a synchronized manner. For
example, if you make a configuration change and need to save it, the master MSM ensures that the
configuration is saved to both the master MSM and the slave MSM at the same time. That way, the
updated slave is ready to take over as the master if the master MSM fails.
Despite the master-slave switch management role, the switch fabrics on both MSMs actively switch core
traffic in a load-shared fashion. Load-sharing switches core traffic from different I/O modules.
All MSMs share the control (or management) bus on the backplane, and use it to communicate to each
other and to installed I/O modules to perform system health checks and status polling.
Causes of MSM Failover and System Behavior
A number of events can cause an MSM failover to occur, including:
•Software exception; system watchdog timer expiry
•Diagnostic failure (extended diagnostics, transceiver check/scan, FDB scan failure/remap)
•Hot removal of the master MSM or hard-reset of the master MSM
MSM64i (A)
MSM64i (B)
Switching
Fabric
Fault Tolerant
Switch Fabric
and
System
Management
I/O Module
I/O Module
I/O Module
I/O Module
I/O Module
Backplane
I/O Module
I/O Module
I/O Module
CPU
Sub-
system
Switching
Fabric
CPU Control Path
Data Path
CPU
Sub-
system
DN_028A