Nortel IW SPM IP User manual

IW SPM IP Fault Management
Nortel Networks ConfidentialCopyright © 2004, Nortel Networks
IW SPM IP Fault Management
Fault management strategy
Alarm reporting system for SPM-based equipment
The alarm reporting system integrates event detection and alarm
notification functions. An alarm becomes active when a reduced
service, reliability, or test condition occurs in the network or network
element. The alarm remains active until a system event or activity
performed by operating company personnel clears the alarm condition.
The alarm system includes audible notification and visual display
through warning lights and the Maintenance and Administration
Position (MAP) terminal.
• Critical alarm
A critical alarm indicates a reduced service condition or complete
loss of service. A critical alarm indicates that the system can no
longer perform its design function. The alarm condition requires
immediate correcting action so that the performance of the system
can return to its design function.
• Major alarm
A major alarm indicates lost redundancy. The next fault of the same
type can cause a reduction or complete loss of service. There is no
backup if another fault occurs on the active system. this alarm level
can be generated when service decreases below an operating
company defined threshold.
• Minor alarm
A minor alarm indicates a small loss of redundancy. The next fault
of the same type will not cause degradation of service.
Equipment alarms and alarm severity codes indicate the effect that a
fault or event has on a single piece of equipment. There are three types
of alarm severities:
For IW SPM IP-based equipment, the MAP terminal displays alarm
codes in the banner and the subsystem status summary field (SSSF).
The alarm banner displays alarm codes that indicate the effect of the

NN10078-911 SN06.2 Preliminary 02.03 March 2004
Nortel Networks Confidential
2
Copyright © 2004, Nortel Networks
alarm event on the network or network element. The SSSF displays
alarm codes that indicate equipment faults of system states.
Log reports
A log report is a record of a message that your system generates
whenever a significant event has occurred in the switch or one of its
peripherals. Log reports include status and activity reports, as well as
reports on hardware or software faults, test results, changes in state,
and other events or conditions likely to affect the performance of the
switch. Either a system or a manual action can generate a log report.
The figure below, Log report example for SPM630 log, gives an
example of an information log.
Figure 1 Log report example for SPM630 log
Figure 2, Log report example for SPM332 log shows a REX failure log
along with explanations for the various fields.
Figure 2 Log report example for SPM332 log
SPM630 Feb 07 10:22:11 4700 INFO Device Protection
From: Working
Location SPM: 01 OC3 01
To: Spare
Field Description
SPM REX Aborted Indicates that the information to follow mentions why REX
test on the SPM node was aborted
Node type Indicates that the node type for this log is SPM
Node Number SPM number (range is 0-85)
Type Indicates the variant of the SPM (valid values are DMSCP,
MG4K, IW, and DPT).
Reason Provides the reason why the REX test was aborted (limited to
three lines of text).
SPM332 JAN 24 01:57:54 2112 TBL SPM REX Aborted
Reason: Stability check on CEM 0 failed
SPM: 0 Type: IW
SYSB in the past 12 hours

IW SPM IP Fault Management
Nortel Networks Confidential
3
Copyright © 2004, Nortel Networks
Tools and utilities
Fault management for alarms and logs is performed through the
Maintenance and Administration Position (MAP) terminal.

NN10078-911 SN06.2 Preliminary 02.03 March 2004
Nortel Networks Confidential
4
Copyright © 2004, Nortel Networks
Fault management indicators
The following table lists the fault management indicators generated by
the IW SPM IP.
IW SPM IP fault management indicators
Log name
and number Explanation
CLOCK500 Logs related to system synchronization state changes
IOAU112 The System REX Controller noted changes to the System REX
Controller operation or schedule.
IWBM500 The C-side link, STS3cP carrier, the network state, or the address
state for the IW bridge software goes out-of-service.
IWBM501 The C-side link, STS3cP carrier, the network state, or the address
state for the IW bridge software returns to service.
IWBM600 The IW bridge receives an invalid terminal ID during an attempt to
free a bridge.
IWBM601 The audit finds a problem and performs a corresponding action. For
Example an audit action occurs when a bridge has incorrect
connectivity or a pointer to a bridge’s protected data is corrupt.
IWBM602 No bridge available - info.
IWBM603 Audit log - info.
IWBM700 Maintenance action log - info.
IWBM800 Number of available IW bridges exceeds the first threshold when
attempting to retrieve and IW-bridge ID from the IW bridge manager.
The first threshold is 70% of the available IW bridges in the
IW-bridge manager exhausted or in use. Does not include offline
carriers.
IWBM801 Number of available IW bridges falls to less than 65% of bridges in
the pool in use. This log always occurs after IWBM800. Does not
include offline carriers.
IWBM802 Number of available IW bridges exceeds second threshold when
attempting to retrieve and IW-bridge ID from the IW bridge manager.
The second threshold is 90% of the available IW bridges in the
IW-bridge manager exhausted or in use. Does not include offline
carriers.

IW SPM IP Fault Management
Nortel Networks Confidential
5
Copyright © 2004, Nortel Networks
IWBM803 Number of available IW bridges falls to less than 85% of bridges in
the pool in use. This log always occurs after IWBM802. Does not
include offline carriers.
IWBM900 Nortel design debug tool.
LINK300 A DS-512 hostlink is detected as misconnected between the ENET
and the SPM-CEM.
Check to find out which links are misconnected.
• BSY the links from the ENET map
• Connect the links from the ENET to correspond correctly with the
CEM. The port on the CEM must be exactly as datafilled in table
MNLINK.
• RTS the MANB links from the ENET map
Note: Refer to solution level logs to see other possible causes for
the LINK300 log
LINK500 Logs generated from the MS and ENET for the DS-512 links.
NODE300 Logs related to hardware faults in the RMs and CEM.
Refer to log NODE303 below.
NODE302 Software Alarm
NODE303 Logs to indicate Wrong Application Data on the CEM.
Core Integrated Node Maintenance (INM) detected wrong
application data for one of the following reasons:
• CEM restarted with wrong application data due to manually
initiated Reset/Loadmod/InsvLD.
• CEM restarted with wrong application data due to system
initiated Reset/Loadmod.
• Manual/system initiated RTS (bringing CEM back into service -
INSV/ISTB) was attempted on CEM with wrong application data.
Note 1: Refer to Log NODE303 format on page 10 for NODE303
format details.
Note 2: See log Link300 (above) to help identify the misconnected
DS-512 Link.
IW SPM IP fault management indicators
Log name
and number Explanation

NN10078-911 SN06.2 Preliminary 02.03 March 2004
6
Copyright © 2004, Nortel Networks Nortel Networks Confidential
NODE326 Log NODE326 is generated when a hardware fault occurs in a unit
of a node.
NODE500 Logs related to node state change and activity switching. Log
NODE500 is generated whenever a state change happens in any of
the units of a dual unit PM. NODE500 is an information log.
NODE600 System recovery action. Info log
NODE601 System status notification Info log.
SPM300 Logs related to hardware faults in the RMs (clock hardware, RM
hardware, or link hardware) which indicates a device fault has
occurred. There are three severities of problem descriptions:
• Info - information only log.
• Non-Critical - A Non-Critical fault reported by the active CEM
causes the affected RM to change to an ISTB state.
• Critical - A Critical fault reported by the active CEM causes the
RM to change to a SYSB state and spare, if it is active.
Faults reported by the inactive CEM affect the RM state when that
CEM becomes active, if they are not cleared prior to that CEM
becoming active.
The problem description field contains a text string indicating which
problem occurred.
The B-channel is busied or returned to service.
Clear the fault using the appropriate procedure.
Note: See Log SPM300 format on page 11 and SPM300 log
problem descriptions on page 12 for detailed explanations.
SPM301 The clock oscillator tuning range has reached 70% or 90% of the
maximum range.
SPM311 Software related to RM sparing.
SPM312 TBL trap.
SPM313 Module information memory (MIM) on an SPM.
SPM314 TBL IMC.
SPM330 The two CEMs have either come in to datasync or out of datasync.
IW SPM IP fault management indicators
Log name
and number Explanation

IW SPM IP Fault Management
Nortel Networks Confidential
7
Copyright © 2004, Nortel Networks
SPM331 A device had a protection switch failure.
SPM332 SPM REX aborted. Reports stability or screening failures for REX
operation.
Note: See Log SPM332 Problem Descriptions (below) for REX test
failure explanations.
SPM333 SPM REX failed. Reports failures reasons for REX operation.
Reasons for an SPM REX failure:
• RESET step failed
• OOS test step failed
• RTS step failed CEM
• CEM drops to SYSB or does not come to Insv state during
post-RTS wait period
• SWACT step failed (either rejected or failed from Local Agent)
Note: If the reason in SPM333 is RESET, OOS test, or RTS step,
collect NODE500 and NODE600 logs. A SYSB alarm will be raised
on the MAP and PM Alarm Banner will be highlighted.
SPM335 A device had a protection switch failure.
Clear the PROTFAIL alarm using the appropriate procedure
SPM340 Fail CM Warm Swact.
SPM341 An RM has entered into Holdover state.
SPM344 The SRM Loss of BITS Redundancy (LOR) alarm has been set.
SPM350 There is the potential for resource exhaustion of a particular
resource type. The log also generates when the alarm is cleared.
IW SPM IP fault management indicators
Log name
and number Explanation

NN10078-911 SN06.2 Preliminary 02.03 March 2004
8
Copyright © 2004, Nortel Networks Nortel Networks Confidential
SPM370 An SPM370 log (Health Monitor Event log) is generated whenever
a Health Monitor alarm is raised or a Health Monitor RMM action is
taken.
When the status is “Alarm Raised”, investigate the cause. The
“Correlating logs with alarms” section of this document provides
possible causes for the associated alarms.
Refer to the SPMCP (SPM call processing) commands in the
Security and Administration document, NN10163-611 for additional
information.
Note: When the status is “Info”, no action is required.
SPM500 Logs related to RM state change and activity switching (sparing).
SPM501 The clock mode has changed from sync, freerun, holdover, or
acquire to sync, freerun, holdover, or acquire.
SPM600 Logs generated when the MS changes synchronization modes and
is unable to notify an in-service SPM of the change.
SPM630 A successful sparing event has occurred.
SPM632 SPM REX Info. A log generated to record the start time of REX
operation.
SPM633 SPM REX Success. A successful REX operation has occurred.
Note: Node 500 logs report state changes on the CEM during REX
tests.
SPM642 An RM has exited Holdover24 state.
SPM644 The SRM Loss of BITS Redundancy (LOR) alarm has been cleared.
SPM645 Link Protocol & Messaging Interface Controller (LPMIC) Event
Report. This information log is generated on every CEM and
transported to the core. The log is generated periodically and
whenever the number of events crosses a threshold value.
SPM650 A successful in-service loading procedure has occurred.
SPM651 Logs generated when an in-service CEM/RM loading operation on
a circuit pack fails.
IW SPM IP fault management indicators
Log name
and number Explanation

IW SPM IP Fault Management
Nortel Networks Confidential
9
Copyright © 2004, Nortel Networks
SPM670 An SPM670 log is generated when a Health Monitor event clears.
SPM680 “Message Buffer Manager (MBM) buffer usage high” log message.
SPM681 “Message Buffer Manager (MBM) buffer pool low” log message.
SPM684 The Erase Flash information log indicates when the:
• Erase flash command is initiated,
• Erase flash command is completed,
• Erase flash command failed, and
• Erase flash command is rejected.
None
(This is an information log)
The log displays the CEM and SPM number on which the Erasefl
command is executed in addition to the userid of the craft who
initiated the command.
SPM700 DDM audit fail.
SPM701 DDM audit succeeded.
SPM702 DDM dynamic update failed.
VOIP800 Cyclical Redundancy Check (CRC) alignment error.
VOIP801 Undersize packets
VOIP802 Oversize packets.
VOIP803 Fragments.
VOIP804 Jabber.
VOIP805 Drop events.
VOIP806 Broadcast packets.
VOIP807 Jitter.
VOIP808 Latency
IW SPM IP fault management indicators
Log name
and number Explanation

NN10078-911 SN06.2 Preliminary 02.03 March 2004
10
Copyright © 2004, Nortel Networks Nortel Networks Confidential
Log NODE303 Format
Log NODE303 reports Wrong Application Data. This means the CEM
in slot 7 of the SPM is getting application data for CEM1, and the CEM
in slot 8 is getting application data for CEM 0. Node303 displays actions
required in response to this.
Note: The CEM cannot Return to Service with Wrong Application
Data.
Log NODE303 format
NODE 303 **<mmmdd hh:mm:ss> <ssdd> TBL Wrong
Application Data
Location: <NODE>
Trouble: CEM has Wrong Application Data
Action: Check and re-connect DS-512 Links correctly
Integrated Node Maintenance Detailed Information
Trouble Reason: DS-512 Link/Links may be
misconnected
Trouble Detail: CEM in Slot <slot_number(7 or 8)>
has Application Data of CEM <cem_number(0 or 1)>
VOIP809 Packets lost.
VOIP810 Decoder under run.
IW SPM IP fault management indicators
Log name
and number Explanation
Log NODE303 format
Field Description
TBL Indicates the trouble condition with the node.
Wrong Application
Data
Indicates that the information to follow is about Wrong
Application Data on the node.
Location Indicates the location of the peripheral event to which the
event applies.
Trouble Identifies the type of problem.
Action (User) Indicates the suggested user action.