Nortel IW SPM IP User manual

IW SPM IP Fault Management
Nortel Networks ConfidentialCopyright © 2004, Nortel Networks
IW SPM IP Fault Management
Fault management strategy
Alarm reporting system for SPM-based equipment
The alarm reporting system integrates event detection and alarm
notification functions. An alarm becomes active when a reduced
service, reliability, or test condition occurs in the network or network
element. The alarm remains active until a system event or activity
performed by operating company personnel clears the alarm condition.
The alarm system includes audible notification and visual display
through warning lights and the Maintenance and Administration
Position (MAP) terminal.
• Critical alarm
A critical alarm indicates a reduced service condition or complete
loss of service. A critical alarm indicates that the system can no
longer perform its design function. The alarm condition requires
immediate correcting action so that the performance of the system
can return to its design function.
• Major alarm
A major alarm indicates lost redundancy. The next fault of the same
type can cause a reduction or complete loss of service. There is no
backup if another fault occurs on the active system. this alarm level
can be generated when service decreases below an operating
company defined threshold.
• Minor alarm
A minor alarm indicates a small loss of redundancy. The next fault
of the same type will not cause degradation of service.
Equipment alarms and alarm severity codes indicate the effect that a
fault or event has on a single piece of equipment. There are three types
of alarm severities:
For IW SPM IP-based equipment, the MAP terminal displays alarm
codes in the banner and the subsystem status summary field (SSSF).
The alarm banner displays alarm codes that indicate the effect of the

NN10078-911 SN06.2 Preliminary 02.03 March 2004
Nortel Networks Confidential
2
Copyright © 2004, Nortel Networks
alarm event on the network or network element. The SSSF displays
alarm codes that indicate equipment faults of system states.
Log reports
A log report is a record of a message that your system generates
whenever a significant event has occurred in the switch or one of its
peripherals. Log reports include status and activity reports, as well as
reports on hardware or software faults, test results, changes in state,
and other events or conditions likely to affect the performance of the
switch. Either a system or a manual action can generate a log report.
The figure below, Log report example for SPM630 log, gives an
example of an information log.
Figure 1 Log report example for SPM630 log
Figure 2, Log report example for SPM332 log shows a REX failure log
along with explanations for the various fields.
Figure 2 Log report example for SPM332 log
SPM630 Feb 07 10:22:11 4700 INFO Device Protection
From: Working
Location SPM: 01 OC3 01
To: Spare
Field Description
SPM REX Aborted Indicates that the information to follow mentions why REX
test on the SPM node was aborted
Node type Indicates that the node type for this log is SPM
Node Number SPM number (range is 0-85)
Type Indicates the variant of the SPM (valid values are DMSCP,
MG4K, IW, and DPT).
Reason Provides the reason why the REX test was aborted (limited to
three lines of text).
SPM332 JAN 24 01:57:54 2112 TBL SPM REX Aborted
Reason: Stability check on CEM 0 failed
SPM: 0 Type: IW
SYSB in the past 12 hours

IW SPM IP Fault Management
Nortel Networks Confidential
3
Copyright © 2004, Nortel Networks
Tools and utilities
Fault management for alarms and logs is performed through the
Maintenance and Administration Position (MAP) terminal.

NN10078-911 SN06.2 Preliminary 02.03 March 2004
Nortel Networks Confidential
4
Copyright © 2004, Nortel Networks
Fault management indicators
The following table lists the fault management indicators generated by
the IW SPM IP.
IW SPM IP fault management indicators
Log name
and number Explanation
CLOCK500 Logs related to system synchronization state changes
IOAU112 The System REX Controller noted changes to the System REX
Controller operation or schedule.
IWBM500 The C-side link, STS3cP carrier, the network state, or the address
state for the IW bridge software goes out-of-service.
IWBM501 The C-side link, STS3cP carrier, the network state, or the address
state for the IW bridge software returns to service.
IWBM600 The IW bridge receives an invalid terminal ID during an attempt to
free a bridge.
IWBM601 The audit finds a problem and performs a corresponding action. For
Example an audit action occurs when a bridge has incorrect
connectivity or a pointer to a bridge’s protected data is corrupt.
IWBM602 No bridge available - info.
IWBM603 Audit log - info.
IWBM700 Maintenance action log - info.
IWBM800 Number of available IW bridges exceeds the first threshold when
attempting to retrieve and IW-bridge ID from the IW bridge manager.
The first threshold is 70% of the available IW bridges in the
IW-bridge manager exhausted or in use. Does not include offline
carriers.
IWBM801 Number of available IW bridges falls to less than 65% of bridges in
the pool in use. This log always occurs after IWBM800. Does not
include offline carriers.
IWBM802 Number of available IW bridges exceeds second threshold when
attempting to retrieve and IW-bridge ID from the IW bridge manager.
The second threshold is 90% of the available IW bridges in the
IW-bridge manager exhausted or in use. Does not include offline
carriers.

IW SPM IP Fault Management
Nortel Networks Confidential
5
Copyright © 2004, Nortel Networks
IWBM803 Number of available IW bridges falls to less than 85% of bridges in
the pool in use. This log always occurs after IWBM802. Does not
include offline carriers.
IWBM900 Nortel design debug tool.
LINK300 A DS-512 hostlink is detected as misconnected between the ENET
and the SPM-CEM.
Check to find out which links are misconnected.
• BSY the links from the ENET map
• Connect the links from the ENET to correspond correctly with the
CEM. The port on the CEM must be exactly as datafilled in table
MNLINK.
• RTS the MANB links from the ENET map
Note: Refer to solution level logs to see other possible causes for
the LINK300 log
LINK500 Logs generated from the MS and ENET for the DS-512 links.
NODE300 Logs related to hardware faults in the RMs and CEM.
Refer to log NODE303 below.
NODE302 Software Alarm
NODE303 Logs to indicate Wrong Application Data on the CEM.
Core Integrated Node Maintenance (INM) detected wrong
application data for one of the following reasons:
• CEM restarted with wrong application data due to manually
initiated Reset/Loadmod/InsvLD.
• CEM restarted with wrong application data due to system
initiated Reset/Loadmod.
• Manual/system initiated RTS (bringing CEM back into service -
INSV/ISTB) was attempted on CEM with wrong application data.
Note 1: Refer to Log NODE303 format on page 10 for NODE303
format details.
Note 2: See log Link300 (above) to help identify the misconnected
DS-512 Link.
IW SPM IP fault management indicators
Log name
and number Explanation

NN10078-911 SN06.2 Preliminary 02.03 March 2004
6
Copyright © 2004, Nortel Networks Nortel Networks Confidential
NODE326 Log NODE326 is generated when a hardware fault occurs in a unit
of a node.
NODE500 Logs related to node state change and activity switching. Log
NODE500 is generated whenever a state change happens in any of
the units of a dual unit PM. NODE500 is an information log.
NODE600 System recovery action. Info log
NODE601 System status notification Info log.
SPM300 Logs related to hardware faults in the RMs (clock hardware, RM
hardware, or link hardware) which indicates a device fault has
occurred. There are three severities of problem descriptions:
• Info - information only log.
• Non-Critical - A Non-Critical fault reported by the active CEM
causes the affected RM to change to an ISTB state.
• Critical - A Critical fault reported by the active CEM causes the
RM to change to a SYSB state and spare, if it is active.
Faults reported by the inactive CEM affect the RM state when that
CEM becomes active, if they are not cleared prior to that CEM
becoming active.
The problem description field contains a text string indicating which
problem occurred.
The B-channel is busied or returned to service.
Clear the fault using the appropriate procedure.
Note: See Log SPM300 format on page 11 and SPM300 log
problem descriptions on page 12 for detailed explanations.
SPM301 The clock oscillator tuning range has reached 70% or 90% of the
maximum range.
SPM311 Software related to RM sparing.
SPM312 TBL trap.
SPM313 Module information memory (MIM) on an SPM.
SPM314 TBL IMC.
SPM330 The two CEMs have either come in to datasync or out of datasync.
IW SPM IP fault management indicators
Log name
and number Explanation

IW SPM IP Fault Management
Nortel Networks Confidential
7
Copyright © 2004, Nortel Networks
SPM331 A device had a protection switch failure.
SPM332 SPM REX aborted. Reports stability or screening failures for REX
operation.
Note: See Log SPM332 Problem Descriptions (below) for REX test
failure explanations.
SPM333 SPM REX failed. Reports failures reasons for REX operation.
Reasons for an SPM REX failure:
• RESET step failed
• OOS test step failed
• RTS step failed CEM
• CEM drops to SYSB or does not come to Insv state during
post-RTS wait period
• SWACT step failed (either rejected or failed from Local Agent)
Note: If the reason in SPM333 is RESET, OOS test, or RTS step,
collect NODE500 and NODE600 logs. A SYSB alarm will be raised
on the MAP and PM Alarm Banner will be highlighted.
SPM335 A device had a protection switch failure.
Clear the PROTFAIL alarm using the appropriate procedure
SPM340 Fail CM Warm Swact.
SPM341 An RM has entered into Holdover state.
SPM344 The SRM Loss of BITS Redundancy (LOR) alarm has been set.
SPM350 There is the potential for resource exhaustion of a particular
resource type. The log also generates when the alarm is cleared.
IW SPM IP fault management indicators
Log name
and number Explanation

NN10078-911 SN06.2 Preliminary 02.03 March 2004
8
Copyright © 2004, Nortel Networks Nortel Networks Confidential
SPM370 An SPM370 log (Health Monitor Event log) is generated whenever
a Health Monitor alarm is raised or a Health Monitor RMM action is
taken.
When the status is “Alarm Raised”, investigate the cause. The
“Correlating logs with alarms” section of this document provides
possible causes for the associated alarms.
Refer to the SPMCP (SPM call processing) commands in the
Security and Administration document, NN10163-611 for additional
information.
Note: When the status is “Info”, no action is required.
SPM500 Logs related to RM state change and activity switching (sparing).
SPM501 The clock mode has changed from sync, freerun, holdover, or
acquire to sync, freerun, holdover, or acquire.
SPM600 Logs generated when the MS changes synchronization modes and
is unable to notify an in-service SPM of the change.
SPM630 A successful sparing event has occurred.
SPM632 SPM REX Info. A log generated to record the start time of REX
operation.
SPM633 SPM REX Success. A successful REX operation has occurred.
Note: Node 500 logs report state changes on the CEM during REX
tests.
SPM642 An RM has exited Holdover24 state.
SPM644 The SRM Loss of BITS Redundancy (LOR) alarm has been cleared.
SPM645 Link Protocol & Messaging Interface Controller (LPMIC) Event
Report. This information log is generated on every CEM and
transported to the core. The log is generated periodically and
whenever the number of events crosses a threshold value.
SPM650 A successful in-service loading procedure has occurred.
SPM651 Logs generated when an in-service CEM/RM loading operation on
a circuit pack fails.
IW SPM IP fault management indicators
Log name
and number Explanation

IW SPM IP Fault Management
Nortel Networks Confidential
9
Copyright © 2004, Nortel Networks
SPM670 An SPM670 log is generated when a Health Monitor event clears.
SPM680 “Message Buffer Manager (MBM) buffer usage high” log message.
SPM681 “Message Buffer Manager (MBM) buffer pool low” log message.
SPM684 The Erase Flash information log indicates when the:
• Erase flash command is initiated,
• Erase flash command is completed,
• Erase flash command failed, and
• Erase flash command is rejected.
None
(This is an information log)
The log displays the CEM and SPM number on which the Erasefl
command is executed in addition to the userid of the craft who
initiated the command.
SPM700 DDM audit fail.
SPM701 DDM audit succeeded.
SPM702 DDM dynamic update failed.
VOIP800 Cyclical Redundancy Check (CRC) alignment error.
VOIP801 Undersize packets
VOIP802 Oversize packets.
VOIP803 Fragments.
VOIP804 Jabber.
VOIP805 Drop events.
VOIP806 Broadcast packets.
VOIP807 Jitter.
VOIP808 Latency
IW SPM IP fault management indicators
Log name
and number Explanation

NN10078-911 SN06.2 Preliminary 02.03 March 2004
10
Copyright © 2004, Nortel Networks Nortel Networks Confidential
Log NODE303 Format
Log NODE303 reports Wrong Application Data. This means the CEM
in slot 7 of the SPM is getting application data for CEM1, and the CEM
in slot 8 is getting application data for CEM 0. Node303 displays actions
required in response to this.
Note: The CEM cannot Return to Service with Wrong Application
Data.
Log NODE303 format
NODE 303 **<mmmdd hh:mm:ss> <ssdd> TBL Wrong
Application Data
Location: <NODE>
Trouble: CEM has Wrong Application Data
Action: Check and re-connect DS-512 Links correctly
Integrated Node Maintenance Detailed Information
Trouble Reason: DS-512 Link/Links may be
misconnected
Trouble Detail: CEM in Slot <slot_number(7 or 8)>
has Application Data of CEM <cem_number(0 or 1)>
VOIP809 Packets lost.
VOIP810 Decoder under run.
IW SPM IP fault management indicators
Log name
and number Explanation
Log NODE303 format
Field Description
TBL Indicates the trouble condition with the node.
Wrong Application
Data
Indicates that the information to follow is about Wrong
Application Data on the node.
Location Indicates the location of the peripheral event to which the
event applies.
Trouble Identifies the type of problem.
Action (User) Indicates the suggested user action.

IW SPM IP Fault Management
Nortel Networks Confidential
11
Copyright © 2004, Nortel Networks
Log SPM300 Format
Log SPM300 reports a device fault.
Log SPM300 format
SPM 300 ** <mmmdd hh:mm:ss> <ssdd> TBL Device Fault Report
Location: <NODE> <DEVICE>
Status : Alarm Raised
Problem Description: Critical
7A: Check messaging: Inactive CEM to RM loopback test timeout.
Action : Refer to NTPs for Alarm Clearing Procedures.
Cardlist : No Cardlist Available.dlist Available.
Trouble Reason Indicates the reason for the trouble condition.
Trouble Detail Provides more information about the trouble condition.
Log NODE303 format
Field Description
Log SPM300 format
Field Description
TBL Indicates the trouble condition with the node.
Device Fault Report Indicates that the information to follow is a device fault report
on the indicated device.
Location Indicates the location of the peripheral event and device to
which the event applies.
Status Indicates whether the alarm is raised or cleared
Problem
Description
Indicates the type of fault (Info, Non-critical, Critical) along with
the information on which CEM reported the fault and a text
string indicating what fault occurred.
Action (User) Indicates the suggested user action.
Cardlist Indicates the cards to be replaced.

NN10078-911 SN06.2 Preliminary 02.03 March 2004
12
Copyright © 2004, Nortel Networks Nortel Networks Confidential
Log SPM300 Problem Descriptions
The following table provides details for the problem descriptions in the
SPM300 log.
Log SPM332 Problem Descriptions
The following table provides details for the problem descriptions in the
SPM332 log.
SPM300 log problem descriptions
Failure reason Explanation
Check messaging:
Inactive CEM to RM
loopback test
timeout
This log indicates that there is a problem communicated to the
indicated RM from the inactive CEM. If the inactive CEM
becomes active while this condition exists, the RM will go
SYSB.
If this log occurs:
1) Collect sysbuf and /aer/display all from the inactive CEM
dshell.
2) Try to remlogin to the RM from the inactive CEM. If this fails,
remlogin to the RM from the active CEM. Collect the footprint
logs from the RM.
3) BSY/RTS the RM.
4) If the log reoccurs, BSY/RESETMOD/RTS the inactive
CEM.
SPM332 log - REX test failure
Failure reason Explanation
Node not in InSv/IsTb state Stability check on node failed.
<Node state> state
SPM REX aborted
CEM not in InSv state, RM
not in InSv/MANB/OFFL state
Stability check on <CPK type> <CPK no:> failed.
<CPK state> state
SPM REX aborted
CEM/RM having a non-state
alarm
Stability check on <CPK type> <CPK no:> failed.
<Alarmid> alarm
SPM REX aborted

IW SPM IP Fault Management
Nortel Networks Confidential
13
Copyright © 2004, Nortel Networks
Protection group having an
alarm
Stability check on <Prt grp type> <Prt grp id> failed.
<Alarmid> alarm
SPM REX aborted
Node having a non-state
alarm
Stability check on node failed.
<Alarmid> alarm
SPM REX aborted
CM SWACT has occurred in
the past 12 hours
CM SWACTed in the past 12 hours.
Stability database may be stale.
SPM REX aborted
CEMs do not have the same
running load
CEM 0 and CEM 1 do not have the same load.
SPM REX aborted
CEM/RM has been in
ISTB/SYSB state in the past
12 hours
Stability check on <CPK type> <CPK no> failed.
<CPK state> in the past 12 hours
SPM REX aborted
CEM/RM had an alarm raised
in the past 12 hours
Stability check on <CPK type> <CPK no> failed.
<Alarm name> alarm in the past 12 hours
SPM REX aborted
CEM/RM has been in
Patched/Loaded/Provisioned
within the stability period
Stability check on <CPK type> <CPK no> failed.
<Event name> in the past <stability period> hours
SPM REX aborted
Node had an alarm raised in
the past 12 hours
Stability check on node failed.
<Alarm id> alarm in the past 12 hours
SPM REX aborted
Protection group had an
alarm raised in the past 12
hours
Stability check on protection group failed.
<Grp type> <Grp id> <Alarm id> alarm in the past 12
hours
SPM REX aborted
SPM332 log - REX test failure
Failure reason Explanation

NN10078-911 SN06.2 Preliminary 02.03 March 2004
14
Copyright © 2004, Nortel Networks Nortel Networks Confidential
Long duration maintenance
activity in progress on RM
Long Maintenance activity on <RM type> <RM no>.
SPM REX aborted
Long duration maintenance
activity in progress on CEM
CEM <CEM no>. Long maintenance activity in
progress.
SPM REX aborted
REX test aborted by manual
ABTK command from CEM
MAP level
Rex test aborted manually.
SPM REX aborted
REX test aborted by the
system REX
REX test aborted by the system.
SPM REX aborted
SPM332 log - REX test failure
Failure reason Explanation

NN10078-911 SN06.2 Preliminary 02.03 March 2004
Nortel Networks Confidential
15
Copyright © 2004, Nortel Networks
Viewing alarms
Viewing alarms on the SPM
At the MAP level
1Post the SPM by typing
>MAPCI;MTC;PM;POST SPM <spm_no>
and pressing the Enter key.
where
spm_no
is the number of the SPM (0 to 85)
Example of a MAP screen:
2Display alarms on the RMs on the SPM by typing
>QUERYPM FLT
and pressing the Enter key.
3Display alarms on the SPM by typing
>LISTALM
and pressing the Enter key.

IW SPM IP Fault Management
Nortel Networks Confidential
16
Copyright © 2004, Nortel Networks
Clearing a CLKOOS alarm
Clearing a CLKOOS alarm
At the MAP terminal
1Access the PM level of the map screen by typing
>MAPCI;MTC;PM
and pressing the Enter key.
2Display all the inservice-trouble (ISTb) SPMs by typing
>DISP STATE ISTb SPM
and pressing the enter key.
3Post each ISTb SPM by typing
>POST SPM <spm_no>
and pressing the Enter key.
where
spm_no
is the number of the SPM (0 to 85)
Example of a MAP screen:

NN10078-911 SN06.2 Preliminary 02.03 March 2004
17
Copyright © 2004, Nortel Networks Nortel Networks Confidential
4Select the ISTb common equipment module (CEM) by typing
>SELECT CEM <cem_no>
where
cem_no
is the number of the ISTb CEM (0 or 1)
Example of a MAP screen:
5List the alarms on the CEM by typing
>LISTALM
and pressing the Enter key.
Example of a MAP screen:
6Record the number of each SPM exhibiting the CLKOOS
condition.
7Access the MTC level of the MAP screen by typing
>MAPCI;MTC
and pressing the Enter key.
SPM 11 CEM 0 Act ISTb
Loc : Row F FrPos 64 ShPos 6 ShId 0 Slot 7
Default Load: SPMLOAD
Clock:
Input Ref: Source: Current Mode:
SPM 11 CEM 0 Act ISTb
Loc : Row F FrPos 64 ShPos 6 ShId 0 Slot 7
Default Load: SPMLOAD
Clock:
Input Ref: Source: Current Mode:
ListAlm
ListAlm: SPM 11 CEM 0
SEVERITY ALARM ACTION
––––––––––––––––––––––––––––––
Critical None
Major CLKOOS RPT
Minor None
No_Alarm None

IW SPM IP Fault Management
Nortel Networks Confidential
18
Copyright © 2004, Nortel Networks
8Check the alarm banner and determine whether there is an MS
clock alarm.
9Clear the MS clock alarm using the appropriate alarm clearing
procedures. When you have completed the procedures, go to
Step 13.
10 List the alarms on the CEM by typing
>LISTALM
and pressing the Enter key.
11 Determine if sync fault condition is present on the downstream
equipment driving the OC3 into the SPM.
12 Clear sync fault condition on downstream equipment driving the
OC3 into the SPM.
Note: Once a sync fault condition is corrected, the SPM
clears the CLKOOS condition, but there will be a delay of
between 20 and 40 minutes while the SPM evaluates the
stability of the OC3 carrier frequency.
13 List the alarms on the CEM by typing
>LISTALM
and pressing the Enter key.
14 Force the CEMs to switch activity by typing
>PROT;FORCE;QUIT
If there is Do
an MS clock alarm step 9
not an MS clock alarm step 10
If the alarm list shows Do
None step 22
CLKOOS step 11
If sync fault condition is Do
present step 12
not present step 14
If the alarm list shows Do
None step 22
CLKOOS step 14

NN10078-911 SN06.2 Preliminary 02.03 March 2004
19
Copyright © 2004, Nortel Networks Nortel Networks Confidential
and pressing the enter key.
15 Select the active (A) CEM by typing
>SELECT CEM <cem_no>
and pressing the Enter key.
where
cem_no
is the number of the active CEM (0 or 1)
16 List the alarms on the CEM by typing
>LISTALM
and pressing the enter key.
17 Force the CEMs to switch activity by typing
>PROT;FORCE;QUIT
and pressing the Enter key.
18 Select the active (A) CEM by typing
>SELECT CEM <cem_no>
and pressing the Enter key.
where
cem_no
is the number of the active CEM (0 or 1)
19 List the alarms on the CEM by typing
>LISTALM
and pressing the Enter key.
Example of a MAP screen:
If the alarm list shows Do
None step 17
CLKOOS step 20

IW SPM IP Fault Management
Nortel Networks Confidential
20
Copyright © 2004, Nortel Networks
20 Return to the SPM level and list the alarms on the CEM by typing
>LISTALM
and pressing the Enter key.
21 For further assistance, contact the personnel responsible for the
next level of support.
22 You have completed this procedure. Return to the CI level of the
MAP screen by typing
>QUIT ALL
and pressing the Enter key.
If the alarm list shows Do
None step 22
CLKOOS step 20
If the alarm list shows Do
None step 22
CLKOOS step 21
SPM 11 CEM 0 Act ISTb
Loc : Row F FrPos 64 ShPos 6 ShId 0 Slot 7
Default Load: SPMLOAD
Clock:
Input Ref: Source: Current Mode:
ListAlm
ListAlm: SPM 11 CEM 0
SEVERITY ALARM ACTION
––––––––––––––––––––––––––––––
Critical None
Major CLKOOS RPT
Minor None
No_Alarm None