Cisco ONS 15454 DWDM User manual

CHAPTER
1-1
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
1
General Troubleshooting
This chapter provides procedures for troubleshooting the most common problems encountered when
operating a Cisco ONS 15454 DWDM shelf in ANSI or ETSI platforms. To troubleshoot specific alarms,
see Chapter 2, “Alarm Troubleshooting.” If you cannot find what you are looking for, contact Cisco
Technical Support (1-800-553-2447).
Alarms can occur even in those cards that are not explicitly mentioned in the Alarm sections. When an
alarm is raised, refer to its clearing procedure.
Note In this chapter, “ONS 15454” refers to both ANSI and ETSI versions of the platform unless otherwise
noted.
This chapter includes the following sections on network problems:
Note For dense wavelength division multiplexing (DWDM) network acceptance tests, refer to the
Cisco ONS 15454 DWDM Procedure Guide.
•1.1 Loopback Description, page 1-2—Explains in general terms the types of loopback tests (facility
and terminal) that can be run along with general loopback behavior and card loopback behavior.
•1.2 Troubleshooting MXP or TXP Circuit Paths With Loopbacks, page 1-7—Explains how to use
loopbacks tests described in “1.1 Loopback Description” section on page 1-2 to isolate trouble on
muxponder (MXP) and transponder (TXP) circuits.
•1.3 Troubleshooting DWDM Circuit Paths With ITU-T G.709 Monitoring, page 1-24—Explains
how to utilize performance monitoring (PM) and threshold crossing alerts (TCA) to locate signal
degrades on DWDM circuit paths.
The remaining sections describe symptoms, problems, and solutions that are categorized according to
the following topics:
•1.4 Using CTC Diagnostics, page 1-32—Explains how to perform card LED tests and download a
diagnostic file for Cisco Technical Support.
•1.5 Restoring the Database and Default Settings, page 1-35—Provides procedures for restoring
software data and restoring the node to the default setup.
•1.6 PC Connectivity Troubleshooting, page 1-36—Provides troubleshooting procedures for PC and
network connectivity to the ONS 15454.
•1.7 CTC Operation Troubleshooting, page 1-41—Provides troubleshooting procedures for
Cisco Transport Controller (CTC) login or operation problems.

1-2
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
Chapter 1 General Troubleshooting
1.1 Loopback Description
•1.8 Timing, page 1-51—Provides troubleshooting procedures for circuit creation and error
reporting as well as timing reference errors and alarms.
•1.9 Fiber and Cabling, page 1-54—Provides troubleshooting procedures for fiber and cabling
connectivity errors.
•1.10 Power Supply Problems, page 1-58—Provides troubleshooting procedures for rack power
supply problems.
•1.11 Power Up Problems for Node and Cards, page 1-60—Provides troubleshooting procedures for
shelf and card power supply problems.
•1.12 Network Level (Internode) Problems, page 1-60—Provides troubleshooting procedures for
problems between nodes, such as fiber cuts and optical channel network connection (OCHNC)
circuit creation failure.
•1.13 Node Level (Intranode) Problems, page 1-84—Provides troubleshooting procedures for
variable optical attenuator (VOA) startup problems as well as internal VOA control loop problems
in the 32DMX-C and 32DMX-L cards that occur due to counter-propagating light.
1.1 Loopback Description
Use loopbacks and hairpin circuits to test newly created circuits before running live traffic or to logically
locate the source of a network failure. All ONS 15454 and ONS 15454 SDH TXP and MXP cards allow
loopbacks and hairpin test circuits. Other cards do not allow loopbacks, including OPT-BST, OPT-PRE,
OSC-CSM, AD-xB-xx.x, and AD-xC-xx.x cards.
To create a loopback on an ANSI or SONET port, the port must be in the Out-of-Service and
Management, Maintenance (OOS-MA,MT) service state. After you create the loopback, the service state
becomes Out-of-Service and Management, Loopback and Maintenance (OOS-MA,LPBK & MT).
To create a loopback on an SDH or ETSI port, the port must be in the Locked, maintenance
administrative state and the Locked-Enabled, loopback & maintenance administrative state.
Caution Facility or terminal loopbacks can be service-affecting. To protect traffic, apply a lockout or Force
switch to the target loopback port. Basic directions for these procedures exist in Chapter 2, “Alarm
Troubleshooting.” For more information about these operations, refer to the “Maintain the Node” chapter
in the Cisco ONS 15454 DWDM Procedure Guide.
Note In CTC, a facility loopback is sometimes called “facility (line)” loopback, and a terminal loopback is
sometimes called a “terminal (inward)” loopback. This is done to indicate the terminating direction of
the signal: a facility loopback is sent outward toward the span, whereas a terminal loopback is redirected
inward toward its originating port.
1.1.1 Facility Loopbacks
The following sections give general information about facility loopback operations and specific
information about ONS 15454 or ONS 15454 SDH card loopback activity.

1-3
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
Chapter 1 General Troubleshooting
1.1.1 Facility Loopbacks
1.1.1.1 General Behavior
A facility loopback tests the line interface unit (LIU) of a card, the electrical interface assembly (EIA),
and related cabling. After applying a facility loopback on a port, use a test set to run traffic over the
loopback. A successful facility loopback isolates the LIU, the EIA, or the cabling plant as the potential
cause of a network problem.
To test a card LIU, connect an optical test set to a trunk or client port and perform a facility loopback.
Alternately, use a loopback or hairpin circuit on a card that is farther along the circuit path. For example,
Figure 1-1 shows a facility loopback at a trunk port and at a client port on a TXP card.
Figure 1-1 Facility Loopback Path on a Near-End Transponder Card
Caution Before performing a facility loopback on a TXP card, be sure that the card contains at least two data
communications channel (DCC) paths to the node where the card is installed. A second DCC provides
a nonlooped path to log into the node after the loopback is applied, enabling you to remove the facility
loopback. Ensuring a second DCC is not necessary if you are directly connected to the node containing
the loopback card.
Caution Ensure that the facility being loopbacked is not being used by the node for line timing. If it is, a timing
loop will be created.
1.1.1.2 Card Behavior
Port loopbacks either terminate or bridge the loopback signal. All MXP and TXP facility loopbacks are
terminated as shown in Table 1-1.
TXP
Test Set
TXP
145758
Client port
Trunk port
Client port
Trunk port
Test Set
Facility loopback at trunk port
Facility loopback at client port

1-4
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
Chapter 1 General Troubleshooting
1.1.2 Terminal Loopbacks
When a port terminates a facility loopback signal, the signal only loops back to the originating port and
is not transmitted downstream. When a port bridges a loopback signal, the signal loops back to the
originating port and is also transmitted downstream.
Note In Table 1-1, no alarm indication signal (AIS) is injected if the signal is bridged. If the signal is
terminated, an applicable AIS is injected downstream.
The loopback itself is listed in the Conditions window. For example, the window would list the
LPBKFACILITY condition for a tested port. (The Alarms window would show the AS-MT condition
which means that alarms are suppressed on the facility during loopback.)
With a client-side SONET or ANSI facility loopback, the client port service state is OOS-MA,LPBK &
MT. However, any remaining client and trunk ports can be in any other service state. For SONET or
ANSI cards in a trunk-side facility loopback, the trunk port service state is OOS-MA,LPBK & MT and
the remaining client and trunk ports can be in any other service state.
With a client-side SDH or ESTI facility loopback, the client port is in the Locked-enabled,maintenance
& loopback service state. However, the remaining client and trunk ports can be in any other service state.
For MXP and TXP cards in a SDH or ETSI trunk-side facility loopback, the trunk port is in the
Locked-enabled,maintenance & loopback service state and the remaining client and trunk ports can be
in any other service state.
1.1.2 Terminal Loopbacks
The following sections give general information about terminal loopback operations and specific
information about ONS 15454 card loopback activity.
Table 1-1 DWDM Card Facility Loopback Behavior
Card/Port Facility Loopback Signal
TXP_MR_10E/TXP_MR_10E_C/TXP_MR_10E_L client
ports
Bridged
TXP_MR_10E/TXP_MR_10E_C/TXP_MR_10E_L trunk
ports
Terminated
TXP_MR_2.5G/TXPP_MR_2.5G client ports Terminated
TXP_MR_2.5G/TXPP_MR_2.5G trunk ports Terminated
MXP_2.5G_10E_C/MXP_2.5G_10E_L client ports Bridged
MXP_2.5G_10E_C/MXP_2.5G_10E_L trunk ports Terminated
MXP_MR_10DME client ports Bridged
MXP_MR_10DME trunk ports Terminated
MXP_MR_2.5G/MXPP_MR_2.5G client ports Bridged
MXP_MR_2.5G/MXPP_MR_2.5G trunk ports Terminated

1-5
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
Chapter 1 General Troubleshooting
1.1.2 Terminal Loopbacks
1.1.2.1 General Behavior
A terminal loopback tests a circuit path as it passes through a TXP or MXP card and loops back. For
example, as shown in Figure 1-2, there are two types of terminal loopbacks shown for a TXP card.
The first is a terminal loopback at the client port. In this situation, the test set traffic comes in through
the TXP trunk port, travels through the card, and turns around because of the terminal loopback in effect
on the card just before it reaches the LIU of the client port. The signal is then sent back through the card
to the trunk port and back to the test set.
The second is a terminal loopback at the trunk port. In this situation, the test set traffic comes in through
the TXP client port, travels through the card, and turns around because of the terminal loopback in effect
on the card just before it reaches the LIU of the trunk port. The signal is then sent back through the card
to the client port and back to the test set.
This test verifies that the terminal circuit paths are valid, but does not test the LIU on the TXP card.
Figure 1-2 Terminal Loopback on a TXP Card
1.1.2.2 Card Behavior
ONS 15454 and ONS 15454 SDH terminal port loopbacks can either terminate or bridge the signal. TXP
terminal loopbacks are terminated as shown in Table 1-2. During terminal loopbacks, if a port terminates
a terminal loopback signal, the signal only loops back to the originating port and is not transmitted
downstream. If the port bridges a loopback signal, the signal loops back to the originating port and is
also transmitted downstream. Client card terminal loopback bridging and terminating behaviors are
listed in Table 1-2.
TXP
Test Set
TXP
145760
Client port
Trunk port
Client port
Trunk port
Terminal loopback at client port
Terminal loopback at trunk port
Test Set

1-6
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
Chapter 1 General Troubleshooting
1.1.2 Terminal Loopbacks
Note In Table 1-2, no AIS signal is injected if the signal is bridged. If the signal is terminated, an applicable
AIS is injected downstream.
The MXP and TXP trunk and client ports can simultaneously maintain different service states:
•For SONET or ANSI TXP and TXPP cards with a client-side terminal loopback, the client port is in
the OOS-MA,LPBK & MT service state and trunk port must be in IS-NR service state.
•For SONET or ANSI MXP and MXPP cards with a client-side terminal loopback, the client port is
in the OOS-MA,LPBK & MT service state and the remaining client and trunk ports can be in any
service state.
•In SONET or ANSI MXP or TXP trunk-side terminal loopbacks, the trunk port is in the
OOS-MA,LPBK & MT service state and the client ports must be in IS-NR service state for complete
loopback functionality. A terminal loopback affects all client ports because it is performed on the
aggregate signal.
•For SDH or ETSI TXP and TXPP client-side facility loopbacks, the client port is in the
Locked-enabled,maintenance & loopback service state and the trunk port must be in
Unlocked-enabled service state.
•For SDH or ETSI MXP and MXPP cards with a client-side terminal loopback, the client port is in
the Locked-enabled,maintenance & loopback service state and remaining client and trunk ports can
be in any service state.
•In SDH and ETSI MXP or TXP trunk-side terminal loopbacks, the trunk port is in the
Locked-enabled,maintenance & loopback service state and the client ports must be in
Unlocked-enabled service state for complete loopback functionality. A facility loopback affects all
client ports because it is performed on the aggregate signal.
The loopback itself is listed in the Conditions window. For example, the window would list the
LPBKTERMINAL condition or LPBKFACILITY condition for a tested port. (The Alarms window
would show the AS-MT condition, which indicates that all alarms are suppressed on the port during
loopback testing.)
Table 1-2 Client Card Terminal Loopback Behavior
Card/Port Terminal Loopback Signal
TXP_MR_10E/TXP_MR_10E_C/TXP_MR_10E_L client ports Bridged
TXP_MR_10E/TXP_MR_10E_C/TXP_MR_10E_L trunk ports Bridged
TXP_MR_2.5G/TXPP_MR_2.5G client ports Bridged
TXP_MR_2.5G/TXPP_MR_2.5G trunk ports Bridged
MXP_2.5G_10E_C/MXP_2.5G_10E_L client ports Bridged
MXP_2.5G_10E_C/MXP_2.5G_10E_L trunk ports Bridged
MXP_MR_10DME client ports Bridged
MXP_MR_10DME trunk ports Bridged
MXP_MR_2.5G/MXPP_MR_2.5G client ports Terminated
MXP_MR_2.5G/MXPP_MR_2.5G trunk ports Bridged

1-7
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
Chapter 1 General Troubleshooting
1.2 Troubleshooting MXP or TXP Circuit Paths With Loopbacks
1.2 Troubleshooting MXP or TXP Circuit Paths With Loopbacks
Facility loopbacks and terminal loopbacks are often used together to test the circuit path through the
network or to logically isolate a fault. Performing a loopback test at each point along the circuit path
systematically isolates possible points of failure. MXP or TXP loopback tests differ from other testing
in that loopback testing does not require circuit creation. MXP or TXP client ports are statically mapped
to the trunk ports so no signal needs to traverse the cross-connect card (in a circuit) to test the loopback.
You can use these procedures on transponder cards (TXP, TXPP) or muxponder cards (MXP, MXPP)
cards. The example in this section tests an MXP or TXP circuit on a three-node bidirectional line
switched ring (BLSR) or multiplex section-shared protection ring (MS-SPRing). Using a series of
facility loopbacks and terminal loopbacks, the example scenario traces the circuit path, tests the possible
failure points, and eliminates them. The logical progression contains six network test procedures:
Note MXP and TXP card client ports do not appear when you click the Maintenance > Loopback tab unless
they have been provisioned. Do this in the card view by clicking the Provisioning > Pluggable Port
Modules tab. For information about provisioning client ports, refer to the Cisco ONS 15454 DWDM
Procedure Guide.
Note The test sequence for your circuits will differ according to the type of circuit and network topology.
1. A facility loopback on the source-node MXP or TXP port
2. A terminal loopback on the source-node MXP or TXP port
3. A facility loopback on the intermediate-node MXP or TXP port
4. A terminal loopback on the intermediate-node MXP or TXP port
5. A facility loopback on the destination-node MXP or TXP port
6. A terminal loopback on the destination-node MXP or TXP port
Note Facility and terminal loopback tests require on-site personnel.
1.2.1 Perform a Facility Loopback on a Source-Node MXP or TXP Port
This facility loopback test is performed on the node source port in the network circuit. In the testing
situation used in this example, the source muxponder or transponder port under test is located in the
source node. Facility loopback can be performed at the trunk port or at a client port. Completing a
successful facility loopback on this port isolates the source MXP or TXP port as a possible failure point.
Figure 1-3 shows the facility loopback examples on source ONS node TXP ports (client and trunk).

1-8
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
Chapter 1 General Troubleshooting
1.2.1 Perform a Facility Loopback on a Source-Node MXP or TXP Port
Figure 1-3 Facility Loopback on a Circuit Source MXP or TXP Port
Caution Performing a loopback on an in-service circuit is service-affecting.
Note Facility loopbacks require on-site personnel.
Complete the “Create the Facility Loopback on the Source-Node MXP or TXP Port” procedure on
page 1-8.
Create the Facility Loopback on the Source-Node MXP or TXP Port
Step 1 Connect an optical test set to the port you are testing.
Note For specific procedures to connect, set up, and use the test set equipment, consult the
manufacturer.
Use appropriate cabling to attach the transmit (Tx) and receive (Rx) terminals of the optical test set to
the port you are testing. The Tx and Rx terminals connect to the same port.
Step 2 Adjust the test set accordingly. (Refer to manufacturer instructions for test set use.)
TXP
Destination
ONS Node
TXPTXP
Source
ONS Node
Source
ONS Node
Test Set
TXP
Destination
ONS Node
TXPTXP
145757
Client port
Trunk port
Client port
Test Set
Trunk port

1-9
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
Chapter 1 General Troubleshooting
1.2.1 Perform a Facility Loopback on a Source-Node MXP or TXP Port
Step 3 In node view (single-shelf mode) or shelf view (multishelf mode), double-click the card to display the card
view.
Step 4 Click the Maintenance > Loopback tabs.
Step 5 Choose OOS,MT (or locked,maintenance) from the Admin State column for the port being tested. If
this is a multiport card, select the appropriate row for the desired port.
Step 6 Choose Facility (Line) from the Loopback Type column for the port being tested. If this is a multiport
card, select the appropriate row for the desired port.
Step 7 Click Apply.
Step 8 Click Yes in the confirmation dialog box.
Note It is normal for the “LPBKFACILITY (ESCON)” condition on page 2-95, “LPBKFACILITY
(FC)” condition on page 2-95, “LPBKFACILITY (GE)” condition on page 2-96,
“LPBKFACILITY (ISC)” condition on page 2-96or the “LPBKFACILITY (TRUNK)” condition
on page 2-97 to appear during loopback setup. The condition clears when you remove the
loopback.
Step 9 Complete the “Test and Clear the MXP or TXP Facility Loopback Circuit” procedure on page 1-9.
Test and Clear the MXP or TXP Facility Loopback Circuit
Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.
Step 2 Examine the traffic received by the test set. Look for errors or any other signal information that the test
set is capable of indicating.
Step 3 If the test set indicates no errors, no further testing is necessary with the facility loopback. Clear the
facility loopback:
a. Click the Maintenance > Loopback tabs.
b. Choose None from the Loopback Type column for the port being tested.
c. Choose the appropriate state to place the port in service, out of service and disabled, out of service
for maintenance, or automatically in service from the Admin State column for the port being tested.
d. Click Apply.
e. Click Yes in the confirmation dialog box.
Step 4 Complete the “Test the MXP or TXP Card” procedure on page 1-9.
Test the MXP or TXP Card
Step 1 Complete the “Physically Replace a Card” procedure on page 2-172 for the suspected bad card and
replace it with a known-good one.

1-10
Cisco ONS 15454 DWDM Troubleshooting Guide, R7.0.1
78-17742-02
Chapter 1 General Troubleshooting
1.2.2 Perform a Terminal Loopback on a Source-Node MXP or TXP Port
Warning High-performance devices on this card can get hot during operation. To remove the card, hold it by
the faceplate and bottom edge. Allow the card to cool before touching any other part of it or before
placing it in an antistatic bag. Statement 201
Caution Removing a card that currently carries traffic on one or more ports can cause a traffic hit. To avoid this,
perform an external switch if a switch has not already occurred. Refer to the procedures in the
“2.8.1 Protection Switching, Lock Initiation, and Clearing” section on page 2-167. For more
information, refer to the “Maintain the Node” chapter in the Cisco ONS 15454 DWDM Procedure Guide.
Step 2 Resend test traffic on the loopback circuit with a known-good card installed.
Step 3 If the test set indicates no errors, the problem was probably the defective card. Return the defective card
to Cisco through the Return Materials Authorization (RMA) process. Contact Cisco Technical Support
(1-800-553-2447).
Step 4 Complete the “Physically Replace a Card” procedure on page 2-172 for the faulty card.
Step 5 Clear the facility loopback:
a. Click the Maintenance > Loopback tabs.
b. Choose None from the Loopback Type column for the port being tested.
c. Choose the appropriate state to place the port in service, out of service and disabled, out of service
for maintenance, or automatically in service from the Admin State column for the port being tested.
d. Click Apply.
e. Click Yes in the confirmation dialog box.
Step 6 Complete the “1.2.2 Perform a Terminal Loopback on a Source-Node MXP or TXP Port” procedure on
page 1-10.
1.2.2 Perform a Terminal Loopback on a Source-Node MXP or TXP Port
The terminal loopback test is performed on the node source MXP or TXP muxponder or transponder
port. For the circuit in this example, it is the source TXP trunk port or a client port in the source node.
Completing a successful terminal loopback to a node source port verifies that the circuit is through the
source port. Figure 1-4 shows an example of a terminal loopback on a source TXP port and a client TXP
port.