Cisco ONS 15454 User manual

CHAPTER
1-1
Cisco ONS 15454 Troubleshooting Guide, R4.6
November 2004
1
General Troubleshooting
This chapter provides procedures for troubleshooting the most common problems encountered when
operating a Cisco ONS 15454. To troubleshoot specific ONS 15454 alarms, see Chapter 2, “Alarm
Troubleshooting.” If you cannot find what you are looking for, contact the Cisco Technical Assistance
Center (1 800 553-2447).
Note The terms "UnidirectionalPath Switched Ring" and "UPSR" may appear in Cisco literature. These terms
do not refer to using Cisco ONS 15xxx products in a unidirectional path switched ring configuration.
Rather, these terms, as well as "Path Protected Mesh Network" and "PPMN," refer generally to Cisco's
path protection feature, which may be used in any topological network configuration. Cisco does not
recommend using its path protection feature in any particular topological network configuration.
This chapter includes the following sections on network problems:
•Network Troubleshooting Tests—Describes loopbacks and hairpin circuits, which you can use to
test circuit paths through the network or logically isolate faults.
Note For network acceptance tests, refer to the Cisco ONS 15454 Procedure Guide.
•Identify Points of Failure on a DS-N Circuit Path—Explains how to perform the tests described in
the “1.1 Network Troubleshooting Tests” section on a DS-N circuit.
•Using the DS3XM-6 Card FEAC (Loopback) Functions—Describes the far-end alarm and control
(FEAC) functions on the DS3XM-6 card.
•Identify Points of Failure on an Optical Circuit Path—Explains how to perform the tests described
in the “1.1 Network Troubleshooting Tests” section on an OC-N circuit.
The remaining sections describe symptoms, problems, and solutions that are categorized according to
the following topics:
•Restoring the Database and Default Settings—Provides procedures for restoring software data and
restoring the node to the default setup.
•PC Connectivity Troubleshooting—Provides troubleshooting procedures for PC and network
connectivity to the ONS 15454.
•CTC Operation Troubleshooting—Provides troubleshooting procedures for CTC login or operation
problems.
•Circuits and Timing—Provides troubleshooting procedures for circuit creation and error reporting
as well as timing reference errors and alarms.

1-2
Cisco ONS 15454 Troubleshooting Guide, R4.6 November 2004
Chapter 1 General Troubleshooting
1.1 Network Troubleshooting Tests
•Fiber and Cabling—Provides troubleshooting procedures for fiber and cabling connectivity errors.
•Power and LED Tests—Provides troubleshooting procedures for power supply and LED indicator
problems.
1.1 Network Troubleshooting Tests
Use loopbacks and hairpin circuits to test newly created SONET circuits before running live traffic or to
logically locate the source of a network failure. All ONS 15454 OC-N cards, transponder cards (TXP,
TXPP), muxponder (MXP) cards, and G-Series Ethernet cards allow loopbacks and hairpin test circuits.
Other cards do not allow loopbacks. These include ONS E-Series Ethernet, ML-Series Ethernet, and
DWDM cards such as OPT-BST, OPT-PRE, OSC-CSM, band add-drop cards, and channel add-drop
cards.
Caution Facility (line) or terminal loopbacks can be service-affecting. To protect traffic, apply a lockout or Force
switch to the target loopback port. For more information about these operations, refer to the
Cisco ONS 15454 Procedure Guide.
Caution On OC-N cards, a facility (line) loopback applies to the entire card and not an individual circuit. Exercise
caution when using loopbacks on an OC-N card carrying live traffic.
A facility (line) loopback tests the line interface unit (LIU) of a card, the EIA (electrical interface
assembly), and related cabling. After applying a facility loopback on a port, use a test set to run traffic
over the loopback. A successful facility loopback isolates the LIU, the EIA, or the cabling plant as the
potential cause of a network problem. Figure 1-1 shows a facility loopback on a DS-N card.
Figure 1-1 Facility (Line) Loopback Process on a DS-N Card
To test the LIU on an OC-N card, connect an optical test set to the OC-N port and perform a facility (line)
loopback or use a loopback or hairpin circuit on a card that is farther along the circuit path. Figure 1-2
shows a facility loopback on an OC-N card.
DS-N OC-NXC
5
5261
Test Set

1-3
Cisco ONS 15454 Troubleshooting Guide, R4.6
November 2004
Chapter 1 General Troubleshooting 1.1 Network Troubleshooting Tests
Figure 1-2 Facility (Line) Loopback Process on an OC-N Card
Caution Before performing a facility (line) loopback on an OC-N card, be sure the card contains at least two data
communications channel (DCC) paths to the node where the card is installed. A second DCC provides
a nonlooped path to log into the node after the loopback is applied, enabling you to remove the facility
loopback. Ensuring a second DCC is not necessary if you are directly connected to the ONS 15454
containing the loopback OC-N card.
A terminal loopback tests a circuit path as it passes through the cross-connect card (XC10G) and loops
back from the card with the loopback. Figure 1-3 shows a terminal loopback on an OC-N card. (Ethernet,
transponder, and muxponder loopbacks follow the same model.) The test-set traffic comes into the DS-N
card and goes through the cross-connect card to the OC-N card. The terminal loopback on the OC-N card
turns the signal around before it reaches the LIU and sends it back through the cross-connect card to the
DS-N card. This test verifies that the cross-connect card and terminal circuit paths are valid, but does
not test the LIU on the OC-N card.
Setting a terminal loopback on the G-Series card might not stop the Tx Packets counter or the Rx Packet
counters on the CTC card-level view Performance > Statistics page from increasing. The counters can
increment even though the loopbacked port has temporarily disabled the transmit laser and is dropping
any received packets.
The Tx Packet statistic continues to increment because the statistic is not based on the packets
transmitted by the transmit (Tx) laser but on the Tx signal inside the G-Series card. In normal in-service
port operation, the Tx signal being recorded does result in the Tx laser transmitting packets, but in a
terminal loopback this signal is being looped back within the G-Series card and does not result in the Tx
laser transmitting packets.
The Rx Packet counter might also continue to increment when the G-Series card is in terminal loopback.
Receive (Rx) packets from any connected device are dropped and not recorded, but the internally looped
back packets follow the G-Series card’s normal receive path and register on the Rx Packet counter.
Figure 1-3 Terminal Loopback Process on an OC-N Card
DS-N OC-NXC
Test Set
78260
5
5266
OC-N
Test Set
XCDS-N

1-4
Cisco ONS 15454 Troubleshooting Guide, R4.6 November 2004
Chapter 1 General Troubleshooting
1.1 Network Troubleshooting Tests
Figure 1-4 shows a terminal loopback on a DS-N card. The test-set traffic comes in on the OC-N card
and goes through the cross-connect card to the DS-N card. The terminal loopback on the DS-N card turns
the signal around before it reaches the LIU and sends it back through the cross-connect card to the OC-N
card. This test verifies that the cross-connect card and terminal circuit paths are valid, but does not test
the LIU on the DS-N card.
Figure 1-4 Terminal Loopback Process on a DS-N Card
ONS 15454 port loopbacks either terminate or bridge the loopback signal. In the ONS 15454 system, all
optical, electrical, Ethernet, and TXP/MXP facility loopbacks are terminated as shown in Table 1-1.
During terminal loopbacks, some ONS cards bridge the loopback signal while others terminate it.
If an optical, electrical, Ethernet, and TXP/MXP port terminates a terminal or facility loopback signal,
this means that the signal only loops back to the originating port and is not transmitted downstream. If
the port bridges a loopback signal, the signal loops back to the originating port and is also transmitted
downstream.
All ONS 15454 card bridging and terminating behaviors are listed in Table 1-1. When a port on a card
in the left column of this table originates a terminal or facility loopback, the signal behaves as listed in
the middle and right columns.
Note In Table 1-1, no AIS signal is injected if the signal is bridged. If the signal is terminated, an applicable
AIS is injected downstream for all cards except Ethernet cards.
DS-N
Test Set
XC OC-N
78259
Table 1-1 DS-N, OC-N, EC-N, and TXP/MXP Card Loopback Behavior
Card/Port Terminal loopback signal Facility loopback signal
DS-1 Terminated Terminated
DS-3 Bridged Terminated
DS-N transmux Bridged Terminated
All OC-N cards Bridged Terminated
EC-1 Bridged Terminated
G-Series Ethernet Terminated1
1. G-Series Ethernet terminal loopback is terminated and Ethernet transmission is disabled. No AIS is
inserted for Ethernet, but a TPTFAIL alarm is raised on the far-end Ethernet port.
Terminated2
2. G-Series facility loopback is terminated and no AIS is sent downstream. However, the Cisco Link
Integrity signal continues to be sent downstream.
TXP_MR_2.5G Bridged Terminated
TXP_MR_10G Bridged Terminated3
MXP_2.5G_10G Bridged Terminated4

1-5
Cisco ONS 15454 Troubleshooting Guide, R4.6
November 2004
Chapter 1 General Troubleshooting 1.1 Network Troubleshooting Tests
The loopback itself is listed in the Alarms window. For example, the window would list the
LPBKTERMINAL condition or LPBKFACILITY condition for a tested port.
In addition to the Alarms window listing, the following behaviors occur:
•If a DS-N, OC-N, or EC-1 port is placed in out of service (OOS) state, it injects an AIS signal
upstream and downstream.
•If a DS-N, OC-N, or EC-1 port is placed in out of service auto in-service (OOS_AINS) state or in
the out of service maintenance (OOS_MT) state before loopback testing, the port clears the AIS
signal upstream and downstream unless there is a service-affecting defect that would also cause an
AIS signal to be injected. For more information about placing ports into alternate states for testing,
refer to the Cisco ONS 15454 Procedure Guide.
•If a TXP_MR_2.5G port is placed in the out of service maintenance (OOS_MT) state before
loopback testing, the port raises an Alarms Suppressed for Maintenance (AS-MT) condition during
the loopback on the client side port.
•If a TXP_MR_10G port is placed in the out of service maintenance (OOS_MT) state before
loopback testing, the port raises a Loss of Signal (LOS) alarm on the DWDM-side port during the
loopback.
•If an MXP_2.5_10G port is placed in the out of service maintenance (OOS_MT) state before
loopback testing, the port raises a Loss of Signal (LOS) alarm on the DWDM-side port during the
loopback.
Bridged DS-N and OC-N terminal loopback examples are shown in Figure 1-5 and Figure 1-6.
Figure 1-5 Terminal Loopback on a DS-N Card with Bridged Signal
Figure 1-6 Terminal Loopback on an OC-N Card with Bridged Signal
3. A facility loopback on the client or trunk squelches the far end.
4. A facility loopback when the card is in transparent mode on the client or trunk squelches the far end.
Source
ONS Node
Destination
ONS Node
XCDS-N OC-N DS-NXCOC-N
Test Se
t
Test Set
1
15049
Source
ONS Node
Destination
ONS Node
XCOC-N OC-N OC-NXCOC-N
Test Se
t
Test Set
115050

1-6
Cisco ONS 15454 Troubleshooting Guide, R4.6 November 2004
Chapter 1 General Troubleshooting
1.2 Identify Points of Failure on a DS-N Circuit Path
A hairpin circuit brings traffic in and out on a DS-N port rather than sending the traffic onto the OC-N
card. A hairpin loops back only the specific STS or VT circuit and does not cause an entire OC-N port
to loop back, thus preventing a drop of all traffic on the OC-N port. The hairpin allows you to test a
specific STS or VT circuit on nodes running live traffic. Figure 1-9 shows the hairpin circuit process on
a DS-N card.
Figure 1-7 Hairpin Circuit Process on a DS-N Card
A cross-connect loopback tests a circuit path as it passes through the cross-connect card and loops back
to the port being tested. Testing and verifying circuit integrity often involves taking down the whole line;
however, a cross-connect loopback allows you to create a loopback on any embedded channel at
supported payloads at the STS-1 granularity and higher. For example, you can loop back a single STS-1,
STS-3c, STS-6c, etc. on an optical facility (line) without interrupting the other STS circuits.
The following restrictions apply to cross-connect loopbacks:
•You can create a cross-connect loopback on all working or protect optical ports unless the protect
port is used in a 1+1 protection group and is in working mode.
•If a terminal or facility loopback exists on a port, you cannot use the cross-connect loopback.
1.2 Identify Points of Failure on a DS-N Circuit Path
Facility (line) loopbacks, terminal (inward) loopbacks, and hairpin circuits are often used totest a circuit
path through the network or to logically isolate a fault. Performing a loopback test at each point along
the circuit path systematically isolates possible points of failure.
The example in this section tests a DS-N circuit on a two-node, bidirectional line switched ring (BLSR).
Using a series of facility loopbacks, terminal loopbacks, and hairpins, the path of the circuit is traced
and the possible points of failure are tested and eliminated. A logical progression of five network test
procedures apply to this sample scenario:
Note The test sequence for your circuits will differ according to the type of circuit and network topology.
1. A facility (line) loopback on the source node DS-N
2. A hairpin on the source node DS-N
3. A terminal (inward) loopback on the destination node DS-N
4. A hairpin on the destination node DS-N
5. A facility (line) loopback on the destination DS-N
Note All loopback tests require on-site personnel.
55485
OC-N
Test Set
XCDS-N

1-7
Cisco ONS 15454 Troubleshooting Guide, R4.6
November 2004
Chapter 1 General Troubleshooting 1.2.1 Perform a Facility (Line) Loopback on a Source DS-N Port
1.2.1 Perform a Facility (Line) Loopback on a Source DS-N Port
The facility (line) loopback test is performed on the node source port in the network circuit, in this
example, the DS-N port in the source node. Completing a successful facility (line) loopback on this port
isolates the cabling, the DS-N card, and the EIA as possible failure points. Figure 1-10 shows an example
of a facility loopback on a source DS-N port.
Figure 1-8 A Facility (Line) Loopback on a Circuit Source DS-N Port
A hairpin circuit brings traffic in and out on a DS-N port rather than sending the traffic onto the OC-N
card. A hairpin loops back only the specific STS or VT circuit and does not cause an entire OC-N port
to loop back, thus preventing a drop of all traffic on the OC-N port. The hairpin allows you to test a
specific STS or VT circuit on nodes running live traffic. Figure 1-9 shows the hairpin circuit process on
a DS-N card.
Figure 1-9 Hairpin Circuit Process on a DS-N Card
A cross-connect loopback tests a circuit path as it passes through the cross-connect card and loops back
to the port being tested. Testing and verifying circuit integrity often involves taking down the whole line;
however, a cross-connect loopback allows you to create a loopback on any embedded channel at
supported payloads at the STS-1 granularity and higher. For example, you can loop back a single STS-1,
STS-3c, STS-6c, etc. on an optical facility (line) without interrupting the other STS circuits.
The following restrictions apply to cross-connect loopbacks:
•You can create a cross-connect loopback on all working or protect optical ports unless the protect
port is used in a 1+1 protection group and is in working mode.
•If a terminal or facility loopback exists on a port, you cannot use the cross-connect loopback.
Source
ONS Node Destination ONS Node
55262
DS-N OC-NXC OC-N DS-NXC
Test Set
55485
OC-N
Test Set
XCDS-N

1-8
Cisco ONS 15454 Troubleshooting Guide, R4.6 November 2004
Chapter 1 General Troubleshooting
1.3 Identify Points of Failure on a DS-N Circuit Path
1.3 Identify Points of Failure on a DS-N Circuit Path
Facility (line) loopbacks, terminal (inward) loopbacks, and hairpin circuits are often used totest a circuit
path through the network or to logically isolate a fault. Performing a loopback test at each point along
the circuit path systematically isolates possible points of failure.
The example in this section tests a DS-N circuit on a two-node, bidirectional line switched ring (BLSR).
Using a series of facility loopbacks, terminal loopbacks, and hairpins, the path of the circuit is traced
and the possible points of failure are tested and eliminated. A logical progression of five network test
procedures apply to this sample scenario:
Note The test sequence for your circuits will differ according to the type of circuit and network topology.
West to east direction (left to right):
1. A facility (line) loopback on the source node DS-N
2. A hairpin on the source node DS-N
3. An XC loopback on the destination node OC-N STS (carrying the DS-N circuit)
4. A terminal (inward) loopback on the destination node DS-N
East to west direction (right to left):
1. A facility (line) loopback on the destination node DS-N
2. A hairpin on the destination node DS-N
3. An XC loopback on the source node OC-N STS (carrying the DS-N circuit)
4. A terminal (inward) loopback on the source node DS-N
Note All loopback tests require on-site personnel.
1.3.1 Perform a Facility (Line) Loopback on a Source DS-N Port (West to East)
The facility (line) loopback test is performed on the node source port in the network circuit, in this
example, the DS-N port in the source node. Completing a successful facility (line) loopback on this port
isolates the cabling, the DS-N card, and the EIA as possible failure points. Figure 1-10 shows an example
of a facility loopback on a source DS-N port.
Figure 1-10 A Facility (Line) Loopback on a Circuit Source DS-N Port
Source
ONS Node
Destination
ONS Node
55262
DS-N OC-NXC OC-N DS-NXC
Test Set

1-9
Cisco ONS 15454 Troubleshooting Guide, R4.6
November 2004
Chapter 1 General Troubleshooting 1.3.1 Perform a Facility (Line) Loopback on a Source DS-N Port (West to East)
Caution Performing a loopback on an in-service circuit is service-affecting. To protect traffic, apply a lockout or
Force switch to the target loopback port. For more information about these operations, refer to the
Cisco ONS 15454 Procedure Guide.
Note DS-3 facility (line) loopbacks do not transmit an alarm indication signal (AIS) condition in the direction
away from the loopback. Instead of a DS-3 AIS, a continuance of the signal transmitted to the loopback
is provided.
Create the Facility (Line) Loopback on the Source DS-N Port
Step 1 Connect an electrical test set to the port you are testing.
Use appropriate cabling to attach the Tx and Rx terminals of the electrical test set to the EIA connectors
or DSx panel for the port you are testing. The Tx and Rx terminals connect to the same port. Adjust the
test set accordingly.
Step 2 In node view, double-click the DS-N card to open the card view.
Step 3 Depending upon the card type, click the Maintenance > Loopback tabs, Maintenance > DS1 tabs, or
Maintenance > DS3 tabs.
Step 4 Choose OOS_MT from the State column for the port being tested. If this is a multiport card, select the
appropriate row for the port being tested.
Step 5 Choose Facility (Line) from the Loopback Type column for the port being tested. If this is a multiport
card, select the appropriate row for the port being tested.
Step 6 Click Apply.
Step 7 Click Yes in the confirmation dialog box.
Note It is normal for the “LPBKFACILITY (DS1, DS3)” condition on page 2-141 to appear during
loopback setup. The condition clears when you remove the loopback.
Step 8 Complete the “Test and Clear the Facility (Line) Loopback Circuit” procedure on page 1-9.
Test and Clear the Facility (Line) Loopback Circuit
Step 1 If the test set is not already sending traffic, send test traffic on the loopback circuit.
Step 2 Examine the traffic received by the test set. Look for errors or any other signal information that the test
set is capable of indicating.
Step 3 If the test set indicates a good circuit, no further testing is necessary with the facility loopback.
Double-click the card to open the card view.
Step 4 Depending upon the card type, click the Maintenance > Loopback tabs, Maintenance > DS1 tabs, or
Maintenance > DS3 tabs.
Step 5 Choose None from the Loopback Type column for the port being tested.

1-10
Cisco ONS 15454 Troubleshooting Guide, R4.6 November 2004
Chapter 1 General Troubleshooting
1.3.1 Perform a Facility (Line) Loopback on a Source DS-N Port (West to East)
Step 6 Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.
Step 7 Click Apply.
Step 8 Click Yes in the confirmation dialog box.
Step 9 Complete the “Perform a Hairpin Test on a Source Node Port (West to East)” procedure on page 1-12.
If the test set indicates a faulty circuit, the problem might be a faulty DS-N card, faulty cabling from the
DS-N card to the DSx panel or the EIA, or a faulty EIA.
Step 10 Complete the “Test the DS-N Cabling” procedure on page 1-10.
Test the DS-N Cabling
Step 1 Replace the suspected bad cabling (the cables from the test set to the DSx panel or the EIA ports) with
a known-good cable.
If a known-good cable is not available, test the suspected bad cable with a test set. Remove the suspected
bad cable from the DSx panel or the EIA and connect the cable to the Tx and Rx terminals of the test
set. Run traffic to determine whether the cable is good or defective.
Step 2 Resend test traffic on the loopback circuit with a known-good cable installed. If the test set indicates a
good circuit, the problem was probably the defective cable.
Step 3 Replace the defective cable.
Step 4 In card view for the DS-N card, depending upon the type, click the Maintenance > Loopback tabs,
Maintenance > DS1 tabs, or Maintenance > DS3 tabs.
Step 5 Choose None from the Loopback Type column for the port being tested.
Step 6 Choose the appropriate state (IS, OOS, OOS_AINS) from the State column for the port being tested.
Step 7 Click Apply.
Step 8 Click Yes in the confirmation dialog box.
Step 9 Complete the “Perform a Hairpin Test on a Source Node Port (West to East)” procedure on page 1-12.
If the test set indicates a faulty circuit, the problem might be a faulty card or a faulty EIA.
Step 10 Complete the “Test the DS-N Card” procedure on page 1-10.
Test the DS-N Card
Step 1 Complete the “Physically Replace a Card” procedure on page 2-219 for the suspected bad card and
replace it with a known-good one.
Step 2 Resend test traffic on the loopback circuit with a known-good card installed.
Step 3 If the test set indicates a good circuit, the problem was probably the defective card. Return the defective
card to Cisco through the Return Materials Authorization (RMA) process. Contact Cisco TAC (1 800
553-2447).
Step 4 Complete the “Physically Replace a Card” procedure on page 2-219 for the faulty card.
Step 5 In card view for the DS-N card, depending upon the type, click the Maintenance > Loopback tabs,
Maintenance > DS1 tabs, or Maintenance > DS3 tabs.