Cisco BTS 10200 Softswitch User manual

Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Cisco BTS 10200 Softswitch
Troubleshooting Guide, Release 5.0.x
Revised: July 22, 2009
Text Part Number: OL-8723-19

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at
www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1005R)
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
Copyright © 2009 Cisco Systems, Inc. All rights reserved.

iii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
CONTENTS
Preface xlix
Introduction xlix
Objective xlix
Audience xlix
Organization l
Conventions li
Updates to this Document for Release 5.0 li
New Events and Alarms (Release 4.5.1 to Release 5.0) lii
Modified Events and Alarms (Release 4.5.1 to Release 5.0) lv
Deleted Events and Alarms (Release 4.5.1 to Release 5.0) lv
Obtaining Documentation and Submitting a Service Request lvi
CHAPTER 1Troubleshooting Overview 1-1
Introduction 1-1
Interoperability 1-1
Symptoms, Problems, and Solutions 1-2
General Problem-Solving Model 1-2
Resolving Network Problems 1-5
Resolving System Problems 1-6
Managing Events and Alarms 1-9
Managing Event and Alarm Reports 1-10
Show Alarm Command 1-10
Report Alarm Command 1-11
Ack Alarm Command 1-12
Clear Alarm Command 1-12
Format of Alarm Reports 1-13
Events and Alarm Logs 1-14
Viewing Event or Alarm Logs 1-14
Show, Add, and Delete Event Queue Commands 1-16
Saving Events to Log Files 1-16
Show Report-Properties Command 1-16
Changing Report Properties 1-16
Changing Threshold and Throttle Values 1-17

Contents
iv
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Managing and Responding to Events and Alarms 1-18
Events and Alarms Descriptions and Corrective Actions 1-19
Format of Events and Alarms 1-19
CHAPTER 2Audit Troubleshooting 2-1
Introduction 2-1
Audit Events and Alarms 2-1
AUDIT (1) 2-2
AUDIT (2) 2-2
AUDIT (3) 2-3
AUDIT (4) 2-3
AUDIT (5) 2-4
AUDIT (6) 2-4
AUDIT (7) 2-5
AUDIT (8) 2-5
AUDIT (9) 2-5
AUDIT (10) 2-6
AUDIT (11) 2-6
AUDIT (12) 2-7
AUDIT (13) 2-7
AUDIT (14) 2-8
AUDIT (15) 2-8
AUDIT (16) 2-9
AUDIT (17) 2-10
AUDIT (18) 2-11
AUDIT (19) 2-12
AUDIT (20) 2-13
AUDIT (21) 2-13
AUDIT (22) 2-14
AUDIT (23) 2-14
AUDIT (24) 2-15
AUDIT (25) 2-16
Monitoring Audit Events 2-17
Test Report—Audit (1) 2-17
Start or Stop of Signaling System 7–Circuit Identification Code Audit—Audit (2) 2-18
Signaling System 7 Circuit Identification Code Audit Terminated Before Successful
Completion—Audit (3) 2-18
Call Exceeds a Long-Duration Threshold—Audit (4) 2-18
Critical Internal Audit Failure—Audit (5) 2-18

Contents
v
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Major Internal Audit Failure—Audit (6) 2-19
Minor Internal Audit Failure—Audit (7) 2-19
Warning From Internal Audit—Audit (8) 2-19
Call Data Audit Complete—Audit (10) 2-19
Critical Network Time Protocol Service Failure—Audit (11) 2-19
Major Network Time Protocol Service Failure—Audit (12) 2-19
Minor Network Time Protocol Service Failure—Audit (13) 2-19
Network Time Protocol Service Warning—Audit (14) 2-20
Critical Index Shared Memory Error—Audit (15) 2-20
Process Heap Memory Usage Exceeds Minor Threshold Level—Audit (16) 2-20
Process Heap Memory Usage Exceeds Major Threshold Level—Audit (17) 2-20
Process Heap Memory Usage Exceeds Critical Threshold Level—Audit (18) 2-20
Recovered Memory of Stale Call—Audit (19) 2-21
Audit Found Lost Call Data Record—Audit (20) 2-21
Quality of Service Gate Memory Audit Complete—Audit (21) 2-21
Quality of Service Gate Status Audit Complete—Audit (22) 2-21
Recover Memory of Dangling Gate—Audit (23) 2-22
No Gate in the Cable Modem Termination System for Active Connection—Audit (24) 2-22
Core File Present—Audit (25) 2-22
Troubleshooting Audit Alarms 2-23
Critical Internal Audit Failure—Audit (5) 2-23
Major Internal Audit Failure—Audit (6) 2-23
Minor Internal Audit Failure—Audit (7) 2-23
Critical Network Time Protocol Service Failure—Audit (11) 2-24
Major Network Time Protocol Service Failure—Audit (12) 2-24
Minor Network Time Protocol Service Failure—Audit (13) 2-25
Critical Index Shared Memory Error—Audit (15) 2-25
Process Heap Memory Usage Exceeds Minor Threshold Level—Audit (16) 2-25
Process Heap Memory Usage Exceeds Major Threshold Level—Audit (17) 2-26
Process Heap Memory Usage Exceeds Critical Threshold Level—Audit (18) 2-26
Audit Found Lost Call Data Record—Audit (20) 2-27
Core File Present—Audit (25) 2-27
CHAPTER 3Billing Troubleshooting 3-1
Introduction 3-1
Billing Events and Alarms 3-2
BILLING (1) 3-3
BILLING (2) 3-3
BILLING (3) 3-4

Contents
vi
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
BILLING (4) 3-5
BILLING (5) 3-5
BILLING (6) 3-6
BILLING (7) 3-7
BILLING (8) 3-7
BILLING (9) 3-8
BILLING (10) 3-8
BILLING (11) 3-8
BILLING (12) 3-8
BILLING (13) 3-8
BILLING (14) 3-9
BILLING (15) 3-10
BILLING (16) 3-10
BILLING (17) 3-10
BILLING (18) 3-10
BILLING (19) 3-10
BILLING (20) 3-11
BILLING (21) 3-11
BILLING (22) 3-11
BILLING (23) 3-11
BILLING (24) 3-11
BILLING (25) 3-11
BILLING (26) 3-11
BILLING (27) 3-11
BILLING (28) 3-11
BILLING (29) 3-12
BILLING (30) 3-12
BILLING (31) 3-13
BILLING (32) 3-13
BILLING (33) 3-14
BILLING (34) 3-14
BILLING (35) 3-14
BILLING (36) 3-15
BILLING (37) 3-15
BILLING (38) 3-16
BILLING (39) 3-16
BILLING (40) 3-17
BILLING (41) 3-17
BILLING (42) 3-18
BILLING (43) 3-18

Contents
vii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
BILLING (44) 3-19
BILLING (45) 3-19
BILLING (46) 3-20
BILLING (47) 3-20
BILLING (48) 3-21
BILLING (49) 3-21
BILLING (50) 3-21
BILLING (51) 3-21
BILLING (52) 3-22
BILLING (53) 3-22
BILLING (54) 3-23
BILLING (55) 3-23
BILLING (56) 3-24
BILLING (57) 3-24
BILLING (58) 3-25
BILLING (59) 3-25
Monitoring Billing Events 3-26
Test Report—Billing (1) 3-27
Billing Partition Disk Usage Minor Threshold Exceeded—Billing (2) 3-27
Billing Partition Disk Usage Major Threshold Exceeded—Billing (3) 3-27
Billing Partition Disk Usage Critical Threshold Exceeded—Billing (4) 3-27
Billing Partition Disk Usage Within Normal Range—Billing (5) 3-27
File Transfer Protocol/Secure File Transfer Protocol Transfer Failed—Billing (6) 3-28
Transmission Control Protocol Connection Error—Billing (7) 3-28
Transmission Control Protocol Packet Receive Failure—Billing (8) 3-28
Database Connection Error—Billing (13) 3-28
File Open Error—Billing (14) 3-28
File Write Error—Billing (15) 3-29
Call Data Block Send Failed—Billing (29) 3-29
Domain Name Mapping Failed—Billing (30) 3-29
Port not Specified—Billing (31) 3-29
Element Management System Address not Specified—Billing (32) 3-29
File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid—Billing (33) 3-29
All Billing Links at Billing Server Down—Billing (35) 3-30
Billing Link Restored—Billing (36) 3-30
Billing Link Failure—Billing (37) 3-30
Event Message Log File Access Error—Billing (38) 3-30
Event Message Encode Failure—Billing (40) 3-30
Message Content Error—Billing (41) 3-30
Error Reading Provisioned Data–Using Default—Billing (42) 3-31

Contents
viii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Record Keeping System Switch Occurred—Billing (44) 3-31
Event Message Log File Opened—Billing (45) 3-31
Event Message Log File Closed—Billing (46) 3-31
Record Keeping System Unreachable for One Hour—Billing (47) 3-31
Record Keeping System Unreachable for Three Hours—Billing (48) 3-31
Record Keeping System Unreachable for Five Hours—Billing (49) 3-32
Bulk Data Management System Stopped Generating New Billing File—Billing (52) 3-32
Event Message Disk Space 50 Percent Full—Billing (53) 3-32
Event Message Disk Space 70 Percent Full—Billing (54) 3-32
Event Message Disk Space 100 Percent Full—Billing (55) 3-32
Billing Data Corruption Detected—Billing (56) 3-32
Prepaid Subscriber Call Attempt Failed Because of Balance—Billing (57) 3-33
Signaling Prepaid Server Inaccessible—Billing (58) 3-33
Billing File Name Type Change in Command Line Interface is Inconsistent—Billing (59) 3-33
Troubleshooting Billing Alarms 3-34
Billing Partition Disk Usage Minor Threshold Exceeded—Billing (2) 3-35
Billing Partition Disk Usage Major Threshold Exceeded—Billing (3) 3-35
Billing Partition Disk Usage Critical Threshold Exceeded—Billing (4) 3-36
File Transfer Protocol/Secure File Transfer Protocol Transfer Failed—Billing (6) 3-36
Transmission Control Protocol Connection Error—Billing (7) 3-37
Transmission Control Protocol Packet Receive Failure—Billing (8) 3-38
Database Connection Error—Billing (13) 3-38
File Write Error—Billing (15) 3-38
Call Data Block Send Failed—Billing (29) 3-38
Domain Name Mapping Failed—Billing (30) 3-39
Port not Specified—Billing (31) 3-39
Element Management System Address not Specified—Billing (32) 3-39
File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid—Billing (33) 3-39
All Billing Links at Billing Server Down—Billing (35) 3-39
Billing Link Failure—Billing (37) 3-40
Event Message Log File Access Error—Billing (38) 3-40
Event Message Encode Failure—Billing (40) 3-40
Message Content Error—Billing (41) 3-40
Record Keeping System Switch Occurred—Billing (44) 3-40
Event Message Log File Opened—Billing (45) 3-41
Event Message Log File Closed—Billing (46) 3-41
Record Keeping System Unreachable for One Hour—Billing (47) 3-41
Record Keeping System Unreachable for Three Hours—Billing (48) 3-41
Record Keeping System Unreachable for 5 Hours—Billing (49) 3-41
Bulk Data Management System Stopped Generating New Billing File—Billing (52) 3-42

Contents
ix
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Event Message Disk Space 50 Percent Full—Billing (53) 3-42
Event Message Disk Space 70 Percent Full—Billing (54) 3-42
Event Message Disk Space 100 Percent Full—Billing (55) 3-42
Billing Data Corruption Detected—Billing (56) 3-43
Signaling Prepaid Server Inaccessible—Billing (58) 3-43
CHAPTER 4Call Processing Troubleshooting 4-1
Introduction 4-1
Call Processing Events and Alarms 4-2
CALLP (1) 4-3
CALLP (2) 4-3
CALLP (3) 4-3
CALLP (4) 4-3
CALLP (5) 4-3
CALLP (6) 4-3
CALLP (7) 4-3
CALLP (8) 4-4
CALLP (9) 4-5
CALLP (10) 4-5
CALLP (11) 4-6
CALLP (12) 4-6
CALLP (13) 4-7
CALLP (14) 4-7
CALLP (15) 4-7
CALLP (16) 4-8
CALLP (17) 4-8
CALLP (18) 4-9
CALLP (19) 4-9
CALLP (20) 4-10
CALLP (21) 4-10
CALLP (22) 4-11
CALLP (23) 4-11
CALLP (24) 4-12
CALLP (25) 4-12
CALLP (26) 4-13
CALLP (27) 4-13
CALLP (28) 4-14
CALLP (29) 4-14
CALLP (30) 4-15

Contents
x
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
CALLP (31) 4-15
CALLP (32) 4-16
CALLP (33) 4-16
CALLP (34) 4-17
CALLP (35) 4-17
CALLP (36) 4-18
CALLP (37) 4-18
CALLP (38) 4-19
CALLP (39) 4-19
CALLP (40) 4-20
CALLP (41) 4-20
CALLP (42) 4-21
CALLP (43) 4-21
CALLP (44) 4-22
CALLP (45) 4-22
Monitoring Call Processing Events 4-23
Test Report—Call Processing (1) 4-24
No Route Available for Called Number—Call Processing (8) 4-24
No Route Available for Carrier Dialed—Call Processing (9) 4-24
Feature Server One Link Down—Call Processing (11) 4-25
Feature Server Both Links Down—Call Processing (12) 4-25
Network Access Server Create Connection Error—Call Processing (13) 4-25
Network Access Server Authentication Failure—Call Processing (14) 4-25
Cable Modem Termination System Easily Recognizable Identification Not Found in Media Gateway
Table—Call Processing (15) 4-25
Route Index has No Trunk Group Assigned—Call Processing (16) 4-25
Invalid Route Index Used—Call Processing (17) 4-25
Unable to Play Announcement—Call Processing (18) 4-26
Call Routed to Unprovisioned Subscriber—Call Processing (19) 4-26
No Route or Trunk Group Available to Route Call—Call Processing (20) 4-26
Call Released Due to Maximum Hop Counts Exceeded—Call Processing (21) 4-26
Trunk Group Index Read Failure—Call Processing (22) 4-26
Routing Error: Termination is Not a Subscriber—Call Processing (23) 4-26
Invalid Route for Subscriber Index—Call Processing (24) 4-27
Invalid Route Group for Subscriber Routing—Call Processing (25) 4-27
Invalid Trunk Group for Subscriber Routing—Call Processing (26) 4-27
Unable to Route: Blocked by Destination Subscriber Status—Call Processing (27) 4-27
Route Name Does Not Exist—Call Processing (28) 4-27
Routing Selection Failure—Call Processing (29) 4-27
Customer-Originated Trace Test Failed—Call Processing (30) 4-28

Contents
xi
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Call Authorization Failure—Call Processing (31) 4-28
Country Code Dialing Plan Error—Call Processing (32) 4-28
Invalid Call—Call Processing (33) 4-28
Dial Plan Information Not Found for Digits Received—Call Processing (34) 4-28
Dial Plan Information for Test Call Not Found—Call Processing (35) 4-29
Invalid or Unknown Nature of Address—Call Processing (36) 4-29
Call Failure—Call Processing (37) 4-29
Release Cause 25 Exchange Routing Error Received—Call Processing (38) 4-29
Test Call Blocked Due to Congestion or Isolation—Call Processing (39) 4-29
Interactive Voice Response Real Time Transport Protocol Session Fail—Call Processing (40) 4-30
INVITE Message From Unauthorized Call Agent—Call Processing (41) 4-30
Call Failed after Local Number Portability Query with Location Routing Number of this BTS 10200 and
the Directory Number—Call Processing (42) 4-30
Call Processing Session Initiation Protocol Trigger Provisioning Error—Call Processing (43) 4-30
Call Processing No Session Initiation Protocol Trigger Context Found—Call Processing (44) 4-31
Context in call from Application Server not Found—Call Processing (45) 4-31
Troubleshooting Call Processing Alarms 4-32
Feature Server One Link Down—Call Processing (11) 4-33
Feature Server Both Links Down—Call Processing (12) 4-35
Release Cause 25 Exchange Routing Error Received—Call Processing (38) 4-36
INVITE Message From Unauthorized Call Agent—Call Processing (41) 4-36
CHAPTER 5Configuration Troubleshooting 5-1
Introduction 5-1
Configuration Events and Alarms 5-1
CONFIG (1) 5-2
CONFIG (2) 5-2
CONFIG (3) 5-3
CONFIG (4) 5-3
CONFIG (5) 5-4
CONFIG (6) 5-4
CONFIG (7) 5-5
CONFIG (8) 5-5
Monitoring Configuration Events 5-6
Test Report—Configuration (1) 5-6
Signaling Media Gateway Adapter Wrongly Configured Domain Name—Configuration (2) 5-6
Mate Configuration Error—Configuration (3) 5-6
Configuration Error—Configuration (4) 5-7
Feature–Server Database and Command Line Host Mismatch—Configuration (5) 5-7

Contents
xii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
FIMXML Parse Error—Configuration (6) 5-7
Application Server Provisioning Error—Configuration (7) 5-7
BTS 10200 Provisioning for External Applications is not Complete—Configuration (8) 5-7
Troubleshooting Configuration Alarms 5-8
Mate Configuration Error—Configuration (3) 5-8
Configuration Error—Configuration (4) 5-8
Feature–Server Database and Command Line Host Mismatch—Configuration (5) 5-8
CHAPTER 6Database Troubleshooting 6-1
Introduction 6-1
Database Events and Alarms 6-2
DATABASE (1) 6-2
DATABASE (2) 6-3
DATABASE (3) 6-4
DATABASE (4) 6-5
DATABASE (5) 6-6
DATABASE (6) 6-6
DATABASE (7) 6-7
DATABASE (8) 6-8
DATABASE (9) 6-8
DATABASE (10) 6-9
DATABASE (11) 6-10
DATABASE (12) 6-11
DATABASE (13) 6-12
DATABASE (14) 6-13
DATABASE (15) 6-13
DATABASE (16) 6-14
DATABASE (17) 6-14
DATABASE (18) 6-15
DATABASE (19) 6-15
DATABASE (20) 6-16
DATABASE (21) 6-16
DATABASE (22) 6-17
DATABASE (23) 6-17
DATABASE (24) 6-18
DATABASE (25) 6-19
DATABASE (26) 6-19
DATABASE (27) 6-20
Monitoring Database Events 6-21

Contents
xiii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Test Report—Database (1) 6-22
Database Management Update Failure: Master/Slave Database Out of Sync—Database (2) 6-22
There are Errors in Element Management System Database DefError Queue; Contact Database
Administrator—Database (3) 6-22
Element Management System Database HeartBeat: Replication PUSH Job Broken—Database
(4) 6-22
Element Management System Database HeartBeat Process Died—Database (5) 6-22
Element Management System Database Replication DefTranDest Queue Overloaded—Database
(6) 6-23
Element Management System Database DefTran Queue is Overloaded—Database (7) 6-23
Element Management System Database Tablespace is Out of Free Space—Database (8) 6-23
Urgent: Element Management System Database Archive Log Directory is Getting Full—Database
(9) 6-23
Element Management System Database: Backup Fails—Database (10) 6-23
Element Management System Database Alert.log Alerts—Database (11) 6-24
Element Management System Database Process Died—Database (12) 6-24
Element Management System Database Performance Alert—Database (13) 6-24
Table Size Exceeds Minor Threshold Limit—Database (14) 6-24
Table Size Exceeds Major Threshold Limit—Database (15) 6-24
Table Size Exceeds Critical Threshold Limit—Database (16) 6-24
Data Replication Failed—Database (17) 6-25
Unexpected Runtime Data Interaction—Database (18) 6-25
Daily Database Backup Completed Successfully—Database (19) 6-25
Replication Data Flush Timeout During Switchover—Database (20) 6-25
Database Statistics Collection Exception—Database (21) 6-25
Unprovisioned Language—Database (22) 6-25
Element Management System Oracle Database–Minor Error—Database (23) 6-26
Element Management System Oracle Database–Major Error—Database (24) 6-26
Secure File Transfer Protocol Transfer Failed—Database (25) 6-26
File Write Error—Database (26) 6-26
Failure Setting the Index Table Soft Limit—Database (27) 6-26
Troubleshooting Database Alarms 6-27
There Are Errors in Element Management System Database DefError Queue; Contact Database
Administrator—Database (3) 6-28
Element Management System Database HeartBeat: Replication PUSH Job Broken—Database
(4) 6-29
Element Management System Database HeartBeat Process Died—Database (5) 6-29
Element Management System Database Replication DefTranDest Queue Overloaded—Database
(6) 6-30
Element Management System Database DefTran Queue is Overloaded—Database (7) 6-30
Element Management System Database Tablespace is Out of Free Space—Database (8) 6-31

Contents
xiv
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Urgent: Element Management System Database Archive Log Directory is Getting Full—Database
(9) 6-31
Element Management System Database: Backup Fails—Database (10) 6-31
Element Management System Database Alert.log Alerts—Database (11) 6-32
Element Management System Database Process Died—Database (12) 6-32
Element Management System Database Performance Alert—Database (13) 6-32
Table Size Exceeds Minor Threshold Limit—Database (14) 6-33
Table Size Exceeds Major Threshold Limit—Database (15) 6-33
Table Size Exceeds Critical Threshold Limit—Database (16) 6-33
Data Replication Failed—Database (17) 6-33
Replication Data Flush Timeout During Switchover—Database (20) 6-33
Database Statistics Collection Exception—Database (21) 6-34
Unprovisioned Language—Database (22) 6-35
Element Management System Oracle Database–Minor Error—Database (23) 6-36
Element Management System Oracle Database–Major Error—Database (24) 6-37
Secure File Transfer Protocol Transfer Failed—Database (25) 6-37
File Write Error—Database (26) 6-37
Failure Setting the Index Table Soft Limit—Database (27) 6-37
CHAPTER 7Maintenance Troubleshooting 7-1
Introduction 7-1
Maintenance Events and Alarms 7-2
MAINTENANCE (1) 7-3
MAINTENANCE (2) 7-3
MAINTENANCE (3) 7-4
MAINTENANCE (4) 7-5
MAINTENANCE (5) 7-6
MAINTENANCE (6) 7-7
MAINTENANCE (7) 7-8
MAINTENANCE (8) 7-9
MAINTENANCE (9) 7-9
MAINTENANCE (10) 7-10
MAINTENANCE (11) 7-10
MAINTENANCE (12) 7-11
MAINTENANCE (13) 7-11
MAINTENANCE (14) 7-12
MAINTENANCE (15) 7-12
MAINTENANCE (16) 7-13
MAINTENANCE (17) 7-13
MAINTENANCE (18) 7-14

Contents
xv
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
MAINTENANCE (19) 7-14
MAINTENANCE (20) 7-15
MAINTENANCE (21) 7-15
MAINTENANCE (22) 7-16
MAINTENANCE (23) 7-16
MAINTENANCE (24) 7-17
MAINTENANCE (25) 7-17
MAINTENANCE (26) 7-18
MAINTENANCE (27) 7-18
MAINTENANCE (28) 7-18
MAINTENANCE (29) 7-19
MAINTENANCE (30) 7-19
MAINTENANCE (31) 7-20
MAINTENANCE (32) 7-20
MAINTENANCE (33) 7-20
MAINTENANCE (34) 7-21
MAINTENANCE (35) 7-21
MAINTENANCE (36) 7-22
MAINTENANCE (37) 7-22
MAINTENANCE (38) 7-23
MAINTENANCE (39) 7-23
MAINTENANCE (40) 7-23
MAINTENANCE (41) 7-24
MAINTENANCE (42) 7-24
MAINTENANCE (43) 7-25
MAINTENANCE (44) 7-25
MAINTENANCE (45) 7-26
MAINTENANCE (46) 7-26
MAINTENANCE (47) 7-27
MAINTENANCE (48) 7-27
MAINTENANCE (49) 7-28
MAINTENANCE (50) 7-29
MAINTENANCE (51) 7-29
MAINTENANCE (52) 7-30
MAINTENANCE (53) 7-30
MAINTENANCE (54) 7-31
MAINTENANCE (55) 7-31
MAINTENANCE (56) 7-32
MAINTENANCE (57) 7-32
MAINTENANCE (58) 7-33

Contents
xvi
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
MAINTENANCE (59) 7-33
MAINTENANCE (60) 7-33
MAINTENANCE (61) 7-34
MAINTENANCE (62) 7-34
MAINTENANCE (63) 7-35
MAINTENANCE (64) 7-35
MAINTENANCE (65) 7-36
MAINTENANCE (66) 7-36
MAINTENANCE (67) 7-37
MAINTENANCE (68) 7-37
MAINTENANCE (69) 7-38
MAINTENANCE (70) 7-38
MAINTENANCE (71) 7-39
MAINTENANCE (72) 7-39
MAINTENANCE (73) 7-40
MAINTENANCE (74) 7-40
MAINTENANCE (75) 7-41
MAINTENANCE (76) 7-41
MAINTENANCE (77) 7-41
MAINTENANCE (78) 7-42
MAINTENANCE (79) 7-42
MAINTENANCE (80) 7-43
MAINTENANCE (81) 7-43
MAINTENANCE (82) 7-44
MAINTENANCE (83) 7-44
MAINTENANCE (84) 7-45
MAINTENANCE (85) 7-45
MAINTENANCE (86) 7-46
MAINTENANCE (87) 7-46
MAINTENANCE (88) 7-47
MAINTENANCE (89) 7-47
MAINTENANCE (90) 7-48
MAINTENANCE (91) 7-48
MAINTENANCE (92) 7-49
MAINTENANCE (93) 7-49
MAINTENANCE (94) 7-50
MAINTENANCE (95) 7-50
MAINTENANCE (96) 7-51
MAINTENANCE (97) 7-51
MAINTENANCE (98) 7-52

Contents
xvii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
MAINTENANCE (99) 7-53
MAINTENANCE (100) 7-54
MAINTENANCE (101) 7-54
MAINTENANCE (102) 7-55
MAINTENANCE (103) 7-55
MAINTENANCE (104) 7-56
MAINTENANCE (105) 7-56
MAINTENANCE (106) 7-57
MAINTENANCE (107) 7-57
MAINTENANCE (108) 7-58
MAINTENANCE (109) 7-58
MAINTENANCE (110) 7-59
MAINTENANCE (111) 7-59
MAINTENANCE (112) 7-60
MAINTENANCE (113) 7-60
MAINTENANCE (114) 7-61
MAINTENANCE (115) 7-61
MAINTENANCE (116) 7-62
MAINTENANCE (117) 7-62
MAINTENANCE (118) 7-63
MAINTENANCE (119) 7-63
MAINTENANCE (120) 7-64
MAINTENANCE (121) 7-64
MAINTENANCE (122) 7-65
MAINTENANCE (123) 7-65
Monitoring Maintenance Events 7-66
Test Report—Maintenance (1) 7-69
Report Threshold Exceeded—Maintenance (2) 7-70
Local Side has Become Faulty—Maintenance (3) 7-70
Mate Side has Become Faulty—Maintenance (4) 7-70
Changeover Failure—Maintenance (5) 7-70
Changeover Timeout—Maintenance (6) 7-70
Mate Rejected Changeover—Maintenance (7) 7-70
Mate Changeover Timeout—Maintenance (8) 7-70
Local Initialization Failure—Maintenance (9) 7-71
Local Initialization Timeout—Maintenance (10) 7-71
Switchover Complete—Maintenance (11) 7-71
Initialization Successful—Maintenance (12) 7-71
Administrative State Change—Maintenance (13) 7-71
Call Agent Administrative State Change—Maintenance (14) 7-71

Contents
xviii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Feature Server Administrative State Change—Maintenance (15) 7-71
Process Manager: Starting Process—Maintenance (16) 7-72
Invalid Event Report Received—Maintenance (17) 7-72
Process Manager: Process has Died—Maintenance (18) 7-72
Process Manager: Process Exceeded Restart Rate—Maintenance (19) 7-72
Lost Connection to Mate—Maintenance (20) 7-72
Network Interface Down—Maintenance (21) 7-72
Mate is Alive—Maintenance (22) 7-73
Process Manager: Process Failed to Complete Initialization—Maintenance (23) 7-73
Process Manager: Restarting Process—Maintenance (24) 7-73
Process Manager: Changing State—Maintenance (25) 7-73
Process Manager: Going Faulty—Maintenance (26) 7-73
Process Manager: Changing Over to Active—Maintenance (27) 7-73
Process Manager: Changing Over to Standby—Maintenance (28) 7-74
Administrative State Change Failure—Maintenance (29) 7-74
Element Manager State Change—Maintenance (30) 7-74
Process Manager: Sending Go Active to Process—Maintenance (32) 7-74
Process Manager: Sending Go Standby to Process—Maintenance (33) 7-74
Process Manager: Sending End Process to Process—Maintenance (34) 7-74
Process Manager: All Processes Completed Initialization—Maintenance (35) 7-75
Process Manager: Sending All Processes Initialization Complete to Process—Maintenance
(36) 7-75
Process Manager: Killing Process—Maintenance (37) 7-75
Process Manager: Clearing the Database—Maintenance (38) 7-75
Process Manager: Cleared the Database—Maintenance (39) 7-75
Process Manager: Binary Does not Exist for Process—Maintenance (40) 7-75
Administrative State Change Successful with Warning—Maintenance (41) 7-76
Number of Heartbeat Messages Received is Less Than 50% of Expected—Maintenance (42) 7-76
Process Manager: Process Failed to Come Up in Active Mode—Maintenance (43) 7-76
Process Manager: Process Failed to Come Up in Standby Mode—Maintenance (44) 7-76
Application Instance State Change Failure—Maintenance (45) 7-76
Network Interface Restored—Maintenance (46) 7-76
Thread Watchdog Counter Expired for a Thread—Maintenance (47) 7-77
Index Table Usage Exceeded Minor Usage Threshold Level—Maintenance (48) 7-77
Index Table Usage Exceeded Major Usage Threshold Level—Maintenance (49) 7-77
Index Table Usage Exceeded Critical Usage Threshold Level—Maintenance (50) 7-77
A Process Exceeds 70% of Central Processing Unit Usage—Maintenance (51) 7-77
Central Processing Unit Usage is Now Below the 50% Level—Maintenance (52) 7-77
The Central Processing Unit Usage is Over 90% Busy—Maintenance (53) 7-78
The Central Processing Unit has Returned to Normal Levels of Operation—Maintenance (54) 7-78

Contents
xix
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
The Five Minute Load Average is Abnormally High—Maintenance (55) 7-78
The Load Average has Returned to Normal Levels—Maintenance (56) 7-78
Memory and Swap are Consumed at Critical Levels—Maintenance (57) 7-78
Memory and Swap are Consumed at Abnormal Levels—Maintenance (58) 7-79
No Heartbeat Messages Received Through the Interface—Maintenance (61) 7-79
Link Monitor: Interface Lost Communication—Maintenance (62) 7-79
Outgoing Heartbeat Period Exceeded Limit—Maintenance (63) 7-79
Average Outgoing Heartbeat Period Exceeds Major Alarm Limit—Maintenance (64) 7-80
Disk Partition Critically Consumed—Maintenance (65) 7-80
Disk Partition Significantly Consumed—Maintenance (66) 7-80
The Free Inter-Process Communication Pool Buffers Below Minor Threshold—Maintenance
(67) 7-80
The Free Inter-Process Communication Pool Buffers Below Major Threshold—Maintenance
(68) 7-80
The Free Inter-Process Communication Pool Buffers Below Critical Threshold—Maintenance
(69) 7-81
The Free Inter-Process Communication Pool Buffer Count Below Minimum Required—Maintenance
(70) 7-81
Local Domain Name System Server Response Too Slow—Maintenance (71) 7-81
External Domain Name System Server Response Too Slow—Maintenance (72) 7-81
External Domain Name System Server not Responsive—Maintenance (73) 7-81
Local Domain Name System Service not Responsive—Maintenance (74) 7-82
Mismatch of Internet Protocol Address Local Server and Domain Name System—Maintenance
(75) 7-82
Mate Time Differs Beyond Tolerance—Maintenance (77) 7-82
Bulk Data Management System Admin State Change—Maintenance (78) 7-82
Resource Reset—Maintenance (79) 7-82
Resource Reset Warning—Maintenance (80) 7-82
Resource Reset Failure—Maintenance (81) 7-83
Average Outgoing Heartbeat Period Exceeds Critical Limit—Maintenance (82) 7-83
Swap Space Below Minor Threshold—Maintenance (83) 7-83
Swap Space Below Major Threshold—Maintenance (84) 7-83
Swap Space Below Critical Threshold—Maintenance (85) 7-83
System Health Report Collection Error—Maintenance (86) 7-83
Status Update Process Request Failed—Maintenance (87) 7-84
Status Update Process Database List Retrieval Error—Maintenance (88) 7-84
Status Update Process Database Update Error—Maintenance (89) 7-84
Disk Partition Moderately Consumed—Maintenance (90) 7-84
Internet Protocol Manager Configuration File Error—Maintenance (91) 7-84
Internet Protocol Manager Initialization Error—Maintenance (92) 7-84
Internet Protocol Manager Interface Failure—Maintenance (93) 7-84

Contents
xx
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Internet Protocol Manager Interface State Change—Maintenance (94) 7-85
Internet Protocol Manager Interface Created—Maintenance (95) 7-85
Internet Protocol Manager Interface Removed—Maintenance (96) 7-85
Inter-Process Communication Input Queue Entered Throttle State—Maintenance (97) 7-85
Inter-Process Communication Input Queue Depth at 25% of its Hi-Watermark—Maintenance
(98) 7-85
Inter-Process Communication Input Queue Depth at 50% of its Hi-Watermark—Maintenance
(99) 7-85
Inter-Process Communication Input Queue Depth at 75% of its Hi-Watermark—Maintenance
(100) 7-86
Switchover in Progress—Maintenance (101) 7-86
Thread Watchdog Counter Close to Expiry for a Thread—Maintenance (102) 7-86
Central Processing Unit is Offline—Maintenance (103) 7-86
Aggregration Device Address Successfully Resolved—Maintenance (104) 7-86
Unprovisioned Aggregration Device Detected—Maintenance (105) 7-86
Aggregration Device Address Resolution Failure—Maintenance (106) 7-87
No Heartbeat Messages Received Through Interface From Router—Maintenance (107) 7-87
A Log File Cannot be Transferred—Maintenance (108) 7-87
Five Successive Log Files Cannot be Transferred—Maintenance (109) 7-87
Access to Log Archive Facility Configuration File Failed or File Corrupted—Maintenance (110) 7-87
Cannot Login to External Archive Server—Maintenance (111) 7-88
Congestion Status—Maintenance (112) 7-88
Central Processing Unit Load of Critical Processes—Maintenance (113) 7-88
Queue Length of Critical Processes—Maintenance (114) 7-88
Inter-Process Communication Buffer Usage Level—Maintenance (115) 7-88
Call Agent Reports the Congestion Level of Feature Server—Maintenance (116) 7-88
Side Automatically Restarting Due to Fault—Maintenance (117) 7-88
Domain Name Server Zone Database does not Match Between the Primary Domain Name Server and
the Internal Secondary Authoritative Domain Name Server—Maintenance (118) 7-89
Periodic Shared Memory Database Backup Failure—Maintenance (119) 7-89
Periodic Shared Memory Database Backup Success—Maintenance (120) 7-89
Invalid SOAP Request—Maintenance (121) 7-89
Northbound Provisioning Message is Retransmitted—Maintenance (122) 7-89
Northbound Provisioning Message Dropped Due To Full Index Table—Maintenance (123) 7-89
Troubleshooting Maintenance Alarms 7-90
Local Side has Become Faulty—Maintenance (3) 7-92
Mate Side has Become Faulty—Maintenance (4) 7-92
Changeover Failure—Maintenance (5) 7-93
Changeover Timeout—Maintenance (6) 7-93
Mate Rejected Changeover—Maintenance (7) 7-94