Cisco BTS 10200 Softswitch User manual

Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Cisco BTS 10200 Softswitch
Troubleshooting Guide, Release 5.0.x
Revised: July 22, 2009
Text Part Number: OL-8723-19

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at
www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1005R)
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
Copyright © 2009 Cisco Systems, Inc. All rights reserved.

iii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
CONTENTS
Preface xlix
Introduction xlix
Objective xlix
Audience xlix
Organization l
Conventions li
Updates to this Document for Release 5.0 li
New Events and Alarms (Release 4.5.1 to Release 5.0) lii
Modified Events and Alarms (Release 4.5.1 to Release 5.0) lv
Deleted Events and Alarms (Release 4.5.1 to Release 5.0) lv
Obtaining Documentation and Submitting a Service Request lvi
CHAPTER 1Troubleshooting Overview 1-1
Introduction 1-1
Interoperability 1-1
Symptoms, Problems, and Solutions 1-2
General Problem-Solving Model 1-2
Resolving Network Problems 1-5
Resolving System Problems 1-6
Managing Events and Alarms 1-9
Managing Event and Alarm Reports 1-10
Show Alarm Command 1-10
Report Alarm Command 1-11
Ack Alarm Command 1-12
Clear Alarm Command 1-12
Format of Alarm Reports 1-13
Events and Alarm Logs 1-14
Viewing Event or Alarm Logs 1-14
Show, Add, and Delete Event Queue Commands 1-16
Saving Events to Log Files 1-16
Show Report-Properties Command 1-16
Changing Report Properties 1-16
Changing Threshold and Throttle Values 1-17

Contents
iv
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Managing and Responding to Events and Alarms 1-18
Events and Alarms Descriptions and Corrective Actions 1-19
Format of Events and Alarms 1-19
CHAPTER 2Audit Troubleshooting 2-1
Introduction 2-1
Audit Events and Alarms 2-1
AUDIT (1) 2-2
AUDIT (2) 2-2
AUDIT (3) 2-3
AUDIT (4) 2-3
AUDIT (5) 2-4
AUDIT (6) 2-4
AUDIT (7) 2-5
AUDIT (8) 2-5
AUDIT (9) 2-5
AUDIT (10) 2-6
AUDIT (11) 2-6
AUDIT (12) 2-7
AUDIT (13) 2-7
AUDIT (14) 2-8
AUDIT (15) 2-8
AUDIT (16) 2-9
AUDIT (17) 2-10
AUDIT (18) 2-11
AUDIT (19) 2-12
AUDIT (20) 2-13
AUDIT (21) 2-13
AUDIT (22) 2-14
AUDIT (23) 2-14
AUDIT (24) 2-15
AUDIT (25) 2-16
Monitoring Audit Events 2-17
Test Report—Audit (1) 2-17
Start or Stop of Signaling System 7–Circuit Identification Code Audit—Audit (2) 2-18
Signaling System 7 Circuit Identification Code Audit Terminated Before Successful
Completion—Audit (3) 2-18
Call Exceeds a Long-Duration Threshold—Audit (4) 2-18
Critical Internal Audit Failure—Audit (5) 2-18

Contents
v
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Major Internal Audit Failure—Audit (6) 2-19
Minor Internal Audit Failure—Audit (7) 2-19
Warning From Internal Audit—Audit (8) 2-19
Call Data Audit Complete—Audit (10) 2-19
Critical Network Time Protocol Service Failure—Audit (11) 2-19
Major Network Time Protocol Service Failure—Audit (12) 2-19
Minor Network Time Protocol Service Failure—Audit (13) 2-19
Network Time Protocol Service Warning—Audit (14) 2-20
Critical Index Shared Memory Error—Audit (15) 2-20
Process Heap Memory Usage Exceeds Minor Threshold Level—Audit (16) 2-20
Process Heap Memory Usage Exceeds Major Threshold Level—Audit (17) 2-20
Process Heap Memory Usage Exceeds Critical Threshold Level—Audit (18) 2-20
Recovered Memory of Stale Call—Audit (19) 2-21
Audit Found Lost Call Data Record—Audit (20) 2-21
Quality of Service Gate Memory Audit Complete—Audit (21) 2-21
Quality of Service Gate Status Audit Complete—Audit (22) 2-21
Recover Memory of Dangling Gate—Audit (23) 2-22
No Gate in the Cable Modem Termination System for Active Connection—Audit (24) 2-22
Core File Present—Audit (25) 2-22
Troubleshooting Audit Alarms 2-23
Critical Internal Audit Failure—Audit (5) 2-23
Major Internal Audit Failure—Audit (6) 2-23
Minor Internal Audit Failure—Audit (7) 2-23
Critical Network Time Protocol Service Failure—Audit (11) 2-24
Major Network Time Protocol Service Failure—Audit (12) 2-24
Minor Network Time Protocol Service Failure—Audit (13) 2-25
Critical Index Shared Memory Error—Audit (15) 2-25
Process Heap Memory Usage Exceeds Minor Threshold Level—Audit (16) 2-25
Process Heap Memory Usage Exceeds Major Threshold Level—Audit (17) 2-26
Process Heap Memory Usage Exceeds Critical Threshold Level—Audit (18) 2-26
Audit Found Lost Call Data Record—Audit (20) 2-27
Core File Present—Audit (25) 2-27
CHAPTER 3Billing Troubleshooting 3-1
Introduction 3-1
Billing Events and Alarms 3-2
BILLING (1) 3-3
BILLING (2) 3-3
BILLING (3) 3-4

Contents
vi
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
BILLING (4) 3-5
BILLING (5) 3-5
BILLING (6) 3-6
BILLING (7) 3-7
BILLING (8) 3-7
BILLING (9) 3-8
BILLING (10) 3-8
BILLING (11) 3-8
BILLING (12) 3-8
BILLING (13) 3-8
BILLING (14) 3-9
BILLING (15) 3-10
BILLING (16) 3-10
BILLING (17) 3-10
BILLING (18) 3-10
BILLING (19) 3-10
BILLING (20) 3-11
BILLING (21) 3-11
BILLING (22) 3-11
BILLING (23) 3-11
BILLING (24) 3-11
BILLING (25) 3-11
BILLING (26) 3-11
BILLING (27) 3-11
BILLING (28) 3-11
BILLING (29) 3-12
BILLING (30) 3-12
BILLING (31) 3-13
BILLING (32) 3-13
BILLING (33) 3-14
BILLING (34) 3-14
BILLING (35) 3-14
BILLING (36) 3-15
BILLING (37) 3-15
BILLING (38) 3-16
BILLING (39) 3-16
BILLING (40) 3-17
BILLING (41) 3-17
BILLING (42) 3-18
BILLING (43) 3-18

Contents
vii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
BILLING (44) 3-19
BILLING (45) 3-19
BILLING (46) 3-20
BILLING (47) 3-20
BILLING (48) 3-21
BILLING (49) 3-21
BILLING (50) 3-21
BILLING (51) 3-21
BILLING (52) 3-22
BILLING (53) 3-22
BILLING (54) 3-23
BILLING (55) 3-23
BILLING (56) 3-24
BILLING (57) 3-24
BILLING (58) 3-25
BILLING (59) 3-25
Monitoring Billing Events 3-26
Test Report—Billing (1) 3-27
Billing Partition Disk Usage Minor Threshold Exceeded—Billing (2) 3-27
Billing Partition Disk Usage Major Threshold Exceeded—Billing (3) 3-27
Billing Partition Disk Usage Critical Threshold Exceeded—Billing (4) 3-27
Billing Partition Disk Usage Within Normal Range—Billing (5) 3-27
File Transfer Protocol/Secure File Transfer Protocol Transfer Failed—Billing (6) 3-28
Transmission Control Protocol Connection Error—Billing (7) 3-28
Transmission Control Protocol Packet Receive Failure—Billing (8) 3-28
Database Connection Error—Billing (13) 3-28
File Open Error—Billing (14) 3-28
File Write Error—Billing (15) 3-29
Call Data Block Send Failed—Billing (29) 3-29
Domain Name Mapping Failed—Billing (30) 3-29
Port not Specified—Billing (31) 3-29
Element Management System Address not Specified—Billing (32) 3-29
File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid—Billing (33) 3-29
All Billing Links at Billing Server Down—Billing (35) 3-30
Billing Link Restored—Billing (36) 3-30
Billing Link Failure—Billing (37) 3-30
Event Message Log File Access Error—Billing (38) 3-30
Event Message Encode Failure—Billing (40) 3-30
Message Content Error—Billing (41) 3-30
Error Reading Provisioned Data–Using Default—Billing (42) 3-31

Contents
viii
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Record Keeping System Switch Occurred—Billing (44) 3-31
Event Message Log File Opened—Billing (45) 3-31
Event Message Log File Closed—Billing (46) 3-31
Record Keeping System Unreachable for One Hour—Billing (47) 3-31
Record Keeping System Unreachable for Three Hours—Billing (48) 3-31
Record Keeping System Unreachable for Five Hours—Billing (49) 3-32
Bulk Data Management System Stopped Generating New Billing File—Billing (52) 3-32
Event Message Disk Space 50 Percent Full—Billing (53) 3-32
Event Message Disk Space 70 Percent Full—Billing (54) 3-32
Event Message Disk Space 100 Percent Full—Billing (55) 3-32
Billing Data Corruption Detected—Billing (56) 3-32
Prepaid Subscriber Call Attempt Failed Because of Balance—Billing (57) 3-33
Signaling Prepaid Server Inaccessible—Billing (58) 3-33
Billing File Name Type Change in Command Line Interface is Inconsistent—Billing (59) 3-33
Troubleshooting Billing Alarms 3-34
Billing Partition Disk Usage Minor Threshold Exceeded—Billing (2) 3-35
Billing Partition Disk Usage Major Threshold Exceeded—Billing (3) 3-35
Billing Partition Disk Usage Critical Threshold Exceeded—Billing (4) 3-36
File Transfer Protocol/Secure File Transfer Protocol Transfer Failed—Billing (6) 3-36
Transmission Control Protocol Connection Error—Billing (7) 3-37
Transmission Control Protocol Packet Receive Failure—Billing (8) 3-38
Database Connection Error—Billing (13) 3-38
File Write Error—Billing (15) 3-38
Call Data Block Send Failed—Billing (29) 3-38
Domain Name Mapping Failed—Billing (30) 3-39
Port not Specified—Billing (31) 3-39
Element Management System Address not Specified—Billing (32) 3-39
File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid—Billing (33) 3-39
All Billing Links at Billing Server Down—Billing (35) 3-39
Billing Link Failure—Billing (37) 3-40
Event Message Log File Access Error—Billing (38) 3-40
Event Message Encode Failure—Billing (40) 3-40
Message Content Error—Billing (41) 3-40
Record Keeping System Switch Occurred—Billing (44) 3-40
Event Message Log File Opened—Billing (45) 3-41
Event Message Log File Closed—Billing (46) 3-41
Record Keeping System Unreachable for One Hour—Billing (47) 3-41
Record Keeping System Unreachable for Three Hours—Billing (48) 3-41
Record Keeping System Unreachable for 5 Hours—Billing (49) 3-41
Bulk Data Management System Stopped Generating New Billing File—Billing (52) 3-42

Contents
ix
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
Event Message Disk Space 50 Percent Full—Billing (53) 3-42
Event Message Disk Space 70 Percent Full—Billing (54) 3-42
Event Message Disk Space 100 Percent Full—Billing (55) 3-42
Billing Data Corruption Detected—Billing (56) 3-43
Signaling Prepaid Server Inaccessible—Billing (58) 3-43
CHAPTER 4Call Processing Troubleshooting 4-1
Introduction 4-1
Call Processing Events and Alarms 4-2
CALLP (1) 4-3
CALLP (2) 4-3
CALLP (3) 4-3
CALLP (4) 4-3
CALLP (5) 4-3
CALLP (6) 4-3
CALLP (7) 4-3
CALLP (8) 4-4
CALLP (9) 4-5
CALLP (10) 4-5
CALLP (11) 4-6
CALLP (12) 4-6
CALLP (13) 4-7
CALLP (14) 4-7
CALLP (15) 4-7
CALLP (16) 4-8
CALLP (17) 4-8
CALLP (18) 4-9
CALLP (19) 4-9
CALLP (20) 4-10
CALLP (21) 4-10
CALLP (22) 4-11
CALLP (23) 4-11
CALLP (24) 4-12
CALLP (25) 4-12
CALLP (26) 4-13
CALLP (27) 4-13
CALLP (28) 4-14
CALLP (29) 4-14
CALLP (30) 4-15

Contents
x
Cisco BTS 10200 Softswitch Troubleshooting Guide, Release 5.0.x
OL-8723-19
CALLP (31) 4-15
CALLP (32) 4-16
CALLP (33) 4-16
CALLP (34) 4-17
CALLP (35) 4-17
CALLP (36) 4-18
CALLP (37) 4-18
CALLP (38) 4-19
CALLP (39) 4-19
CALLP (40) 4-20
CALLP (41) 4-20
CALLP (42) 4-21
CALLP (43) 4-21
CALLP (44) 4-22
CALLP (45) 4-22
Monitoring Call Processing Events 4-23
Test Report—Call Processing (1) 4-24
No Route Available for Called Number—Call Processing (8) 4-24
No Route Available for Carrier Dialed—Call Processing (9) 4-24
Feature Server One Link Down—Call Processing (11) 4-25
Feature Server Both Links Down—Call Processing (12) 4-25
Network Access Server Create Connection Error—Call Processing (13) 4-25
Network Access Server Authentication Failure—Call Processing (14) 4-25
Cable Modem Termination System Easily Recognizable Identification Not Found in Media Gateway
Table—Call Processing (15) 4-25
Route Index has No Trunk Group Assigned—Call Processing (16) 4-25
Invalid Route Index Used—Call Processing (17) 4-25
Unable to Play Announcement—Call Processing (18) 4-26
Call Routed to Unprovisioned Subscriber—Call Processing (19) 4-26
No Route or Trunk Group Available to Route Call—Call Processing (20) 4-26
Call Released Due to Maximum Hop Counts Exceeded—Call Processing (21) 4-26
Trunk Group Index Read Failure—Call Processing (22) 4-26
Routing Error: Termination is Not a Subscriber—Call Processing (23) 4-26
Invalid Route for Subscriber Index—Call Processing (24) 4-27
Invalid Route Group for Subscriber Routing—Call Processing (25) 4-27
Invalid Trunk Group for Subscriber Routing—Call Processing (26) 4-27
Unable to Route: Blocked by Destination Subscriber Status—Call Processing (27) 4-27
Route Name Does Not Exist—Call Processing (28) 4-27
Routing Selection Failure—Call Processing (29) 4-27
Customer-Originated Trace Test Failed—Call Processing (30) 4-28