Cisco ASR 5000 Series Instruction Manual

Cisco ASR 5000 Series External Storage Server
Installation and Administration Guide
Version 9.0
Last updated February 15, 2012
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Text Part Number: OL-22979-03

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY
OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED
WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED
WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to
part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This
equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio
communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own
expense.
Modifications to this product not authorized by Cisco could void the FCC approval and negate your authority to operate the product.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB‟s public domain
version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL
FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE
PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR
ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at
www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship
between Cisco and any other company.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display
output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any u se of actual IP addresses or phone numbers in
illustrative content is unintentional and coincidental.
Cisco ASR 5000 Series External Storage Server Installation and Administration Guide
© 2012 Cisco Systems, Inc. and/or its affiliated entities. All rights reserved.

Cisco ASR 5000 Series External Storage Server Installation and Administration Guide ▄
OL-22979-03
iii
CONTENTS
About this Guide ................................................................................................. v
Conventions Used ....................................................................................................................................vi
Contacting Customer Support................................................................................................................viii
Additional Information...............................................................................................................................ix
External Storage System Overview ................................................................ 11
Overview.................................................................................................................................................12
Local, Short-Term External Storage System .....................................................................................13
System Requirements............................................................................................................................15
ASR 5000 System Requirements ......................................................................................................15
ESS System Requirements................................................................................................................15
Minimum System Recommendations for Stand-alone Deployment of L-ESS...............................15
Minimum System Recommendations for Cluster Deployment of L-ESS.......................................16
Veritas Cluster Installation and Management................................................ 17
L-ESS Cluster Functional Description....................................................................................................19
Installing Hardware.................................................................................................................................20
Configuring Storage Array......................................................................................................................24
Configuring Veritas Volume Manager and Veritas Cluster.....................................................................32
Monitoring Veritas Cluster......................................................................................................................37
Setup of rootdisk Encapsulation and Mirroring.......................................................................................39
Testing Veritas Cluster...........................................................................................................................41
L-ESS Cluster Failure Handling .............................................................................................................43
L-ESS Installation and Configuration ............................................................. 45
L-ESS Installation Modes.......................................................................................................................46
Installing L-ESS Application in Stand-alone Mode.................................................................................47
Installing L-ESS Application in Cluster Mode.........................................................................................56
Uninstalling L-ESS Application...............................................................................................................66
Configuring PSMON Threshold (Optional).............................................................................................67
Configuring the L-ESS Server ......................................................................... 69
L-ESS Server Configuration...................................................................................................................70
Source and Destination Configuration....................................................................................................72
Starting and Stopping L-ESS .................................................................................................................76
L-ESS Maintenance and Troubleshooting ..................................................... 77
Using the Maintenance Utility.................................................................................................................78
Using L-ESS Logs ..................................................................................................................................80
L-ESS Server Scripts..............................................................................................................................81
Using the add_project Script..............................................................................................................81
Using FSS Scheduler.....................................................................................................................81
Using Resource Pool Facility.........................................................................................................81
Using the start_serv Script.................................................................................................................82
Configuring Veritas Cluster to Start L-ESS Using FSS Scheduler................................................82
Using the Cleanup Script ...................................................................................................................82
How the Cleanup Script Works......................................................................................................82

▀Contents
▄Cisco ASR 5000 Series External Storage Server Installation and Administration Guide
iv
OL-22979-03
Troubleshooting the L-ESS.....................................................................................................................84
Capturing Server Logs Using Script...................................................................................................87
Requirements.................................................................................................................................87
xDR File Push Functionality ............................................................................ 89
Configuring HDD.....................................................................................................................................90
Configuring Push Functionality...............................................................................................................91
Pushing xDR Files Manually ..............................................................................................................92
L-ESS Directory Structure ......................................................................................................................94
Log Maintenance ....................................................................................................................................95

Cisco ASR 5000 Series External Storage Server Installation and Administration Guide ▄
OL-22979-03
v
About this Guide
This document pertains to the features and functionality that run on and/or that are related to the Cisco® ASR 5000
Chassis.
This preface includes the following sections:
Conventions Used
Contacting Customer Support
Additional Information

About this Guide
▀Conventions Used
▄Cisco ASR 5000 Series External Storage Server Installation and Administration Guide
vi
OL-22979-03
Conventions Used
The following tables describe the conventions used throughout this documentation.
Icon
Notice Type
Description
Information Note
Provides information about important features or instructions.
Caution
Alerts you of potential damage to a program, device, or system.
Warning
Alerts you of potential personal injury or fatality. May also alert you of potential
electrical hazards.
Electrostatic Discharge
(ESD)
Warns you to take proper grounding precautions before handling ESD sensitive
components or devices.
Typeface Conventions
Description
Text represented as a screen
display
This typeface represents text that appears on your terminal screen, for example:
Login:
Text represented as commands
This typeface represents commands that you enter at the CLI, for example:
show ip access-list
This document always gives the full form of a command in lowercase letters. Commands
are not case sensitive.
Text represented as a command
variable
This typeface represents a variable that is part of a command, for example:
show card slot_number
slot_number is a variable representing the desired chassis slot number.
Text represented as menu or sub-
menu names
This typeface represents menus and sub-menus that you access within a software
application, for example:
Click the File menu, then click New.
Command Syntax
Conventions
Description
{ keyword or
variable }
Required keywords and variables are surrounded by braces. They must be entered as part of the
command syntax.
[ keyword or
variable ]
Optional keywords or variables that may or may not be used are surrounded by brackets.

About this Guide
Conventions Used ▀
Cisco ASR 5000 Series External Storage Server Installation and Administration Guide ▄
OL-22979-03
vii
Command Syntax
Conventions
Description
|
Some commands support alternative variables. These “options” are documented within braces or
brackets by separating each variable with a vertical bar.
These variables can be used in conjunction with required or optional keywords or variables. For
example:
{ nonce | timestamp }
OR
[ count number_of_packets | size number_of_bytes ]

About this Guide
▀Contacting Customer Support
▄Cisco ASR 5000 Series External Storage Server Installation and Administration Guide
viii
OL-22979-03
Contacting Customer Support
Go to http://www.cisco.com/cisco/support/ to submit a service request. A valid Cisco account (username and password)
is required to access this site. Please contact your Cisco account representative for additional information.
For Existing Customers with support contracts through Starent Networks: Refer to the support area of
https://support.starentnetworks.com/ for up-to-date product documentation or to submit a service request. A valid
username and password is required to this site. Please contact your local sales or service representative for additional
information.

About this Guide
Additional Information ▀
Cisco ASR 5000 Series External Storage Server Installation and Administration Guide ▄
OL-22979-03
ix
Additional Information
Refer to the following guides for supplemental information about the ASR 5000 chassis:
ASR 5000 Command Line Interface Reference
Statistics and Counters Reference
Thresholding Configuration Guide
Cisco ASR 5000 SNMP MIB Reference
Cisco Web Element Manager Installation and Administration Guide
Product-specific and feature-specific Administration guides
Release Notes that accompany updates and upgrades to StarOS

Cisco ASR 5000 Series External Storage Server Installation and Administration Guide ▄
OL-22979-03
11
Chapter 1
External Storage System Overview
An External Storage System (ESS) is used to collect, store, and report billing information from the Enhanced Charging
Service running on the ASR 5000 platform on short term and long term storage basis. This guide contains instructions
for implementing and maintaining the Local, short-term External Storage Server (L-ESS).
Important: The External Storage System is not a part of the Enhanced Charging Service (ECS) and must be
purchased separately. To purchase ESS, contact your designated sales or service representative.
Important: The procedures in this guide assume that you have installed and configured your chassis including
the ECS installation and configuration as described in the Enhanced Charging Services Administration Guide.
Important: The 9.0 release of ESS works in conjuction with the latest embedded versions of StarOS.

External Storage System Overview
▀Overview
▄Cisco ASR 5000 Series External Storage Server Installation and Administration Guide
12
OL-22979-03
Overview
The CDR subsystem, provides 512 MB of volatile memory on the packet processing card RAM to store accounting
information. This on-board memory is intended as a short-term buffer for accounting information so that billing systems
can periodically retrieve the buffered information for bill generation purposes. However if network outages or other
failures cause billing systems to lose contact with the system, it is possible that the CDR subsystem storage area can be
filled with non-retrieved accounting information. When the storage is filled the CDR subsystem starts deleting the oldest
files to make sure that there is room for new billing files and non-retrieved accounting information can be lost. Using an
external storage server with a large storage volume in close proximity to the chassis ensures room for storing a large
amount of billing data that is not lost by any failure.
The ESS has the capability of simultaneously fetching any types of files from one or more chassis. That is, it can fetch
xDRs like CDR, EDR, NBR, UDR file, etc.
In case of Hard Disk Drive (HDD) support on the chassis, the platform has the capability to push the xDR files to L-
ESS, and then L-ESS forwards these files to the required destinations. If HDD is not configured on the platform, L-ESS
pulls the files from the system and forwards them to the destinations. For information on the push functionality and its
configuration, refer to the xDR File Push Functionality appendix.
The External Storage System (ESS) is designed to be used as a safe storage area. A mediation or billing server within
your network must be configured to collect accounting records from the ESS once it retrieves them.
The External Storage System supports a high level of redundancy for secure charging and billing information for post-
processing of xDRs. This system can store charging data of up to 30 days.
This guide discusses the following topics on External Storage System:
Storage System Components:
Local, short-term external storage server (L-ESS)
The following figure shows a typical organization of External Storage System including L-ESS and billing system with
chassis having a AAA server.

External Storage System Overview
Overview ▀
Cisco ASR 5000 Series External Storage Server Installation and Administration Guide ▄
OL-22979-03
13
Figure 1. ESS Architecture with ECS
The system running with ECS stores xDR files on an L-ESS and billing system collects the files from the L-ESS, and
correlates them with the AAA accounting messages using either 3GPP2-Correlation-IDs on a PDSN system or Charging
IDs on a GGSN system.
L-ESS also pushes xDR files to external applications for post-processing, reporting, subscriber profiling, and trend
analysis.
Local, Short-Term External Storage System
The Local, short-term storage system (L-ESS) is a storage server logically connected with the ASR 5000 and acts as an
integrated network system.
The following are the requirements for the deployment of L-ESS:
High speed dedicated redundant connections to chassis to pull xDR files.
High-speed dedicated and redundant connection with billing system to transfer xDR files.

External Storage System Overview
▀Overview
▄Cisco ASR 5000 Series External Storage Server Installation and Administration Guide
14
OL-22979-03
Different management addresses than the management addresses of the chassis and billing system.
Management interface with support of multiple VLANs.
Redundancy support with two or more geographically co-located or isolated chassis to pull xDRs.
In general L-ESS provides the following functionalities:
Stores copy of records pulled from chassis.
Supports storage of up to 7 days worth of records.
Supports storage capacity of carrier-class redundant.
Provides a means of limiting the amount of bandwidth, in term of kbps, used for the file transfer between chassis
and L-ESS.
Provides a means of archiving/compression of the pulled xDR files for the purpose of extending the storage
capacity.
Provides xDR files to the billing system.

External Storage System Overview
System Requirements ▀
Cisco ASR 5000 Series External Storage Server Installation and Administration Guide ▄
OL-22979-03
15
System Requirements
The requirements described in this section must be met in order to ensure proper operation of the ESS system.
ASR 5000 System Requirements
The following configurations must be implemented, as described in Configuring Enhanced Charging Services chapter
of the Enhanced Charging Services Administration Guide:
ECS must be configured for generating billing records.
An administrator or config-administrator account that is enabled for FTP must be configured.
SSH keys must be generated.
The SFTP subsystem must be enabled.
ESS System Requirements
Important: System requirement recommendation is dependent of different parameters including xDR generation,
compression, deployment scenario, etc. Contact your sales representative for system requirements specific to your ESS
deployment.
Minimum System Recommendations for Stand-alone Deployment of L-ESS
OpenSSL must be installed
Sun Microsystems Netra™ T5220 server
1 x 1.2GHz 8 core UltraSPARC T2 processor with 8GB RAM
2 x 146GB SAS hard drives
Internal CDROM drive
AC or DC power supplies depending on your application
PCI-based video card or Keyboard-Video-Mouse (KVM) card (optional)
Quad Gigabit Ethernet interfaces
Important: It is recommended that you have separate interfaces (in IPMP) for mediation
device and chassis. Also, for given IPMP, the two interfaces should be on different cards.
Operating Environment:
Sun Solaris 9 with Solaris Patch dated January 25, 2005
Sun Solaris 10 with Solaris Patch number 137137-09 dated on or after July 16, 2007 to Nov 2008.
PSMON (installed through ESS installation script)
Perl 5.8.5 (installed through ESS installation script)

External Storage System Overview
▀System Requirements
▄Cisco ASR 5000 Series External Storage Server Installation and Administration Guide
16
OL-22979-03
- or -
Sun Microsystems Netra™ X4450 server for L-ESS
Quad-Core Intel Xeon E7340 (2x4MB L2, 2.40 GHz, 1066 MHz FSB)
32 GB RAM
12 x 300 GB 10000 RPM mirrored SAS disks
Four 10/100/1000 Ethernet ports, 2 PCI-X, 8 PCIe
4 redundant AC power supplies
INtelx64 core 4 socket
Operating Environment:
Sun Solaris 10
Important: For information on which server to be used for L-ESS application, contact your local sales
representative.
Minimum System Recommendations for Cluster Deployment of L-ESS
Sun Microsystems Netra™ T5220 server
1 x 1.2GHz 4 core UltraSPARC T2 processor with 8GB RAM
2 x 146GB SAS hard drives
Quad Gigabit Ethernet interfaces
Important: It is recommended that you have separate interfaces (in IPMP) for
mediation device and chassis. Also, for given IPMP, the two interfaces should be on
different cards.
Internal CDROM drive
AC or DC power supplies depending on your application
Fiber channel (FC) based Common Storage System for Servers (Sun Storage Tek 2540)
PCI Dual FC 4GB HBA
Dual RAID Controllers
5 x 300GB 15K drives
AC or DC power supplies depending upon your application

Cisco ASR 5000 Series External Storage Server Installation and Administration Guide ▄
OL-22979-03
17
Chapter 2
Veritas Cluster Installation and Management
The cluster mode functionality enables L-ESS to provide high availability and critical redundancy support to retrieve
CDRs in failure of any one of the systems. An L-ESS cluster comprises two L-ESS systems, or nodes, that work
together as a single, continuously available system to provide applications, system resources, and data to L-ESS users.
Each L-ESS node on a cluster is a fully functional, standalone system. However, in a clustered environment, the L-ESS
nodes are connected by an interconnected network and work together as a single entity to provide increased data
availability.
Highly available clusters provide nearly continuous access to data and applications by keeping the cluster running
through failures that would normally bring down a single server system.
The cluster setup offers several advantages over traditional single-server systems. These advantages include:
Support for failover and scalable services
Capacity for modular growth
Low entry price compared to traditional hardware fault-tolerant systems
Reduce or eliminate system downtime because of software or hardware failure
Ensure availability of data and applications to L-ESS user, regardless of the kind of failure that would normally
take down a single-server system.
Provide enhanced availability of the system by enabling you to perform maintenance without shutting down the
entire cluster.
Following are the cluster components that work with L-ESS to provide this functionality:
L-ESS Cluster Node
A L-ESS cluster node is a L-ESS server that runs both the L-ESS Application software and Cluster Agent
software. The Cluster Agent enables carrier to network two L-ESS nodes in a cluster. Every L-ESS node in the
cluster is aware when another L-ESS node joins or leaves the cluster. Also, every L-ESS node in the cluster is
aware of the resources that are running locally as well as the resources that are running on the other L-ESS
cluster nodes.
Each L-ESS cluster node is a standalone server that runs its own processes. These processes communicate with
one another to form what looks like (to a network client) a single system that co-operatively provides
applications, system resources, and data to L-ESS users.
Common Storage System
A common storage system is a fiber channel (FC) -based cluster storage with FC drives for the servers in the
cluster environment. It is interconnected with L-ESS cluster nodes with carrier class network connectivity to
provide high level redundant storage and backup support for CDRs. It serves as common storage for all
connected L-ESS cluster nodes.
This system provides high storage scalability and redundancy with RAID support.
This chapter includes the following topics:
L-ESS Cluster Functional Description
Installing Hardware

Veritas Cluster Installation and Management
▀System Requirements
▄Cisco ASR 5000 Series External Storage Server Installation and Administration Guide
18
OL-22979-03
Configuring Storage Array
Configuring Veritas Volume Manager and Veritas Cluster
Monitoring the Cluster
Setup of rootdisk Encapsulation and Mirroring
Testing the Cluster
Once the Veritas Volume Manager and Veritas Cluster are configured, proceed to the installation of L-ESS application
on the L-ESS Server. For detailed instructions, refer to the L-ESS Installation and Configuration chapter of this guide.
Then, perform the cluster monitoring and rootdisk encapsulation processes.

Veritas Cluster Installation and Management
L-ESS Cluster Functional Description ▀
Cisco ASR 5000 Series External Storage Server Installation and Administration Guide ▄
OL-22979-03
19
L-ESS Cluster Functional Description
L-ESS clustering application provides the support to two discreet L-ESS servers for retrieving and storing xDRs from
the chassis at a distribution node on a single IP address/network element for the billing system.
Both the L-ESS nodes (L-ESS1 and L-ESS2) are configured identically from the standpoint of the retrieval and storage
of the xDRs to support the following:
The active L-ESS (either L-ESS1 or L-ESS2) is configured to retrieve xDRs from any and all local chassis in
pre-defined intervals and the xDRs are stored on shared disk (between active and standby) by the active so that
whenever active goes down and standby takes over, it has access to fetched data as data is on the shared disk.
The directory structure of both L-ESS1 and L-ESS2 is identical and conform to the carrier standards. A
/fetched_data directory under <less_install_dir>/ess is used to store initial retrieval of the xDRs from the
chassis.
From a process flow perspective, the interaction of the clustered L-ESS and the ECS is as follows:
The L-ESS(s) is statically configured with chassis to pull xDRs.
The chassis continually generates and groups individual records into xDRs, which are marked as a 'closed' xDR
file based on pre-defined criteria.
The active L-ESS uses SFTP to access the chassis and retrieve all closed xDRs for storage in the /fetched_data
directory.
Active L-ESS fetches xDR files for eventual retrieval by the billing system.

Veritas Cluster Installation and Management
▀Installing Hardware
▄Cisco ASR 5000 Series External Storage Server Installation and Administration Guide
20
OL-22979-03
Installing Hardware
To install the hardware components required for the installation of L-ESS cluster:
Step 1 Rack the Sun Netra T5220 servers and storage array and connect power to each of them.
Step 2 Connect Ethernet port 0 on each server to an Ethernet switch.
Step 3 Connect Ethernet port 1 on server 1 to Ethernet port 1 on server 2 with a cross-over cable.
Step 4 Connect Ethernet port 2 on server 1 to Ethernet port 2 on server 2 with a cross-over cable.
Step 5 Connect a terminal (pc with terminal emulation such as HyperTerm) to the console port. Settings for the console are
9600 8, 1, N. Console cable and DB9 to RJ45 adapter are included with each server.
Step 6 Connect one SCSI cable from CH 0 on the Storage Array to Single Bus Conf as shown in the following figure. DO
NOT make any connections to Sun Servers at this time.
Step 7 Connect the Ethernet ports on each array controller to an Ethernet switch.
Step 8 Insert install DVD into DVD-ROM in the first Sun server. Make sure the server is NOT cabled to the storage array.
Step 9 Power on the server.
Step 10 Wait for the ok prompt on the console.
Step 11 To boot the machine from the DVD, enter:
ok> boot cdrom install
Step 12 The install will run for some time. After the image has been loaded, you will be prompted for the host information
shown below:
# Please enter the desired hostname for this machine.
# Please enter the desired IP address for bge0.
# Please enter the netmask for bge0.
# Please enter the default router for bge0.
Step 13 After entering hostname, IP address, netmask, and default router information, you must confirm the inputs.
Please verify your configuration information:
hostname: