Cisco ASR 9000 Serie User manual

Cisco ASR 9000 Series Aggregation Services Router System
Monitoring Configuration Guide, Release 4.2.x
First Published: 2011-12-01
Last Modified: 2012-06-01
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,
INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH
THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY,
CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version
of the UNIX operating system. All rights reserved. Copyright ©1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS.
CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT
LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS
HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network
topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional
and coincidental.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://
www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1110R)
©2012 Cisco Systems, Inc. All rights reserved.

CONTENTS
Preface Preface xv
Changes to This Document xv
Obtaining Documentation and Submitting a Service Request xv
CHAPTER 1 Implementing and Monitoring Alarms and Alarm Log Correlation 1
Prerequisites for Implementing and Monitoring Alarms and Alarm Log Correlation 2
Information About Implementing Alarms and Alarm Log Correlation 2
Alarm Logging and Debugging Event Management System 2
Correlator 3
System Logging Process 4
Alarm Logger 4
Logging Correlation 4
Correlation Rules 4
Types of Correlation 5
Application of Rules and Rule Sets 5
Root Message and Correlated Messages 5
Alarm Severity Level and Filtering 6
Bistate Alarms 6
Capacity Threshold Setting for Alarms 7
Hierarchical Correlation 7
Context Correlation Flag 7
Duration Timeout Flags 8
Reparent Flag 8
Reissue Nonbistate Flag 8
Internal Rules 9
SNMP Alarm Correlation 9
How to Implement and Monitor Alarm Management and Logging Correlation 9
Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 4.2.x
iii

Configuring Logging Correlation Rules 9
Configuring Logging Correlation Rule Sets 10
Configuring Root-cause and Non-root-cause Alarms 11
Configuring Hierarchical Correlation Rule Flags 13
Applying Logging Correlation Rules 14
Applying Logging Correlation Rule Sets 16
Modifying Logging Events Buffer Settings 17
Modifying Logging Correlator Buffer Settings 19
Displaying Alarms by Severity and Severity Range 20
Displaying Alarms According to a Time Stamp Range 22
Displaying Alarms According to Message Group and Message Code 23
Displaying Alarms According to a First and Last Range 24
Displaying Alarms by Location 25
Displaying Alarms by Event Record ID 26
Displaying the Logging Correlation Buffer Size, Messages, and Rules 27
Clearing Alarm Event Records and Resetting Bistate Alarms 28
Defining SNMP Correlation Buffer Size 30
Defining SNMP Rulesets 31
Configuring SNMP Correlation Rules 31
Applying SNMP Correlation Rules 32
Applying SNMP Correlation Ruleset 33
Configuration Examples for Alarm Management and Logging Correlation 34
Increasing the Severity Level for Alarm Filtering to Display Fewer Events and Modifying
the Alarm Buffer Size and Capacity Threshold: Example 34
Configuring a Nonstateful Correlation Rule to Permanently Suppress Node Status Messages:
Example 34
Configuring a Stateful Correlation Rule for LINK UPDOWN and SONET ALARM Alarms:
Example 36
Additional References 37
CHAPTER 2 Configuring and Managing Embedded Event Manager Policies 41
Prerequisites for Configuring and Managing Embedded Event Manager Policies 42
Information About Configuring and Managing Embedded Event Manager Policies 42
Event Management 42
System Event Detection 42
Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 4.2.x
iv
Contents

Policy-Based Event Response 43
Reliability Metrics 43
System Event Processing 43
Embedded Event Manager Management Policies 43
Embedded Event Manager Scripts and the Scripting Interface (Tcl) 44
Script Language 45
Regular Embedded Event Manager Scripts 45
Embedded Event Manager Callback Scripts 46
Embedded Event Manager Policy Tcl Command Extension Categories 46
Cisco File Naming Convention for Embedded Event Manager 47
Embedded Event Manager Built-in Actions 48
Application-specific Embedded Event Management 49
Event Detection and Recovery 49
General Flow of EEM Event Detection and Recovery 49
System Manager Event Detector 50
Timer Services Event Detector 51
Syslog Event Detector 51
None Event Detector 51
Watchdog System Monitor Event Detector 52
Distributed Event Detectors 53
Embedded Event Manager Event Scheduling and Notification 53
Reliability Statistics 53
Hardware Card Reliability Metric Data 53
Process Reliability Metric Data 54
How to Configure and Manage Embedded Event Manager Policies 55
Configuring Environmental Variables 55
Environment Variables 55
Registering Embedded Event Manager Policies 56
Embedded Event Manager Policies 56
How to Write Embedded Event Manager Policies Using Tcl 59
Registering and Defining an EEM Tcl Script 59
Displaying EEM Registered Policies 61
Unregistering EEM Policies 61
Suspending EEM Policy Execution 62
Managing EEM Policies 63
Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 4.2.x
v
Contents

Displaying Software Modularity Process Reliability Metrics Using EEM 64
Sample EEM Policies 64
Programming EEM Policies with Tcl 66
Tcl Policy Structure and Requirements 66
EEM Entry Status 68
EEM Exit Status 68
EEM Policies and Cisco Error Number 69
_cerrno: 32-Bit Error Return Values 69
Error Class Encodings for XY 70
Creating an EEM User Tcl Library Index 74
Creating an EEM User Tcl Package Index 77
Configuration Examples for Event Management Policies 80
Environmental Variables Configuration: Example 80
User-Defined Embedded Event Manager Policy Registration: Example 80
Display Available Policies: Example 81
Display Embedded Event Manager Process: Example 81
Configuration Examples for Writing Embedded Event Manager Policies Using Tcl 82
EEM Event Detector Demo: Example 82
EEM Sample Policy Descriptions 82
Event Manager Environment Variables for the Sample Policies 82
Registration of Some EEM Policies 84
Basic Configuration Details for All Sample Policies 85
Using the Sample Policies 85
Running the sl_intf_down.tcl Sample Policy 85
Running the tm_cli_cmd.tcl Sample Policy 86
Running the tm_crash_reporter.tcl Sample Policy 86
Running the tm_fsys_usage.tcl Sample Policy 87
Programming Policies with Tcl: Sample Scripts Example 87
tm_cli_cmd.tcl Sample Policy 87
sl_intf_down.tcl Sample Policy 90
Tracing Tcl set Command Operations: Example 92
Additional References 92
Embedded Event Manager Policy Tcl Command Extension Reference 93
Embedded Event Manager Event Registration Tcl Command Extensions 94
event_register_appl 94
Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 4.2.x
vi
Contents

event_register_cli 95
event_register_config 97
event_register_counter 98
event_register_hardware 99
event_register_none 101
event_register_oir 102
event_register_process 103
event_register_snmp 105
event_register_snmp_notification 108
event_register_stat 109
event_register_syslog 112
event_register_timer 114
event_register_timer_subscriber 119
event_register_track 121
event_register_wdsysmon 123
Embedded Event Manager Event Information Tcl Command Extension 129
event_reqinfo 129
event_reqinfo_multi 145
Embedded Event Manager Event Publish Tcl Command Extension 145
event_publish appl 145
Embedded Event Manager Multiple Event Support Tcl Command Extensions 148
Attribute 148
Correlate 148
Trigger 149
Embedded Event Manager Action Tcl Command Extensions 150
action_process 150
action_program 152
action_script 153
action_setver_prior 153
action_setnode 154
action_syslog 154
action_track_read 155
Embedded Event Manager Utility Tcl Command Extensions 156
appl_read 156
appl_reqinfo 157
Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 4.2.x
vii
Contents

appl_setinfo 157
counter_modify 158
fts_get_stamp 159
register_counter 160
register_timer 161
timer_arm 163
timer_cancel 165
unregister_counter 166
Embedded Event Manager System Information Tcl Command Extensions 167
sys_reqinfo_cpu_all 167
sys_reqinfo_crash_history 168
sys_reqinfo_mem_all 169
sys_reqinfo_proc 171
sys_reqinfo_proc_all 173
sys_reqinfo_proc_version 173
sys_reqinfo_routername 174
sys_reqinfo_syslog_freq 174
sys_reqinfo_syslog_history 175
sys_reqinfo_stat 176
sys_reqinfo_snmp 177
sys_reqinfo_snmp_trap 178
sys_reqinfo_snmp_trapvar 178
SMTP Library Command Extensions 178
smtp_send_email 179
smtp_subst 180
CLI Library Command Extensions 181
cli_close 181
cli_exec 182
cli_get_ttyname 182
cli_open 183
cli_read 183
cli_read_drain 184
cli_read_line 184
cli_read_pattern 185
cli_write 186
Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 4.2.x
viii
Contents

Tcl Context Library Command Extensions 189
context_retrieve 189
context_save 192
CHAPTER 3 Implementing IP Service Level Agreements 195
Prerequisites for Implementing IP Service Level Agreements 196
Restrictions for Implementing IP Service Level Agreements 196
Information About Implementing IP Service Level Agreements 198
About IP Service Level Agreements Technology 198
Service Level Agreements 198
Benefits of IP Service Level Agreements 200
Measuring Network Performance with IP Service Level Agreements 200
Operation Types for IP Service Level Agreements 202
IP SLA Responder and IP SLA Control Protocol 203
Response Time Computation for IP SLA 204
IP SLA VRF Support 204
IP SLA Operation Scheduling 205
IP SLA—Proactive Threshold Monitoring 205
IP SLA Reaction Configuration 205
IP SLA Threshold Monitoring and Notifications 205
MPLS LSP Monitoring 205
How MPLS LSP Monitoring Works 206
BGP Next-hop Neighbor Discovery 206
IP SLA LSP Ping and LSP Traceroute Operations 207
Proactive Threshold Monitoring for MPLS LSP Monitoring 208
Multi-operation Scheduling for the LSP Health Monitor 208
LSP Path Discovery 208
How to Implement IP Service Level Agreements 209
Configuring IP Service Levels Using the UDP Jitter Operation 209
Enabling the IP SLA Responder on the Destination Device 209
Configuring and Scheduling a UDP Jitter Operation on the Source Device 210
Prerequisites for Configuring a UDP Jitter Operation on the Source Device 212
Configuring and Scheduling a Basic UDP Jitter Operation on the Source Device 212
Configuring and Scheduling a UDP Jitter Operation with Additional Characteristics 214
Configuring the IP SLA for a UDP Echo Operation 219
Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 4.2.x
ix
Contents

Prerequisites for Configuring a UDP Echo Operation on the Source Device 219
Configuring and Scheduling a UDP Echo Operation on the Source Device 219
Configuring and Scheduling a UDP Echo Operation with Optional Parameters on the
Source Device 222
Configuring an ICMP Echo Operation 226
Configuring and Scheduling a Basic ICMP Echo Operation on the Source Device 226
Configuring and Scheduling an ICMP Echo Operation with Optional Parameters on
the Source Device 229
Configuring the ICMP Path-echo Operation 232
Configuring and Scheduling a Basic ICMP Path-echo Operation on the Source Device
232
Configuring and Scheduling an ICMP Path-echo Operation with Optional Parameters
on the Source Device 235
Configuring the ICMP Path-jitter Operation 238
Configuring and Scheduling a Basic ICMP Path-jitter Operation 239
Configuring and Scheduling an ICMP Path-jitter Operation with Additional
Parameters 242
Configuring IP SLA MPLS LSP Ping and Trace Operations 246
Configuring and Scheduling an MPLS LSP Ping Operation 246
Configuring and Scheduling an MPLS LSP Trace Operation 250
Configuring IP SLA Reactions and Threshold Monitoring 254
Configuring Monitored Elements for IP SLA Reactions 254
Configuring Triggers for Connection-Loss Violations 254
Configuring Triggers for Jitter Violations 255
Configuring Triggers for Packet Loss Violations 255
Configuring Triggers for Round-Trip Violations 256
Configuring Triggers for Timeout Violations 257
Configuring Triggers for Verify Error Violations 258
Configuring Threshold Violation Types for IP SLA Reactions 259
Generating Events for Each Violation 260
Generating Events for Consecutive Violations 260
Generating Events for X of Y Violations 261
Generating Events for Averaged Violations 262
Specifying Reaction Events 263
Configuring the MPLS LSP Monitoring Instance on a Source PE Router 265
Cisco ASR 9000 Series Aggregation Services Router System Monitoring Configuration Guide, Release 4.2.x
x
Contents