Oracle E5-APP-B Specification sheet

Oracle®Communications
EAGLE LNP Application Processor
Alarms and Maintenance
Release 10.1
E76220 Revision 1
July 2016

Oracle Communications EAGLE LNP Application Processor Alarms and Maintenance, Release 10.1
Copyright ©2000, 2016, Oracle and/or its affiliates. All rights reserved.
This software and related documentation are provided under a license agreement containing restrictions
on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your
license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license,
transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse
engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is
prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free.
If you find any errors, please report them to us in writing.
If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it
on behalf of the U.S. Government, then the following notice is applicable:
U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software,
any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users
are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and
agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation
of the programs, including any operating system, integrated software, any programs installed on the
hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the
programs. No other rights are granted to the U.S. Government.
This software or hardware is developed for general use in a variety of information management applications.
It is not developed or intended for use in any inherently dangerous applications, including applications
that may create a risk of personal injury. If you use this software or hardware in dangerous applications,
then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to
ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by
use of this software or hardware in dangerous applications.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks
of their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks
are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD,
Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced
Micro Devices. UNIX is a registered trademark of The Open Group.
This software or hardware and documentation may provide access to or information about content, products,
and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly
disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise
set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not
be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content,
products, or services, except as set forth in an applicable agreement between you and Oracle.

Table of Contents
Chapter 1: Introduction...............................................................................12
Overview...............................................................................................................................................13
Scope and Audience............................................................................................................................13
Documentation Admonishments.......................................................................................................13
Manual Organization...........................................................................................................................14
My Oracle Support (MOS)..................................................................................................................14
Emergency Response...........................................................................................................................14
Related Specifications..........................................................................................................................15
Customer Training...............................................................................................................................15
Locate Product Documentation on the Oracle Help Center Site...................................................15
Chapter 2: Maintenance..............................................................................17
Introduction..........................................................................................................................................18
Preventive Maintenance......................................................................................................................18
Daily Maintenance Procedures..............................................................................................18
System Health Check Overview........................................................................................................24
Health Check Outputs.............................................................................................................26
Running the System Health Check....................................................................................................30
Running syscheck from the Command line.........................................................................30
Running syscheck Through the ELAP GUI..........................................................................30
Running syscheck Using the syscheck Login.......................................................................31
Chapter 3: Problem Detection and Reporting........................................33
Detecting and Reporting Problems...................................................................................................34
E5-APP-B Card LEDs...........................................................................................................................34
Displaying Errors on ELAP GUI........................................................................................................37
Unsolicited Alarm and Information Messages................................................................................38
Chapter 4: Recovery Support.....................................................................39
Recovering from Problems.................................................................................................................40
Restoring the RTDB from Backup Files................................................................................40
Recovering From Alarms....................................................................................................................42
iii
E76220 Revision 1, July 2016

Decode Alarm Strings..............................................................................................................42
Determine Alarm Cause..........................................................................................................43
Chapter 5: Platform and Application Alarms.........................................45
Alarm Categories.................................................................................................................................46
MPS Alarm Recovery Procedures......................................................................................................50
Critical Platform Alarms.....................................................................................................................50
1000000000002000 - Uncorrectable ECC Memory Error.....................................................50
1000000000008000 –Server NTP Daemon lost NTP synchronization for extended
time.......................................................................................................................................50
1000000000010000 –Server's time has gone backwards.....................................................50
Critical Application Alarms................................................................................................................51
2000000000000001 - LSMS DB Maintenance Required.......................................................51
Major Platform Alarms........................................................................................................................51
3000000000000001 –Server fan failure..................................................................................51
3000000000000002 - Server Internal Disk Error....................................................................52
3000000000000008 - Server Platform Error...........................................................................52
3000000000000010 - Server File System Error......................................................................52
3000000000000020 - Server Platform Process Error.............................................................53
3000000000000080 - Server Swap Space Shortage Failure..................................................53
3000000000000100 - Server provisioning network error.....................................................53
3000000000000200 –Server Eagle Network A error............................................................55
3000000000000400 –Server Eagle Network B error............................................................57
3000000000000800 –Server Sync network error..................................................................60
3000000000001000 - Server Disk Space Shortage Error.......................................................62
3000000000002000 - Server Default Route Network Error.................................................62
3000000000004000 - Server Temperature Error....................................................................68
3000000000008000 - Server Mainboard Voltage Error........................................................69
3000000000010000 - Server Power Feed Error......................................................................69
3000000000020000 - Server Disk Health Test Error.............................................................70
3000000000040000 - Server Disk Unavailable Error............................................................70
3000000000080000 - Device Error...........................................................................................70
3000000000100000 - Device Interface Error..........................................................................70
3000000000200000 - Correctable ECC Memory Error.........................................................70
3000000400000000 - Multipath device access link problem...............................................71
3000000800000000 –Switch Link Down Error.....................................................................71
3000001000000000 - Half-open Socket Limit........................................................................72
3000002000000000 - Flash Program Failure..........................................................................72
3000004000000000 - Serial Mezzanine Unseated.................................................................72
3000000008000000 - Server HA Keepalive Error..................................................................72
iv
E76220 Revision 1, July 2016

3000000010000000 - DRBD block device can not be mounted...........................................73
3000000020000000 - DRBD block device is not being replicated to peer..........................73
3000000040000000 - DRBD peer needs intervention...........................................................73
3000020000000000 - Server NTP Daemon never synchronized.........................................73
Major Application Alarms..................................................................................................................74
4000000000000001 - Mate ELAP Unavailable.......................................................................74
4000000000000002 - RTDB Mate Unavailable......................................................................74
4000000000000004 - Congestion.............................................................................................75
4000000000000008 - File System Full.....................................................................................75
4000000000000010 - Log Failure.............................................................................................76
4000000000000040 - Fatal Software Error..............................................................................76
4000000000000080 - RTDB Corrupt.......................................................................................76
4000000000000100 - RTDB Inconsistent................................................................................76
4000000000000200 - RTDB Incoherent...................................................................................76
4000000000000800 - Transaction Log Full.............................................................................77
4000000000001000 - RTDB 100% Full....................................................................................77
4000000000002000 - RTDB Resynchronization In Progress................................................77
4000000000004000 - RTDB Reload Is Required....................................................................77
4000000000400000 - LVM Snapshot detected that is too old..............................................77
4000000000800000 - LVM Snapshot detected that is too full.............................................78
4000000001000000 - LVM Snapshot detected with invalid attributes..............................78
4000000002000000 - DRBD Split Brain..................................................................................78
4000000010000000 - An instance of Snapmon already running........................................79
Minor Platform Alarms.......................................................................................................................79
1000000000000001 –Breaker panel feed unavailable..........................................................79
5000000000000001 - Server Disk Space Shortage Warning................................................79
5000000000000002 - Server Application Process Error.......................................................80
5000000000000004 - Server Hardware Configuration Error..............................................81
5000000000000008 - Server RAM Shortage Warning..........................................................81
5000000000000020 - Server Swap Space Shortage Warning...............................................81
5000000000000040 - Server Default Router Not Defined....................................................81
5000000000000080 –Server temperature warning...............................................................84
5000000000000100 - Server Core File Detected....................................................................85
5000000000000200 - Server NTP Daemon Not Synchronized............................................86
5000000000000400 - Server CMOS Battery Voltage Low....................................................86
5000000000000800 - Server Disk Self Test Warning............................................................86
5000000000001000 - Device Warning.....................................................................................86
5000000000002000 - Device Interface Warning....................................................................86
5000000000004000 - Server Reboot Watchdog Initiated.....................................................87
5000000000008000 - Server HA Failover Inhibited .............................................................87
5000000000010000 - Server HA Active To Standby Transition .........................................87
v
E76220 Revision 1, July 2016

5000000000020000 - Server HA Standby To Active Transition..........................................87
5000000000040000 - Platform Health Check Failure...........................................................87
5000000000080000 - NTP Offset Check Failure....................................................................88
5000000000100000 - NTP Stratum Check Failure................................................................88
5000000020000000 –Server Kernel Dump File Detected....................................................88
5000000040000000 –TPD Upgrade Failed............................................................................88
5000000080000000–Half Open Socket Warning Limit.......................................................89
5000000000200000 - SAS Presence Sensor Missing..............................................................89
5000000000400000 - SAS Drive Missing................................................................................89
5000000000800000 - DRBD failover busy..............................................................................89
5000000001000000 - HP disk resync.......................................................................................90
5000000400000000 –NTP Source Server is not able to provide correct time...................90
Minor Application Alarms..................................................................................................................90
4000000000020000 - Automatic RTDB Backup is not configured......................................90
6000000000000010 - Minor Software Error...........................................................................90
6000000000000200 - RTDB Backup Failed.............................................................................91
6000000000000400 - Automatic RTDB Backup Failed.........................................................91
6000000000000800 - Automatic Backup cron entry modified............................................91
6000000000002000 - Configurable Quantity Threshold Exceeded....................................91
6000000000020000 - Automatic RTDB Backup is not configured......................................91
Chapter 6: Field Replaceable Units..........................................................92
Introduction..........................................................................................................................................93
Safety Information................................................................................................................................93
E5-APP-B Card FRUs and Part Numbers.........................................................................................93
Removing and Replacing E5-APP-B Cards......................................................................................94
Removing an E5-APP-B Card.................................................................................................94
Replacing an E5-APP-B Card.................................................................................................95
Removing and Replacing a Drive Module Assembly.....................................................................99
Removing a Drive Module Assembly.................................................................................101
Replacing a Drive Module Assembly..................................................................................104
Appendix A: General Procedures...........................................................107
Introduction........................................................................................................................................108
Accessing the ELAP GUI Interface..................................................................................................108
Connecting to the Server Command Line......................................................................................109
Using ssh to Connect to the Server Command Line.........................................................109
Connecting a Local Access Terminal to Server’s Serial Port............................................110
Accessing the ELAP Text Interface..................................................................................................111
Saving Logs Using the ELAP GUI...................................................................................................112
vi
E76220 Revision 1, July 2016

Restarting the ELAP Software..........................................................................................................113
Rebooting the MPS.............................................................................................................................115
Glossary.............................................................................................................................117
vii
E76220 Revision 1, July 2016

List of Figures
Figure 1: Backup the RTDB.............................................................................................................................19
Figure 2: Backup the RTDB Confirmation....................................................................................................19
Figure 3: Start ELAP Software........................................................................................................................19
Figure 4: Start ELAP Software - Success.......................................................................................................20
Figure 5: Automatic RTDB Backup Menu Item...........................................................................................21
Figure 6: Automatic RTDB Backup GUI Screen..........................................................................................22
Figure 7: Login Window.................................................................................................................................30
Figure 8: Run Health Check............................................................................................................................31
Figure 9: Displaying System Health Check on ELAP GUI........................................................................31
Figure 10: E5-APP-B Card LEDs....................................................................................................................35
Figure 11: Errors Displayed on ELAP GUI...................................................................................................37
Figure 12: Viewing Alarm Details.................................................................................................................37
Figure 13: Stop ELAP Software......................................................................................................................41
Figure 14: Stop ELAP Software - Success.....................................................................................................41
Figure 15: Restoring the RTDB.......................................................................................................................41
Figure 16: Restore the RTDB Confirm...........................................................................................................42
Figure 17: Decode MPS Alarm Screen...........................................................................................................43
Figure 18: Decoded MPS Alarm Information..............................................................................................43
Figure 19: Manage Logs and Backups...........................................................................................................62
Figure 20: Enable LSMS Connection Window.............................................................................................65
Figure 21: LSMS Connection Enabled...........................................................................................................66
Figure 22: Enable LSMS Connection Window.............................................................................................67
viii
E76220 Revision 1, July 2016

Figure 23: LSMS Connection Enabled...........................................................................................................67
Figure 24: Coherent RTDB Status..................................................................................................................75
Figure 25: Manage Logs and Backups...........................................................................................................80
Figure 26: Enable LSMS Connection Window.............................................................................................84
Figure 27: LSMS Connection Enabled...........................................................................................................84
Figure 28: E5-APP-B Card Eject Hardware Switch, UNLOCKED............................................................94
Figure 29: E5-APP-B Card UNLOCKED.......................................................................................................95
Figure 30: E5-APP-B Card UNLOCKED.......................................................................................................97
Figure 31: E5-APP-B Card Inject Levers.......................................................................................................98
Figure 32: E5-APP-B Card Inject Hardware Switch, LOCKED.................................................................99
Figure 33: Drive Module Released...............................................................................................................102
Figure 34: Drive Module UNLOCKED.......................................................................................................103
Figure 35: Drive Module Status...................................................................................................................103
Figure 36: Drive Module Removal...............................................................................................................104
Figure 37: Drive Module Replacement.......................................................................................................104
Figure 38: Drive Module Locked.................................................................................................................105
Figure 39: ELAP User Interface Screen.......................................................................................................108
Figure 40: E5-APP-B on ELAP Console Connectivity...............................................................................110
Figure 41: Laptop Connection Settings.......................................................................................................111
Figure 42: Capture Logs File Screen............................................................................................................112
Figure 43: Capture Logs Success..................................................................................................................112
Figure 44: Deleting Captured Log Files......................................................................................................113
Figure 45: Delete Log Files Success.............................................................................................................113
Figure 46: Stop Software Confirmation.......................................................................................................114
Figure 47: Stop Software Completion Screen.............................................................................................114
ix
E76220 Revision 1, July 2016

Figure 48: Start ELAP Software....................................................................................................................114
Figure 49: Start Software Completion Screen.............................................................................................115
Figure 50: Request Reboot of the MPS........................................................................................................115
Figure 51: Confirm Requested Reboot the MPS........................................................................................116
Figure 52: Reboot Information.....................................................................................................................116
x
E76220 Revision 1, July 2016

List of Tables
Table 1: Admonishments................................................................................................................................13
Table 2: System Health Check Operation.....................................................................................................24
Table 3: E5-APP-B LED Table.........................................................................................................................36
Table 4: Platform and Application Alarms...................................................................................................46
Table 5: Server Environmental Conditions..................................................................................................68
Table 6: Server Environmental Conditions..................................................................................................84
Table 7: Usernames........................................................................................................................................109
xi
E76220 Revision 1, July 2016

Chapter
1
Introduction
This chapter provides a brief description of this
manual. This chapter also includes the scope,
Topics:
•Overview.....13 audience, and organization of the manual; how to
•Scope and Audience.....13 find related publications; and how to contact
documentation for assistance.
•Documentation Admonishments.....13
•Manual Organization.....14
•My Oracle Support (MOS).....14
•Emergency Response.....14
•Related Specifications.....15
•Customer Training.....15
•Locate Product Documentation on the Oracle Help
Center Site.....15
12
E76220 Revision 1, July 2016

Overview
This manual contains the information necessary for the maintenance of the E5-APP-B that supports
the EAGLE LNP Application Processor (ELAP). Included are an overview of the E5-APP-B architecture
and functions, routine operational procedures, preventative maintenance techniques, and corrective
maintenance procedures.
Scope and Audience
The scope of this manual covers platform and application alarms, troubleshooting and recovery
procedures, and the System Healthcheck Utility, an application that generates a log file that can be
provided to My Oracle Support (MOS) for alarm resolution. It is intended to aid maintenance personnel
in resolution of ELAP alarms. When instructed by the application, use this manual to locate the platform
alarm number and its recovery procedure.
Documentation Admonishments
Admonishments are icons and text throughout this manual that alert the reader to assure personal
safety, to minimize possible service interruptions, and to warn of the potential for equipment damage.
Table 1: Admonishments
DescriptionIcon
Danger:
(This icon and text indicate the possibility of
personal injury.)
Warning:
(This icon and text indicate the possibility of
equipment damage.)
Caution:
(This icon and text indicate the possibility of
service interruption.)
Topple:
(This icon and text indicate the possibility of
personal injury and equipment damage.)
13
E76220 Revision 1, July 2016
IntroductionAlarms and Maintenance

Manual Organization
This manual is organized into the following chapters:
Introduction contains general information about manual organization, scope and audience, related
documentation, how to locate customer documentation on the Customer Support site, how to get
technical assistance, and RMA requirements.
Maintenance provides the preventative maintenance procedures and system health checks.
Problem Detection and Reporting provides information about problem detection and reporting.
Recovery Support describes the recommended backing up of the RTDB and presents additional recovery
support procedures that may be referred to by alarms recovery actions.
Platform and Application Alarms provides recovery procedures for platform and application alarms.
Field Replaceable Units (FRUs) provides instruction on replacing E5-APP-B cards and FRUs
General Procedures contains miscellaneous general procedures that are referred to within this manual.
My Oracle Support (MOS)
MOS (https://support.oracle.com) is your initial point of contact for all product support and training
needs. A representative at Customer Access Support (CAS) can assist you with MOS registration.
Call the CAS main number at 1-800-223-1711 (toll-free in the US), or call the Oracle Support hotline
for your local country from the list at http://www.oracle.com/us/support/contact/index.html. When calling,
make the selections in the sequence shown below on the Support telephone menu:
1. Select 2for New Service Request
2. Select 3for Hardware, Networking and Solaris Operating System Support
3. Select one of the following options:
•For Technical issues such as creating a new Service Request (SR), Select 1
•For Non-technical issues such as registration or assistance with MOS, Select 2
You will be connected to a live agent who can assist you with MOS registration and opening a support
ticket.
MOS is available 24 hours a day, 7 days a week, 365 days a year.
Emergency Response
In the event of a critical service situation, emergency response is offered by the Customer Access
Support (CAS) main number at 1-800-223-1711 (toll-free in the US), or by calling the Oracle Support
hotline for your local country from the list at http://www.oracle.com/us/support/contact/index.html. The
emergency response provides immediate coverage, automatic escalation, and other features to ensure
that the critical situation is resolved as rapidly as possible.
14
E76220 Revision 1, July 2016
IntroductionAlarms and Maintenance

A critical situation is defined as a problem with the installed equipment that severely affects service,
traffic, or maintenance capabilities, and requires immediate corrective action. Critical situations affect
service and/or system operation resulting in one or several of these situations:
•A total system failure that results in loss of all transaction processing capability
•Significant reduction in system capacity or traffic handling capability
•Loss of the system’s ability to perform automatic system reconfiguration
•Inability to restart a processor or the system
•Corruption of system databases that requires service affecting corrective actions
•Loss of access for maintenance or recovery operations
•Loss of the system ability to provide any required critical or major trouble notification
Any other problem severely affecting service, capacity/traffic, billing, and maintenance capabilities
may be defined as critical by prior discussion and agreement with Oracle.
Related Specifications
For information about additional publications that are related to this document, refer to the Oracle
Help Center site. See Locate Product Documentation on the Oracle Help Center Site for more information
on related product publications.
Customer Training
Oracle University offers training for service providers and enterprises. Visit our web site to view, and
register for, Oracle Communications training:
http://education.oracle.com/communication
To obtain contact phone numbers for countries or regions, visit the Oracle University Education web
site:
www.oracle.com/education/contacts
Locate Product Documentation on the Oracle Help Center Site
Oracle Communications customer documentation is available on the web at the Oracle Help Center
(OHC) site, http://docs.oracle.com. You do not have to register to access these documents. Viewing these
files requires Adobe Acrobat Reader, which can be downloaded at http://www.adobe.com.
1. Access the Oracle Help Center site at http://docs.oracle.com.
2. Click Industries.
3. Under the Oracle Communications subheading, click the Oracle Communications
documentation link.
15
E76220 Revision 1, July 2016
IntroductionAlarms and Maintenance

The Communications Documentation page appears. Most products covered by these documentation
sets will appear under the headings “Network Session Delivery and Control Infrastructure”or
“Platforms.”
4. Click on your Product and then the Release Number.
A list of the entire documentation set for the selected product and release appears.
5. To download a file to your location, right-click the PDF link, select Save target as (or similar
command based on your browser), and save to a local folder.
16
E76220 Revision 1, July 2016
IntroductionAlarms and Maintenance

Chapter
2
Maintenance
This chapter provides maintenance information,
problem detection description, and general recovery
procedures for the E5-APP-B.
Topics:
•Introduction.....18
•Preventive Maintenance.....18
•System Health Check Overview.....24
•Running the System Health Check.....30
17
E76220 Revision 1, July 2016

Introduction
This chapter provides preventive and corrective maintenance information. Customers perform a small
number of daily preventive maintenance tasks. The ELAP application performs automatic monitoring
and problem reporting.
Detailed information about recovery procedures is contained in the remaining chapters of this manual.
Preventive Maintenance
This section describes the following recommended periodic maintenance:
•Daily maintenance procedures:
•Backing Up the RTDB
•Transferring RTDB Backup File
•Automatic RTDB Backup
Daily Maintenance Procedures
Use the Automatic RTDB Backup feature to backup all data stored in the RTDB. The manual backup
procedures are included in this section in case the database backup needs to be performed manually.
Storing database backups in a secure off-site location ensures the ability to recover from system failures.
This section describes the following recommended daily maintenance procedures:
•Backing Up the RTDB
•Transferring RTDB Backup File
Backing Up the RTDB
For ELAP 8.0 or later, a daily RTDB backup is created automatically. For automatic RTDB Backup, see
Automatic RTDB Backup.
1. Log in to the ELAPGUI on MPS A as the elapall user.
For information about how to log in to the ELAPGUI, see Accessing the ELAP GUI Interface.
Note: For ELAP 8.0 or later, the ELAP software can continue to operate while performing the
RTDB backup.
2. From the ELAP menu, select RTDB > Maintenance > Backup RTDB.
The window in Figure 1: Backup the RTDB is displayed.
18
E76220 Revision 1, July 2016
MaintenanceAlarms and Maintenance

Figure 1: Backup the RTDB
3. Click Backup RTDB.
The window in Figure 2: Backup the RTDB Confirmation displays a request for confirmation.
Figure 2: Backup the RTDB Confirmation
4. Click Confirm RTDB Backup.
If the backup starts successfully, the following message will scroll through the GUI banner:
Backup RTDB in progress.
After the backup completes successfully, the success window is displayed.
5. The RTDB backup procedure is complete.
6. Select Process Control > Start Software from the ELAP Menu.
7. On the Start ELAP Software screen as shown in Figure 3: Start ELAP Software, click Start ELAP
Software.
Figure 3: Start ELAP Software
After the ELAP software has started successfully, the screen in Figure 4: Start ELAP Software - Success
is displayed.
19
E76220 Revision 1, July 2016
MaintenanceAlarms and Maintenance

Figure 4: Start ELAP Software - Success
Transferring RTDB Backup File
Perform this procedure once each day. The estimated time required to complete this procedure depends
on network bandwidth. File sizes can be several gigabytes for the database.
1. Log in to the ELAP command line interface with user name elapdev and the password associated
with that name.
2. Use the Secure File Transfer Protocol (sftp) to transfer to a remote, secure location the RTDB
backup file created by the procedure Backing Up the RTDB.
Automatic RTDB Backup
Automatic RTDB Backup can be scheduled during off-peak provisioning hours, eliminated the need
for human intervention. Automatic ELAP RTDB backup intervals are scheduled at 6:a.m. every morning
in the Active Server.
User Interface
The menu item circled in Figure 5: Automatic RTDB Backup Menu Item is available on the ELAP GUI of
the Active ELAP server only:
20
E76220 Revision 1, July 2016
MaintenanceAlarms and Maintenance