HP ProLiant DL280 Manual

Memory Troubleshooting Best Practices for HP ProLiant
Servers
Accurately troubleshooting memory issues in ProLiant server configurations is an important process that can
help prevent unnecessary replacement of hardware components. In addition, accurate problem diagnosis
prevents customers from experiencing unnecessary downtime while waiting for hardware that may not need
to be replaced. Following standard troubleshooting guidelines and using them each time a memory issue is
suspected helps to establish this.
HP has developed several methods for troubleshooting memory problems in ProLiant servers.
The purpose of this white paper is to assist HP customers in troubleshooting memory problems by successfully
isolating the specific DIMMs causing the problem. This helps to prevent nonessential replacement of
unaffected DIMMs or, in some cases, entire banks of memory. In addition, effective troubleshooting can help
determine if a firmware or other software download can resolve a problem without replacing hardware.
This white paper covers the following topics:
•Why Should I Troubleshoot Every Memory Problem?
•How Can I tell if a Memory Problem has occurred?
•What tools are available from HP to help identify a failing DIMM?
•Troubleshooting Using HP Insight Diagnostics Offline Edition
•Troubleshooting Using HP Insight Diagnostics Online Edition
•Troubleshooting Flowchart for Bootable Systems
•Troubleshooting Flowchart for Non-Bootable Systems
•What role does Firmware play in Solving Memory Problems?
•Why buy HP memory?
•Other Troubleshooting Resources

2
Why Should I Troubleshoot Every Memory Problem?
Accurate diagnosis of system memory problems in ProLiant servers has many advantages, including:
•Prevents unnecessary hardware replacement.
•Prevents the return of parts that test NFF (No Fault Found).
•Prevents server downtime.
Best P ract ice
: Many product issues that result in hardware replacement are preventable or
correctable with a firmware update. HP recommends checking for a firmware update before sending
a part back to HP for replacement. Based on the HP ProLiant product return rates, a significant
percentage of all returned hardware products were functioning properly and only needed a firmware
update. Although not all products fall into this category, server downtime and time spent removing,
returning, and ultimately replacing hardware may have been avoided if an attempt had been made
to flash the firmware during the troubleshooting process.
How Can I Tell if a Memory Problem has Occurred?
There are many indicators that a problem has occurred within the memory subsystem. HP has several tools
used to identify the status of hardware and software within a system. Using these tools is a good first step in
the process. When a memory problem is suspected, check one or all of these common places to find
information:
•The HP System Management Homepage
•HP Systems Insight Manager (HP SIM)
•Server Logs
•DIMM Slot LEDs
IMPORTANT:
When a memory error is detected, the firmware illuminates the fault LEDs located near each DIMM
slot on the system board.
If the system identifies an error to a specific slot, that LED illuminates. However, if the system can only
identify an error within a bank, but cannot isolate a specific DIMM, all the LEDs in the bank will
illuminate.
In addition, if the system cannot identify the bank in which the error has occurred, all the LEDs in all
banks illuminate, making the task of isolating the failing DIMM more difficult, and the chance of
replacing functioning banks of memory more likely.

3
Therefore, further troubleshooting is necessary to determine which specific DIMM is failing. Use the
LEDs as a tool in identifying that a memory problem may exist, but don’t rely solely on the status of the
LEDs to determine if hardware should be replaced.
What Tools are Available from HP to Help Identify a Failing DIMM?
Refer to the following HP system tools when a memory problem is suspected.
HP System Management Homepage
The HP System Management Homepage supplies a consolidated view of system hardware health,
configuration, performance and status information for individual HP servers. Details are provided on total
system health, including system memory. Information on memory can be found under the “Performance”
section on the main page (See Figure 1).
Figure 1: Overview of the HP System Management Homepage

4
HP Systems Insight Manager (HP SIM)
HP Systems Insight Manager monitors the health of the hardware in the system and polls installed hardware
for its status every few minutes. Refer to the screenshot below for an example of events displayed on the
System page.
Figure 2: HP Systems Insight Manager

5
System Logs
Server system logs record the status of hardware events, including memory issues. For servers running
Microsoft Windows operating systems, either of the following logs can be a valuable resource:
•Integrated Management Log (IML)
•Event Viewer
For servers running Linux operating systems, refer to either of the following:
•Integrated Management Log (IML)
•varlog/messages file
Microsoft Windows Operating Systems: Using the IML
The IML Viewer is a software tool created by HP and is available as a downloadable component pack from
HP.COM. It can also be accessed via the HP System Management Homepage (SMH). Navigate to this tool
through SMH by clicking on the Logs tab or through the operating system from HP System Tools. Figure 3
below shows the IMH accessed via SMH. System memory issues, if present, will be recorded and will be
visible in the IML.

6
Figure 3: Integrated Management Log
Microsoft Windows Operating Systems: Using Event Viewer
The Event Viewer is a software tool available as part of Microsoft Windows operating systems. It can be
accessed by navigating to HP System Tools via the Start menu. Figure 4 shows an example of server events
that are logged in this tool.

7
Figure 4: Event Viewer (Microsoft Windows Operating Systems)
Linux Operating Systems: Using the IML
The IML Viewer is a software tool created by HP and is available as a downloadable component pack from
HP.COM. For systems running Linux, type the command hplog –v to view the IML log and check for system
memory errors. From the Linux Command Prompt, type “hplog –v” and the entire IML will be displayed. Any
detected system memory error will be logged, including any pre-failure warranty memory events on systems
where applicable.
Linux Operating Systems: Using /var/log/messages file
The Linux system log (/var/log/messages) can be viewed using the cat, more, and less commands. The
following types of messages may be logged here if a memory problem has occurred:
:
Oct 11 01:51:18 dhcp57-150 hpasmd[12039]: WARNING: hpasmd: Corrected Memory Error threshold
exceeded (Slot 3, Memory Module 6)
[root@dhcp57-150 ~]# hplog -v
ID Severity Initial Time Update Time Count
-------------------------------------------------------------
0011 Information 22:50 09/16/2004 22:50 09/16/2004 0001
LOG: IML Cleared (Administrator)
0012 Repaired 00:32 09/21/2004 00:32 09/21/2004 0001
LOG: Memory Cartridge Not Redundant (Slot 5)
0033 Caution 04:02 10/06/2006 04:02 10/06/2006 0001
LOG: Corrected Memory Error threshold exceeded (Slot 3, Memory Module 6)

8
DIMM Slot LEDs as Memory Problem Indicators
DIMM slot LEDs can be helpful indicators of issues with system memory. Figure 5 below gives an example of
the location of LEDs that will illuminate when certain DIMM slots indicate a failure. The example below is from
a ProLiant DL380 G4 system board; however, similar graphics for specific systems can be found in server
User Guides and other system documentation.
9/5/2006 12
Memory Problem Indicators
Amber = Memory
failed
Off = Normal
DIMM failure
slot 1A
10
Amber = Memory
failed
Off = Normal
DIMM failure
slot 2A
9
Amber = Memory
failed
Off = Normal
DIMM failure
slot 3B
8
Amber = Memory
failed
Off = Normal
DIMM failure
slot 4B
7
Amber = Memory
failed
Off = Normal
DIMM failure
slot 5C
6
Amber = Memory
failed
Off = Normal
DIMM failure
slot 6C
5
Example DL 380 G4
DIMM Slot LEDs on the System Board
Figure 5: DiMM Slot LEDs
In the example below, the DIMM Slot LED on a ProLiant DL380 G7 server is illuminated:

9
Once a memory problem is suspected based on one of the methods outlined above, the first step is to
schedule downtime with the customer for troubleshooting. HP recommends using HP Insight Diagnostics,
available on the SmartStart CD and as a standalone download on HP.COM to begin the troubleshooting
process.
HP Insight Diagnostics is a proactive server management tool that is available in offline and online editions. It
provides diagnostics and troubleshooting capabilities to help locate system and component problems.
Features of HP Insight Diagnostics:
•Works on multiple operating systems
•Can be used online or offline
•HP Insight Management Agents and the System Management driver can be leveraged
•You can leverage HP Insight Management Agents and the System Management driver
•Common Diagnostic Model (CDM) compliance.
For more information on HP Insight Diagnostics, refer to the following URL:
http://h18013.www1.hp.com/products/servers/management/hpid/index.html

10
Troubleshooting Using HP Insight Diagnostics Online Edition
HP Insight Diagnostics Online edition is available in HP ProLiant Support Packs and from HP.COM.
The Insight Diagnostics Online edition performs various non-intrusive in-depth system and component
diagnosis while the operating system is running. The “survey feature” can identify and resolve problems
without taking the system down.
Troubleshooting Using HP Insight Diagnostics Offline Edition
Insight Diagnostics Offline edition is available in HP SmartStart by choosing “Launch server diagnostics.”
Some benefits of HP Insight Diagnostics are:
•Performs extensive in-depth system and component testing while in a controlled operating
environment.
•Survey feature enables IT administrators to track hardware and software changes in order to form a
complete and thorough auditing process for the system.
•Test results can be analyzed by IT administrators to diagnose system and component problems in
order to repair and return servers back to the production environment.
Online vs. Offline testing
Offline is the preferred testing method because it is the most accurate. When testing offline, there is minimal
interference from the operating system and address space testing is maximized because a very small Linux
kernel is used.
Helpful Link s:
HP Insight Diagnostics User Guide: www.hp.com/servers/diags
Troubleshooting Flowcharts
The following troubleshooting flowcharts can be used as another tool for diagnosing specific DIMMs that
have failed. They can be used as a general guideline and should be used in conjunction with other tools and
methods outlined in this white paper. Because a lot of system issues can be solved by upgrading firmware,
download the latest version of the server firmware before proceeding with the troubleshooting methods
outlined in the following flowcharts. The latest firmware is available as follows: