R/Evolution 2000 Series User manual

2000 Series
Troubleshooting Guide
P/N 83-00004287-12
Revision A
May 2008

Copyright Protected Material 2002-2008. All rights reserved. R/Evolution and the R/Evolution logo are trademarks of Dot
Hill Systems Corp. All other trademarks and registered trademarks are proprietary to their respective owners.
The material in this document is for information only and is subject to change without notice. While reasonable efforts have
been made in the preparation of this document to assure its accuracy, changes in the product design can be made without
reservation and without notification to its users.

3
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
1. System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Architecture Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Enclosure Chassis and Midplane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Midplane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Enclosure ID Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Drive Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Disk Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Controller Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Drive Expansion Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Power-and-Cooling Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Power Supply Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Cooling Fans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Airflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2. Fault Isolation Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Gather Fault Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Determine Where the Fault Is Occurring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Review the Event Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Isolate the Fault . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4R/Evolution 2000 Series Troubleshooting Guide • May 2008
3. Troubleshooting Using System LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
LED Names and Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Using LEDs to Check System Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Using Enclosure Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Using Drive Module LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Using Controller Module Host Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Using the Controller Module Expansion Port LED . . . . . . . . . . . . . . . . . . . . . . 30
Using Ethernet Management Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Using Controller Module Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Using Power-and-Cooling Module LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Using Expansion Module LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4. Troubleshooting Using RAIDar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Problems Using RAIDar to Access a Storage System . . . . . . . . . . . . . . . . . . . . . . . 36
Determining Storage System Status and Verifying Faults . . . . . . . . . . . . . . . . . . . . 37
Stopping I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Clearing Metadata From Leftover Disk Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Isolating Faulty Disk Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Identifying a Faulty Disk Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Reviewing Disk Drive Error Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Reviewing the Event Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Reconstructing a Virtual Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Isolating Data Path Faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Isolating Internal Data Path Faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Isolating External Data Path Faults on an FC Storage System . . . . . . . . . . . . . 51
Isolating External Data Path Faults on an iSCSI Storage System . . . . . . . . . . . 52
Isolating External Data Path Faults on a SAS Storage System . . . . . . . . . . . . . 53
Resetting a Host Channel on an FC Storage System . . . . . . . . . . . . . . . . . . . . . 54
Changing PHY Fault Isolation Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Contents 5
Resetting Expander Error Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Disabling or Enabling a PHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Disabling or Enabling PHY Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Using Recovery Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Removing a Virtual Disk From Quarantine . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Trusting a Virtual Disk for Disaster Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Problems Scheduling Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Affect of Changing the Date and Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60
Deleting Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60
Errors Associated with Scheduling Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Selecting Individual Events for Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Selecting or Clearing All Events for Notification . . . . . . . . . . . . . . . . . . . . . . . . . . .62
Correcting Enclosure IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Problems After Power-On or Restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63
5. Troubleshooting Using Event Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Event Severities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Viewing the Event Log in RAIDar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Viewing an Event Log Saved From RAIDar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Reviewing Event Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Saving Log Information to a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70
Configuring the Debug Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71
6. Voltage and Temperature Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Resolving Voltage and Temperature Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . .73
Sensor Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Power Supply Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

6R/Evolution 2000 Series Troubleshooting Guide • May 2008
Cooling Fan Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Temperature Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Power-and-Cooling Module Voltage Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . 77
7. Troubleshooting and Replacing FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Static Electricity Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Identifying Controller or Expansion Module Faults . . . . . . . . . . . . . . . . . . . . . . . . . 80
Removing and Replacing a Controller or Expansion Module . . . . . . . . . . . . . . . . . 82
Saving Configuration Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Shutting Down a Controller Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Removing a Controller Module or Expansion Module . . . . . . . . . . . . . . . . . . . 85
Replacing a Controller Module or Expansion Module . . . . . . . . . . . . . . . . . . . 87
Moving a Set of Expansion Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Updating Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Updating Firmware During Controller Replacement . . . . . . . . . . . . . . . . . . . . 90
Updating Firmware Using RAIDar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Identifying SFP Module Faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Removing and Replacing an SFP Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Removing an SFP Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Installing an SFP Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Identifying Cable Faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Identifying Cable Faults on the Host Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Identifying Cable Faults on the Drive Enclosure Side . . . . . . . . . . . . . . . . . . . 95
Disconnecting and Reconnecting SAS Cables . . . . . . . . . . . . . . . . . . . . . . . . . 95
Identifying Drive Module Faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Understanding Disk-Related Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Disk Drive Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Disk Channel Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Identifying Faulty Drive Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Contents 7
Updating Disk Drive Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Removing and Replacing a Drive Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Replacing a Drive Module When the Virtual Disk Is Rebuilding . . . . . . . . . . 104
Identifying the Location of a Faulty Drive Module . . . . . . . . . . . . . . . . . . . . . 105
Removing a Drive Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Installing a Drive Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Verify That the Correct Power-On Sequence Was Performed . . . . . . . . . . . . . 109
Installing an Air Management Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Identifying Virtual Disk Faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Clearing Metadata From a Disk Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Identifying Power-and-Cooling Module Faults . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Removing and Replacing a Power-and-Cooling Module . . . . . . . . . . . . . . . . . . . . 114
Removing a Power-and-Cooling Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Installing a Power-and-Cooling Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Replacing an Enclosure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
A. Troubleshooting Using the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Viewing Command Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
clear cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
clear expander-status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
ping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119
rescan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
reset host-channel-link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
restore defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
set debug-log-parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
set expander-fault-isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
set expander-phy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
set led . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

8R/Evolution 2000 Series Troubleshooting Guide • May 2008
set protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
show debug-log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
show debug-log-parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
show enclosure-status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
show events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
show expander-status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
show frus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
show protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
show redundancy-mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
trust . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Problems Scheduling Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Create the Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Schedule the Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Errors Associated with Scheduling Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Missing Parameter Data Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

9
Preface
This guide describes how to diagnose and troubleshoot a R/Evolution™ storage
system, and how to identify, remove, and replace field-replaceable units (FRUs). It
also describes critical, warning, and informational events that can occur during
system operation. This guide applies to the following enclosures:
■2730 FC Controller Enclosure
■2530 SAS Controller Enclosure
■2330 iSCSI Controller Enclosure
■SAS Expansion Enclosure
This book is written for system administrators and service personnel who are
familiar with Fibre Channel (FC), Internet SCSI (iSCSI), and Serial Attached SCSI
(SAS) configurations, network administration, and RAID technology.
Before You Read This Book
Before you begin to follow procedures in this book, you must have already installed
enclosures and learned of any late-breaking information related to system operation,
as described in the getting started guide and release notes.

10 R/Evolution 2000 Series Troubleshooting Guide • May 2008
Typographic Conventions
Related Documentation
Typeface1
1 The fonts used in your viewer might differ.
Meaning Examples
AaBbCc123 Book title, new term, or
emphasized word See the release notes.
A virtual disk (vdisk) can ....
You must ....
AaBbCc123 Directory or file name,
value, command, or
on-screen output
The default file name is store.logs.
The default user name is manage
Type exit.
AaBbCc123 Text you type, contrasted
with on-screen output # set password Enter new password:
AaBbCc123 Variable text you replace
with an actual value Use the format user@domain
Application Title Part Number
Site planning information R/Evolution Storage System Site Planning Guide 83-00004283
Late-breaking information not
included in the documentation set R/Evolution 2730 Release Notes
R/Evolution 2530 Release Notes
R/Evolution 2330 Release Notes
83-00004282
83-00004396
83-00005032
Installing and configuring hardware R/Evolution 2730 Getting Started Guide
R/Evolution 2530 Getting Started Guide
R/Evolution 2330 Getting Started Guide
83-00004284
83-00004398
83-00005034
Configuring and managing storage R/Evolution 2000 Series Reference Guide 83-00004289
Using the command-line interface
(CLI) R/Evolution 2000 Series CLI Reference Guide 83-00004288
Recommendations for maximizing
reliability, accessibility, and
serviceability
R/Evolution 2000 Series Best Practices Guide
(FC and iSCSI only) 83-00004286

11
CHAPTER 1
System Architecture
This chapter describes the R/Evolution™ storage system architecture. Prior to
troubleshooting any system, it is important to understand the architecture, including
each of the system components, how they relate to each other, and how data passes
through the system. Topics covered in this chapter include:
■“Architecture Overview” on page 11
■“Enclosure Chassis and Midplane” on page 12
■“Drive Modules” on page 14
■“Controller Modules” on page 15
■“Drive Expansion Module” on page 15
■“Power-and-Cooling Modules” on page 15
Architecture Overview
The following figure shows how field-replaceable units (FRUs) connect within a
storage system enclosure:
Figure 1-1 R/Evolution Storage System Architecture Overview
Drive module
Power-and-cooling module
Midplane
I/O module

12 R/Evolution 2000 Series Troubleshooting Guide • May 2008
FRUs include:
■Chassis-and-midplane. An enclosure’s 2U metal chassis and its midplane circuit
board comprise a single FRU. All other FRUs connect and interact through the
midplane.
■Drive module. An enclosure can contain 12 SATA or SAS drive modules.
■I/O module. A controller enclosure can contain one or two controller modules; a
drive enclosure can contain one or two expansion modules. Each type of I/O
module controls I/O between attached hosts and storage system disk drives.
■Power-and-cooling modules.
The following sections describe each FRU in more detail.
Note – Do not remove a FRU until the replacement is on-hand. Removing a FRU
without a replacement will disrupt the system airflow and cause an over-temperature
condition.
Enclosure Chassis and Midplane
An enclosure’s metal chassis is 2U in height. The front of the enclosure has two
rackmount flanges, called ears. The left ear has the enclosure ID display. The right
ear has enclosure status LEDs. The chassis also includes the midplane circuit board.
If the chassis or midplane is damaged they are replaced as a unit.
Midplane
The midplane circuit board is the common connection point for all system
electronics; all other FRUs plug into this board. Drive modules plug into the front of
the midplane. Power-and-cooling modules and I/O modules (controller modules or
drive modules) plug into the back of the midplane.

Chapter 1 System Architecture 13
Enclosure ID Display
The enclosure ID (EID) display provides a visual single-digit identifier for each
enclosure in a storage system. The EID display is located on the left ear, as viewed
from the front of the chassis.
For a storage system that includes a controller module, EID values are set by the
RAID controller. For drive enclosures that are attached to a host for use as JBODs
(just a bunch of disks), EID values are set by the host.
■When drive enclosures are attached to a controller enclosure
■The controller enclosure’s EID is zero.
■A drive enclosure’s EID is nonzero. The EID is 1 for the first drive enclosure,
and the EID is incremented for each subsequent enclosure.
■EIDs are persistent, so will not change during simple reconfigurations.
■EIDs can be used to correlate physical enclosures with logical views of the
storage system provided by system interfaces such as RAIDar.
■When drive enclosures are attached to a host
■A drive enclosure’s EID can be zero or nonzero.
■Each drive enclosure in a storage system must have a unique EID.
■EIDs are persistent, so will not change during simple reconfigurations.
■EIDs can be used to correlate physical enclosures with logical views of the
storage system provided by system interfaces.
When installing a system with drive enclosures attached, the enclosure IDs might
not agree with the physical cabling order. This is because the controller might have
been previously attached to some of the same enclosures and it attempts to preserve
the previous enclosure IDs, if possible. To correct this, make sure that both
controllers are up and perform a rescan using RAIDar (see “Correcting Enclosure
IDs” on page 63) or the CLI (see “rescan” on page 119). This will reorder the
enclosures, but can take up to two minutes for the IDs to be corrected.
EIDs are managed by SES functions of the Expander Controller in each controller
module and expansion module.
For information about how EIDs are affected when expansion modules are moved,
see “Moving a Set of Expansion Modules” on page 89.

14 R/Evolution 2000 Series Troubleshooting Guide • May 2008
Drive Modules
The drive module has a front bezel with a latch that is used to insert or remove the
drive module. When any component of a drive module fails, the entire module is
replaced. Each drive module is inserted into a drive slot (or bay) in an enclosure.
The following figure shows the numbering of drive slots in an enclosure.
Figure 1-2 Drive Slot Numbers
A drive is identified by the numbers of the enclosure and slot that the drive is in.
For example, the last drive in the controller enclosure is identified as 0.11 (EID 0,
slot 11). Drive modules are slot-independent, that is, the drives can be moved to any
slot with the power off. Once power is applied, the RAID controllers use the
metadata held on each disk to locate each member of a virtual disk.
Disk Drives
Each RAID controller has single-port access from the local SAS expander to
internal and drive enclosure drives. Alternate path, dual-port access to all internal
drives is accomplished through the expander inter-controller wide lane connection.
Dual-port access assumes the presence of both controller modules. In a failed over
configuration, where the partner controller module is down or removed, only single-
port access to the drives exists.
The storage system can include either or both SAS or SATA II drives. A drive can
be interchanged with a qualified equivalent drive. In addition, each enclosure can be
populated with disks of various capacities. To ensure the full use of a disk’s
capacity, construct all virtual disks with disks of the same capacity.

Chapter 1 System Architecture 15
Controller Modules
A controller module is a FRU that contains two connected circuit boards: a RAID
I/O module and a host interface module (HIM).
The RAID I/O module is a hot-pluggable board that mates with the enclosure
midplane and provides all RAID controller functions and SAS/SATA disk channels.
The HIM provides the host-side interface and contains dual-port, host target
channels for connection to host systems. The 2730 has a Fibre Channel HIM that
supports 2- or 4-Gbit/sec link speed. The 2330 has an iSCSI HIM that supports
1-Gbit/sec link speed. The 2530 has a SAS HIM that supports 4-lane 3-Gbps host
speeds.
The controller module contains three processing subsystems: the Storage Controller,
the Management Controller, and the Expander Controller.
Note – When a fault occurs in a controller module processor or a bus fault occurs
that is related to the controller module, the entire controller module FRU is
replaced.
Drive Expansion Module
Expansion module architecture is a simplified version of controller module
architecture. Like a controller module, an expansion module has an Expander
Controller and uses the SAS protocol. Each module has a SAS “In” port and a SAS
“Out” port, which enables up to four 2130s to be connected together, and to a host
system. When a fault occurs in the Expander Controller or a bus fault occurs that is
related to the expansion module, the entire module is replaced.
For information about supported configurations for connecting enclosures to each
other and to hosts, see the appropriate getting started guide.
Power-and-Cooling Modules
Each enclosure contains two power-and-cooling modules. A power-and-cooling
module is a FRU that includes a power supply unit and two cooling fans. If a power
supply fault or fan fault occurs, the entire module is replaced.

16 R/Evolution 2000 Series Troubleshooting Guide • May 2008
Power Supply Unit
Each 750-Watt, AC power supply unit (PSU) is auto-sensing and runs in a load-
balanced configuration to ensure that the load is distributed evenly across both
power supplies.
Cooling Fans
The cooling fans are integrated into each of the power-and-cooling module FRUs.
Each module contains two fans mounted in tandem (series). The fans are powered
from the +12V common rail so that a single failed power supply still enables all
fans to continue to operate.
The fans cannot be accidentally removed as they are part of the power-and-cooling
module. Removing this module requires the disengagement of a captive panel
fastener and the operation of an ejector lever to remove it from the chassis.
Should one fan fail in either module, the system continues to operate indefinitely. In
addition, the fan system enables the airflow pattern to remain unchanged and there
is no pressure leak through the failed fan since there are always two fans in tandem,
and they are sealed to each other through a calibrated cavity. Should a power-and-
cooling module be turned off or unplugged, the fans inside the module continue to
operate at normal capacity. This is accomplished by powering each fan from a
power bus on the midplane.
The fans’ variable speed is controlled by the controller modules through an I2C
interface. The fans also provide tachometer speed information through the I2C
interface. Speed control is accomplished through the use of speed commands issued
from the controller module. The controller module has one temperature sensor at the
inlet port of the controller to sense the exhaust air temperature from the disk drives.
Should the controller module sense a rise in temperature, it can increase fan speed
to keep the disk drive temperatures within limits.
Balanced cooling for all of the drives is accomplished through the use of two
mechanisms.
■Tuned port apertures in the midplane placed behind each drive carrier slot
■The use of a cavity behind the entire surface of the midplane (side-to-side and
top-to-bottom) that acts as an air pressure equalization chamber. This chamber is
commonly evacuated by all of the fans.
In this way the amount of mass flow through each drive slot is controlled to be the
same slot to slot.

Chapter 1 System Architecture 17
Airflow is controlled and optimized over the power supply by using the power
supply chassis as the air-duct for the power supply, ensuring that there are no dead
air spaces in the power supply core and increasing the velocity flow (LFM) by
controlling the cross sectional area that the mass flow travels through.
Airflow is controlled and optimized over the RAID I/O board and HIM in a similar
manner. The controller cover is used as an air duct to force air over the entire
surface of the controller from front to back, ensuring no dead air spaces, and
increasing the velocity flow (LFM) by controlling the cross-sectional area that the
mass flow travels through.
Cooling for all hot components is passive. There are no other fans in the system
other than the fans contained in the power-and-cooling module.
Airflow
Caution – To allow for correct airflow and cooling, use an air management module
for removed FRUs. Do not leave a FRU out of its slot for more than two minutes.
As noted above, an enclosures cooling system includes four fans in a tandem
parallel array. These variable speed fans provide low noise and high mass flow
rates. Airflow is from front to back. Each drive slot draws ambient air in at the front
of the drive, sending air over the drive surfaces and then through tuned apertures in
the chassis midplane.
Note that the airflow washes over the top and bottom surface of the disk drive at
high mass flow and velocity flow rates, so both sides of the drive are used for
cooling. The airflow system uses a cavity in the chassis behind the midplane as an
air-pressure equalization chamber to normalize the negative pressure behind each of
the disk drive slots. This mechanism together with the tuned apertures in the
midplane behind each drive assures an even distribution of airflow and therefore
LFM for each drive slot. This even cooling extends the operational envelope of the
system by ensuring no “hot” drive bypass.
Further, airflow is “in line” with the top and bottom surfaces of the drive to reduce
back-pressure and optimize fan performance. All of the mass flow at room ambient
is used for cooling the 12 disk drives. The high velocity flow helps to lower the
thermal resistance of the disk drive assembly to ambient temperature. The thermal
temperature rise of the disk drive is dependent upon the power consumed by the
disk drive, which varies by drive model as well as the level of drive activity.

18 R/Evolution 2000 Series Troubleshooting Guide • May 2008

19
CHAPTER 2
Fault Isolation Methodology
The R/Evolution storage system provides many ways to isolate faults within the
system. This chapter presents the basic methodology used to locate faults and the
associated FRUs.
The basic fault isolation steps are:
■Gather fault information
■Determine where in the system the fault is occurring
■Review event logs
■If required, isolate the fault to a data path component
Gather Fault Information
When a fault occurs, it is important to gather as much information as possible.
Doing so will help you determine the correct action needed to remedy the fault.
Begin by reviewing the reported fault. Is the fault related to an internal data path or
an external data path? Is the fault related to a hardware component such as a drive
module, controller module, or power-and-cooling module? By isolating the fault to
one of the components within the storage system, you will be able to determine the
necessary action more rapidly.
Determine Where the Fault Is Occurring
Once you have an understanding of the reported fault, review the enclosure LEDs.
The enclosure LEDs are designed to alert users of any system faults and might be
what alerted the user to a fault in the first place.
When a fault occurs, the status LEDs on an enclosure’s right ear (see Figure 3-1)
illuminate. Check the LEDs on the back of the enclosure to narrow the fault to a
FRU, connection, or both. The LEDs also help you identify the location of a FRU
reporting a fault.

20 R/Evolution 2000 Series Troubleshooting Guide • May 2008
Use RAIDar to verify any faults found while viewing the LEDs. RAIDar is also a
good tool to use in determining where the fault is occurring if the LEDs cannot be
viewed due to the location of the system. RAIDar provides you with a visual
representation of the system and where the fault is occurring. It can also provide
more detailed information about FRUs, data, and faults. For more information about
LEDs, see “Troubleshooting Using System LEDs” on page 21.
Review the Event Logs
The event logs record all system events. It is very important to review the logs, not
only to identify the fault, but also to search for events that might have caused the
fault to occur. For example, a host could lose connectivity to a virtual disk if a user
changes channel settings without taking the storage resources assigned to it into
consideration. In addition, the type of fault can help you isolate the problem to
hardware or software. For more information about event logs, see “Troubleshooting
Using Event Logs” on page 65.
Isolate the Fault
Occasionally it might become necessary to isolate a fault. This is particularly true
with data paths due to the number of components the data path consists of. For
example, if a host-side data error occurs, it could be caused by any of the
components in the data path: controller module, SFP, cable, switch, or data host. For
more information about isolating faults, see “Troubleshooting Using System LEDs”
on page 21.