IBM Storwize V7000 User manual

IBM Storwize V7000 Unified
Problem Determination Guide
GA32-1057-07


Note
Before using this information and the product it supports, read the general information in “Notices” on page 309, the
information in the “Safety and environmental notices” on page xi, as well as the information in the IBM Environmental
Notices and User Guide , which is provided on a DVD.
This edition applies to IBM Storwize V7000 Unified and to all subsequent releases and modifications until otherwise
indicated in new editions.
This edition replaces GA32-1057-06.
© Copyright IBM Corporation .
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.

Contents
Figures ..............vii
Tables ...............ix
Safety and environmental notices . . . xi
Safety ................xi
Safety statements ............xiii
Sound pressure ............xvii
About this guide ..........xix
Who should use this guide .........xix
Accessibility ..............xix
Emphasis ...............xix
Storwize V7000 Unified library and related
publications ..............xx
How to order IBM publications .......xxii
Sending your comments .........xxiii
Chapter 1. Storwize V7000 Unified
hardware components ........1
Chapter 2. Best practices for
troubleshooting ...........3
Record access information ..........3
Follow power management procedures .....4
Set up event notifications ..........4
Back up your data ............5
Manage your spare and failed drives ......5
Resolve alerts in a timely manner .......5
Keep your software up to date ........6
Keep your records up to date.........6
Subscribe to support notifications .......6
Know your IBM warranty and maintenance
agreement details .............6
Chapter 3. Getting started
troubleshooting ...........9
Installation troubleshooting .........10
Problems with initial setup ........10
Installation error codes .........13
Problems reported by the CLI commands during
software configuration..........20
Management GUI wizard failure ......21
GUI access issues ............22
Health status and recovery .........23
Connectivity issues ............25
Host to file modules connectivity ......25
Ethernet connectivity between file modules. . . 26
Ethernet connectivity from file modules to the
control enclosure ...........29
Fibre Channel connectivity between file modules
and control enclosure ..........34
Understanding LED hardware indicators ....38
File node hardware indicators .......38
Enclosure hardware indicators .......46
Management GUI interface .........51
When to use the management GUI .....52
Accessing the Storwize V7000 Unified
management GUI ...........53
Using fix procedures ..........53
Chapter 4. File module ........55
General file module procedures........55
Rebooting a file module .........55
Removing a file module to perform a
maintenance action ...........55
Removing and replacing file module components 58
Resolving hard disk drive problems .....61
Monitoring memory usage on a file module . . 80
Errors and messages ...........81
Understanding error codes ........81
Understanding event IDs .........84
File module hardware problems .......85
Removing and replacing parts .......85
How to reset/reboot server iMM interface. . . 164
File module software problems .......165
Logical devices and physical port locations . . 165
Management node role failover procedures . . 165
Checking CTDB health ..........169
Checking the GPFS file system mount on each file
module ...............171
Identifying created and mounted file system
mounts ..............171
Resolving problems with missing mounted file
systems ..............172
Resolving stale NFS file systems ......173
Checking user and server authentication issues . . 173
Resolving the “Missing SRV record in DNS”
error...............173
If “netgroup” functionality with NIS or LDAP is
not working .............174
Possible client misconfiguration ......174
Trouble accessing exports when server and
client configurations are correct ......175
Checking client access ..........175
Checking network interface availability.....176
Recovering a GPFS file system .......176
Resolving an ANS1267E error ........178
Resolving issues reported by lshealth .....178
Error for “MGMTNODE_REPL_STATE ERROR
DATABASE_REPLICATION_FAILED” ....178
Resolving network errors .........179
Resolving full condition for GPFS file system. . . 180
Analyzing GPFS logs ...........181
Synchronizing time on the file modules ....181
Chapter 5. Control enclosure.....183
Storwize V7000 system interfaces.......183
© Copyright IBM Corp. iii

Service assistant interface ........183
Storage system command-line interface....185
Service command-line interface ......186
USB flash drive and Initialization tool interface 186
Event reporting.............193
Understanding events .........193
Event notifications...........195
Power-on self-test ...........195
Understanding events .........196
Understanding the error codes ......196
Viewing logs and traces .........196
Understanding the Storwize V7000 Unified battery
operation for the control enclosure ......196
Maintenance discharge cycles .......198
Understanding the medium errors and bad blocks 199
Resolving a problem ...........200
Start here: Use the management GUI
recommended actions .........201
Problem: Another system may be using the
system IP address ...........202
Problem: Unable to change the system IP
address because you cannot access the CLI . . 203
Problem: Management IP address unknown . . 204
Problem: Unable to connect to the management
GUI...............204
Problem: Unable to log on to the management
GUI...............205
Problem: Cannot create a clustered storage
system ..............205
Problem: Node canister service IP address
unknown ..............206
Problem: Cannot connect to the service assistant 207
Problem: Management GUI or service assistant
does not display correctly ........208
Problem: A node canister has a location node
error...............208
Problem: SAS cabling not valid ......208
Problem: New expansion enclosure not detected 209
Problem: Mirrored volume copies no longer
identical ..............209
Problem: Command file not processed from USB
flash drive .............210
Procedure: FCoE host-link ........210
Procedure: Resetting superuser password . . . 211
Procedure: Identifying which enclosure or
canister to service ...........211
Procedure: Checking the status of your system 212
Procedure: Getting node canister and system
information using the service assistant ....212
Procedure: Getting node canister and system
information using a USB flash drive .....213
Procedure: Understanding the system status
using the LEDs ............213
Procedure: Finding the status of the Ethernet
connections .............218
Procedure: Removing system data from a node
canister ..............219
Procedure: Deleting a system completely . . . 219
Procedure: Fixing node errors .......220
Procedure: Changing the service IP address of a
node canister ............220
Procedure: Accessing a canister using a directly
attached Ethernet cable .........221
Procedure: Reseating a node canister ....222
Procedure: Powering off your system ....222
Procedure: Collecting information for support 223
Procedure: Rescuing node canister software
from another node (node rescue) ......223
Removing and replacing parts........224
Preparing to remove and replace parts ....224
Replacing a node canister ........224
Replacing an expansion canister ......226
Replacing an SFP transceiver .......228
Replacing a power supply unit for a control
enclosure ..............230
Replacing a power supply unit for an expansion
enclosure ..............233
Replacing a battery in a power supply unit . . 237
Releasing the cable retention bracket ....241
Replacing a 3.5" drive assembly or blank carrier 241
Replacing a 2.5" drive assembly or blank carrier 243
Replacing enclosure end caps .......244
Replacing a SAS cable .........245
Replacing a control enclosure chassis ....246
Replacing an expansion enclosure chassis . . . 251
Replacing the support rails ........254
General storage system procedures ......255
SAN problem determination .......256
Fibre Channel link failures ........256
Ethernet iSCSI host-link problems .....257
Recover system procedure .........257
When to run the recover system procedure . . 258
Fix hardware errors ..........260
Removing system information for node canisters
with error code 550 or error code 578 using the
service assistant ...........260
Performing system recovery using the service
assistant ..............261
Recovering from offline VDisks using the CLI 263
What to check after running the system
recovery ..............263
Backing up and restoring the system configuration 264
Backing up the system configuration using the
CLI................266
Deleting backup configuration files using the
CLI................268
Chapter 6. Call home and remote
support ..............269
Configuring the remote support system ....269
Establishing an AOS connection ......269
Chapter 7. Recovery procedures . . . 273
User ID and system access .........273
Accessing a file module as root ......273
Recovering from losing the root password. . . 273
Resetting the NAS ssh key for configuration
communications ...........275
Working with NFS clients that fail to mount
NFS shares after a client IP change .....275
iv Storwize V7000 Unified: Problem Determination Guide Version

Working with file modules that report a stale
NFS file handle............276
File module-related issues .........277
Restoring System x firmware (BIOS) settings 277
Recovering from file systems that are offline
after the volumes came back online .....278
Recovering from a multipath event .....278
Recovering from an NFSD service error . . . 279
Recovering from an SCM error ......280
Recovering from an httpd service error ....280
Recovering from an sshd_data service error . . 281
Recovering from an sshd_int service error . . . 281
Recovering from an sshd_mgmt service error 281
Recovering from an sshd_service service error 281
Control enclosure-related issues .......282
Recovering when file volumes come back online 282
Recovering when a file volume does not come
back online .............282
Recovering from offline compressed volumes 283
Recovering from a 1001 error code .....284
Restoring data .............286
Restoring asynchronous data .......286
Restoring Tivoli Storage Manager data ....287
Upgrade recovery ............288
Chapter 8. Troubleshooting
compressed file systems ......297
Recovery scenario: Allocated capacity exceeds
capacity threshold ............297
Recovery scenario: Overestimated compression
ratio ................299
Recovery scenario: File system offline .....301
Monitoring file system compression ......303
Appendix. Accessibility features for
IBM Storwize V7000 Unified ......307
Notices ..............309
Trademarks ..............311
Electronic emission notices .........311
Federal Communications Commission (FCC)
statement ..............311
Industry Canada compliance statement ....312
Avis de conformité à la réglementation
d'Industrie Canada ..........312
Australia and New Zealand Class A Statement 312
European Union Electromagnetic Compatibility
Directive ..............312
Germany Electromagnetic compatibility
directive ..............313
Japan VCCI Council Class A statement ....314
People's Republic of China Class A Electronic
Emission Statement ..........314
International Electrotechnical Commission (IEC)
statement..............314
Korean Communications Commission (KCC)
Class A Statement ...........314
Russia Electromagnetic Interference (EMI) Class
A Statement .............314
Taiwan Class A compliance statement ....315
European Contact Information........315
Taiwan Contact Information ........315
Index ...............317
Contents v

vi Storwize V7000 Unified: Problem Determination Guide Version

Figures
1. Rear view of file module ........26
2. File module Ethernet direct connections. 27
3. Diagram shows how to connect the file
modules to the control enclosure using Fibre
Channel cables. (A) is file module 1 and (B) is
file module 2. (C) is the control enclosure. . . 35
4. LEDs on the power supply units of the control
enclosure .............47
5. LEDs on the node canisters .......49
6. Selecting a file module to display node status 63
7. Displaying node status.........64
8. Example that shows that mirroring is
re-synchronizing ...........66
9. Example that shows that a drive is not
synchronized ............67
10. Example that shows that the mirror is not
created ..............68
11. Example of a SMART error .......69
12. Removing the Ethernet adapter .....113
13. Location of the rubber stopper on the chassis 114
14. Location of the Ethernet adapter filler panel
on the chassis ...........115
15. Location of the port openings on the chassis 115
16. Aligning the Ethernet adapter port connectors
with the port openings ........116
17. Side view of adapter in the server.....116
18. Port connector alignment .......116
19. 16-drive-capable server model ......118
20. Tape-enabled server model .......119
21. SAS riser-card and controller assembly on the
16–drive-capable server model ......120
22. Controller retention brackets on
16-drive-capable server model ......120
23. SAS controller front retention brackets 121
24. Removing the rear controller retention bracket 121
25. Installing the controller retention bracket 122
26. Sliding the controller retention bracket
inward and pressing the release tab ....122
27. SAS riser-card assembly on tape-enabled
server model............123
28. ServeRAID M1000 advanced feature key and
M1015 adapter ...........129
29. ServeRAID M5000 advanced feature key and
M5014 adapter ...........130
30. ServeRAID M1000 advanced feature key and
M1015 adapter ...........131
31. ServeRAID M5000 advanced feature key and
M5014 adapter ...........132
32. Releasing the battery retention clip ....133
33. Disconnecting the remote battery cable 133
34. Removing the battery from the battery carrier 134
35. Connecting the remote battery cable ....135
36. Locating the posts on the battery and battery
carrier ..............135
37. DIMM locations for the Storwize V7000
Unified System x3650 M2 server .....139
38. DIMM locations for the Storwize V7000
Unified System x3650 M3 server .....140
39. Heat-sink release lever ........149
40. Microprocessor release latch ......150
41. Aligning the microprocessor ......152
42. Bottom surface of the heat sink .....154
43. Aligning the heat sink above the
microprocessor ...........154
44. 240 VA safety cover .........162
45. 240 VA safety cover .........163
46. Management GUI showing CTDB status for
both file modules ..........170
47. LEDs on the power supply units of the
control enclosure ..........214
48. LEDs on the node canisters .......216
49. Rear of node canisters that shows the
handles. .............225
50. Removing the canister from the enclosure 226
51. Rear of expansion canisters that shows the
handles. .............227
52. Removing the canister from the enclosure 228
53. SFP transceiver ...........229
54. Directions for lifting the handle on the power
supply unit ............232
55. Using the handle to remove a power supply
unit ...............232
56. Directions for lifting the handle on the power
supply unit ............236
57. Using the handle to remove a power supply
unit ...............236
58. Removing the battery from the control
enclosure power-supply unit ......240
59. Unlocking the 3.5" drive ........242
60. Removing the 3.5" drive ........242
61. Unlocking the 2.5" drive ........243
62. Removing the 2.5" drive ........244
63. SAS cable .............246
64. Removing a rail assembly from a rack cabinet 255
© Copyright IBM Corp. vii

viii Storwize V7000 Unified: Problem Determination Guide Version

Tables
1. Storwize V7000 Unified library ......xx
2. Other IBM publications ........xxii
3. IBM documentation and related websites xxii
4. Access information for your system .....3
5. Installation error code actions ......13
6. Error messages and actions .......14
7. CLI command problems ........21
8. Ethernet connections available with the file
modules .............26
9. Ethernet ports and type of connections. 27
10. How to connect Fibre Channel cables from file
modules to the control enclosure. Refer to the
previous graphic. ..........35
11. Error code port location mapping .....36
12. Fibre Channel cabling from the file module to
the control enclosure. .........37
13. LED states and associated actions. For the
Fibre Channel adapters on the file module
check the amber LED lights next to the port. . 37
14. Fibre Channel connection on the node canister
LED state and associated actions .....37
15. LED indicators, corresponding problem causes,
and corrective actions .........41
16. Power-supply LEDs ..........45
17. Power-supply unit LEDs ........47
18. Power LEDs ............49
19. System status and fault LEDs ......49
20. Control enclosure battery LEDs ......50
21. Status of volume ...........64
22. State of drives............65
23. SMART ASC/ASCQ error codes and messages 70
24. Error code information.........81
25. Originating role information .......82
26. Originating file module and file module
specific hardware code – Code 0, 2, 4 ....82
27. Originating file module specific software code
– Code 1, 3, 5 ............83
28. Storage enclosure hardware code – Code 6 83
29. Error code break down.........84
30. Components identified as customer replaceable
units (CRUs) and field replaceable units
(FRUs) ..............86
31. System board fan locations .......143
32. System status with 460-watt power supplies
installed .............145
33. Storwize V7000 Unified logical devices and
physical port locations ........165
34. Hostname and service IP reference ....166
35. Description of data fields for the event log 194
36. Notification levels ..........195
37. Bad block errors ..........200
38. Power-supply unit LEDs ........215
39. Power LEDs ............216
40. System status and fault LEDs ......216
41. Control enclosure battery LEDs .....218
42. ................283
43. Upgrade error codes from using the
applysoftware command and recommended
actions ..............288
44. Upgrade error codes and recommended
actions ..............291
45. Capacity recovery scenarios.......297
© Copyright IBM Corp. ix

xStorwize V7000 Unified: Problem Determination Guide Version

Safety and environmental notices
Review the multilingual safety notices for the IBM®system before you install and
use the product.
Suitability for telecommunication environment: This product is not intended to
connect directly or indirectly by any means whatsoever to interfaces of public
telecommunications networks.
To find the translated text for a caution or danger notice:
1. Look for the identification number at the end of each caution notice or each
danger notice. In the following examples, the numbers (C001) and (D002) are
the identification numbers.
CAUTION:
A caution notice indicates the presence of a hazard that has the potential of
causing moderate or minor personal injury. (C001)
DANGER
A danger notice indicates the presence of a hazard that has the potential
of causing death or serious personal injury. (D002)
2. Locate IBM Systems Safety Notices with the user publications that were provided
with the Storwize®V7000 Unified hardware.
3. Find the matching identification number in the IBM Systems Safety Notices. Then
review the topics concerning the safety notices to ensure that you are in
compliance.
4. Optionally, read the multilingual safety instructions on the Storwize V7000
Unified website. Go to www.ibm.com/storage/support/storwize/v7000/
unified and click the documentation link.
Safety
Before installing this product, read the Safety Information.
Antes de instalar este produto, leia as Informações de Segurança.
Pred instalací tohoto produktu si prectete prírucku bezpecnostních instrukcí.
© Copyright IBM Corp. xi

Læs sikkerhedsforskrifterne, før du installerer dette produkt.
Lees voordat u dit product installeert eerst de veiligheidsvoorschriften.
Ennen kuin asennat tämän tuotteen, lue turvaohjeet kohdasta Safety Information.
Avant d'installer ce produit, lisez les consignes de sécurité.
Vor der Installation dieses Produkts die Sicherheitshinweise lesen.
Prima di installare questo prodotto, leggere le Informazioni sulla Sicurezza.
Les sikkerhetsinformasjonen (Safety Information) før du installerer dette produktet.
Antes de instalar este produto, leia as Informações sobre Segurança.
Antes de instalar este producto, lea la información de seguridad.
Läs säkerhetsinformationen innan du installerar den här produkten.
xii Storwize V7000 Unified: Problem Determination Guide Version

Safety statements
Each caution and danger statement in this document is labeled with a number.
This number is used to cross reference an English-language caution or danger
statement with translated versions of the caution or danger statement in the Safety
Information document. For example, if a caution statement is labeled “Statement 1”,
translations for that caution statement are in the Safety Information document under
“Statement 1.”
Important:
Be sure to read all caution and danger statements in this document before you
perform the procedures. Read any additional safety information that comes with
the system or optional device before you install the device.
Statement 1
DANGER
Electrical current from power, telephone, and communication cables is
hazardous.
To avoid a shock hazard:
vDo not connect or disconnect any cables or perform installation,
maintenance, or reconfiguration of this product during an electrical storm.
vConnect all power cords to a properly wired and grounded electrical outlet.
vConnect to properly wired outlets any equipment that will be attached to
this product.
vWhen possible, use one hand only to connect or disconnect signal cables.
vNever turn on any equipment when there is evidence of fire, water, or
structural damage.
vDisconnect the attached power cords, telecommunications systems,
networks, and modems before you open the device covers, unless
instructed otherwise in the installation and configuration procedures.
vConnect and disconnect cables as described in the following table when
installing, moving, or opening covers on this product or attached devices.
To Connect: To Disconnect:
1. Turn everything OFF.
2. First, attach all cables to devices.
3. Attach signal cables to connectors.
4. Attach power cords to outlet.
5. Turn device ON.
1. Turn everything OFF.
2. First, remove power cords from outlet.
3. Remove signal cables from connectors.
4. Remove all cables from devices.
Safety and environmental notices xiii

Statement 2
CAUTION:
When replacing the lithium battery, use only IBM Part Number 33F8354 or an
equivalent type battery recommended by the manufacturer. If your system has a
module containing a lithium battery, replace it only with the same module type
made by the same manufacturer. The battery contains lithium and can explode if
not properly used, handled, or disposed of.
Do not:
vThrow or immerse into water
vHeat to more than 100°C (212°F)
vRepair or disassemble
Dispose of the battery as required by local ordinances or regulations.
Statement 3
CAUTION:
When laser products (such as CD-ROMs, DVD drives, fiber optic devices, or
transmitters) are installed, note the following:
vDo not remove the covers. Removing the covers of the laser product could
result in exposure to hazardous laser radiation. There are no serviceable parts
inside the device.
vUse of controls or adjustments or performance of procedures other than those
specified herein might result in hazardous radiation exposure.
xiv Storwize V7000 Unified: Problem Determination Guide Version

DANGER
Some laser products contain an embedded Class 3A or Class 3B laser diode.
Note the following.
Laser radiation when open. Do not stare into the beam, do not view directly
with optical instruments, and avoid direct exposure to the beam.
Class 1 Laser Product
Laser Klasse 1
Laser Klass 1
Luokan 1 Laserlaite
Appareil A Laser de Classe 1
`
Statement 4
≥18 kg (39.7 lb) ≥32 kg (70.5 lb) ≥55 kg (121.2 lb)
CAUTION:
Use safe practices when lifting.
Statement 5
CAUTION:
The power control button on the device and the power switch on the power
supply do not turn off the electrical current supplied to the device. The device
also might have more than one power cord. To remove all electrical current from
the device, ensure that all power cords are disconnected from the power source.
Safety and environmental notices xv

1
2
Statement 8
CAUTION:
Never remove the cover on a power supply or any part that has the following
label attached.
Hazardous voltage, current, and energy levels are present inside any component
that has this label attached. There are no serviceable parts inside these
components. If you suspect a problem with one of these parts, contact a service
technician.
Statement 26
CAUTION:
Do not place any object on top of rack-mounted devices.
This node is suitable for use on an IT power-distribution system whose maximum
phase-to-phase voltage is 240 V under any distribution fault condition.
Important: This product is not suitable for use with visual display workplace
devices according to Clause 2 of the German Ordinance for Work with Visual
Display Units.
xvi Storwize V7000 Unified: Problem Determination Guide Version

Sound pressure
Attention: Depending on local conditions, the sound pressure can exceed 85
dB(A) during service operations. In such cases, wear appropriate hearing
protection.
Safety and environmental notices xvii

xviii Storwize V7000 Unified: Problem Determination Guide Version

About this guide
This guide describes how to service, maintain, and troubleshoot the IBM Storwize
V7000 Unified.
The chapters that follow introduce you to the hardware components and to the
tools that assist you in troubleshooting and servicing the Storwize V7000 Unified,
such as the management GUI and the service assistant.
The troubleshooting procedures can help you analyze failures that occur in a
Storwize V7000 Unified system. With these procedures, you can isolate the
components that fail.
You are also provided with step-by-step procedures to remove and replace parts.
Who should use this guide
This guide is intended for system administrators who use and diagnose problems
with the Storwize V7000 Unified.
Accessibility
IBM has a long-standing commitment to people with disabilities. In keeping with
that commitment to accessibility, IBM strongly supports the U.S. Federal
government's use of accessibility as a criterion in the procurement of Electronic
Information Technology (EIT).
IBM strives to provide products with usable access for everyone, regardless of age
or ability.
This product uses standard Windows navigation keys.
For more information, see “Accessibility features for IBM Storwize V7000 Unified,”
on page 307.
Emphasis
Different typefaces are used in this guide to show emphasis.
The following typefaces are used to show emphasis:
Boldface Text in boldface represents menu items.
Bold monospace Text in bold monospace represents command
names.
Italics Text in italics is used to emphasize a word.
In command syntax, it is used for variables
for which you supply actual values, such as
a default directory or the name of a system.
© Copyright IBM Corp. xix

Monospace Text in monospace identifies the data or
commands that you type, samples of
command output, examples of program code
or messages from the system, or names of
command flags, parameters, arguments, and
name-value pairs.
Storwize V7000 Unified library and related publications
Product manuals, other publications, and websites contain information that relates
to Storwize V7000 Unified.
Storwize V7000 Unified Information Center
The IBM Storwize V7000 Unified Information Center contains all of the
information that is required to install, configure, and manage the Storwize V7000
Unified. The information center is updated between Storwize V7000 Unified
product releases to provide the most current documentation. The information
center is available at the following website:
publib.boulder.ibm.com/infocenter/storwize/unified_ic/index.jsp
Storwize V7000 Unified library
Unless otherwise noted, the publications in the Storwize V7000 Unified library are
available in Adobe portable document format (PDF) from the following website:
www.ibm.com/storage/support/storwize/v7000/unified
Each of the PDF publications in Table 1 is available in this information center by
clicking the number in the “Order number” column:
Table 1. Storwize V7000 Unified library
Title Description Order number
Storwize V7000 Unified Quick
Installation Guide
This guide provides
instructions for unpacking
your shipping order and
installing your system. The
first of three chapters
describes verifying your
order, becoming familiar
with the hardware
components, and meeting
environmental requirements.
The second chapter describes
installing the hardware and
attaching data cables and
power cords. The last
chapter describes accessing
the management GUI to
initially configure your
system.
GA32-1056
Storwize V7000 Unified
Problem Determination Guide
This guide describes how to
service, maintain, and
troubleshoot the Storwize
V7000 Unified system.
GA32-1057
xx Storwize V7000 Unified: Problem Determination Guide Version