IBM Elastic Storage System 3200 User manual

IBM Elastic Storage System 3200
6.1.3
Service Guide
IBM
SC27-9862-00

Note
Before using this information and the product it supports, read the information in “Notices” on page
35.
This edition applies to Version 6 release 1 modication 3 of the following product and to all subsequent releases and
modications until otherwise indicated in new editions:
• IBM Spectrum® Scale Data Management Edition for IBM® ESS (product number 5765-DME)
• IBM Spectrum Scale Data Access Edition for IBM ESS (product number 5765-DAE)
IBM welcomes your comments; see the topic “How to submit your comments” on page xi. When you send information
to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without
incurring any obligation to you.
© Copyright International Business Machines Corporation 2022.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM Corp.

Contents
Figures.................................................................................................................. v
Tables................................................................................................................. vii
About this information.......................................................................................... ix
Who should read this information...............................................................................................................ix
IBM Elastic Storage System information units........................................................................................... ix
Related information.....................................................................................................................................ix
Conventions used in this information.......................................................................................................... x
How to submit your comments...................................................................................................................xi
Chapter 1. Servicing (customer tasks).................................................................... 1
Removing and replacing an NVMe drive - GUI (Commandless)................................................................. 1
Removing and replacing an NVMe drive - CLI............................................................................................. 5
Removing and replacing a drive ller.......................................................................................................... 9
Removing and replacing a power module.................................................................................................12
Removing and replacing a bezel kit...........................................................................................................17
Miscellaneous equipment specication (MES) instructions.....................................................................21
ESS 3200 storage drives concurrent MES upgrade.............................................................................21
Formatting drives example........................................................................................................................27
Manually restarting GPFS on the Enterprise Storage Server 3200 canisters example........................... 29
Chapter 2. Part Listings........................................................................................31
CRU part number list..................................................................................................................................31
FRU part number list..................................................................................................................................31
Accessibility features for the system.................................................................... 33
Accessibility features.................................................................................................................................33
Keyboard navigation.................................................................................................................................. 33
IBM and accessibility.................................................................................................................................33
Notices................................................................................................................35
Trademarks................................................................................................................................................ 36
Terms and conditions for product documentation................................................................................... 36
Glossary..............................................................................................................39
Index.................................................................................................................. 47
Index.................................................................................................................. 47
iii

iv

Figures
1. Displaying enclosure front view.................................................................................................................... 2
2. Displaying drive carrier assembly................................................................................................................. 2
3. Displaying NVMe drive replacement part..................................................................................................... 2
4. Displaying physical disks state..................................................................................................................... 3
5. Unlocking the drive and release latch...........................................................................................................4
6. Removing NVMe drive................................................................................................................................... 4
7. Releasing catch..............................................................................................................................................4
8. Displaying enclosure front view.................................................................................................................... 5
9. Displaying drive carrier assembly................................................................................................................. 6
10. Displaying NVMe drive replacement part................................................................................................... 6
11. Displaying physical disks state................................................................................................................... 7
12. Unlocking the drive and release latch........................................................................................................ 8
13. Removing NVMe drive................................................................................................................................. 8
14. Releasing catch........................................................................................................................................... 9
15. Displaying drive slots................................................................................................................................ 10
16. Displaying drive carrier assembly.............................................................................................................10
17. Displaying drive ller replacement kit......................................................................................................10
18. Unlocking the drive slot and release latch............................................................................................... 11
19. Removing drive ller................................................................................................................................. 11
20. Releasing catch......................................................................................................................................... 11
21. Holding the drive blank............................................................................................................................. 12
22. Inserting the drive ller............................................................................................................................ 12
23. Displaying power modules........................................................................................................................13
v

24. Displaying fan LED.....................................................................................................................................14
25. Displaying power module jacks................................................................................................................ 15
26. Displaying power module tabs................................................................................................................. 16
27. Showing enclosure front view with shipping screws............................................................................... 17
28. Displaying left ear cap front view............................................................................................................. 18
29. Replacing left ear cap............................................................................................................................... 18
30. Displaying right ear cap front view........................................................................................................... 19
31. Replacing right ear cap............................................................................................................................. 19
32. Pressing release tab..................................................................................................................................20
33. Securing the enclosure............................................................................................................................. 20
34. Displaying IBM brand label location.........................................................................................................20
vi

Tables
1. Conventions................................................................................................................................................... x
2. IBM Elastic Storage System 3200 power module status LEDs..................................................................14
3. CRU Part Numbers...................................................................................................................................... 31
4. FRU Part Numbers.......................................................................................................................................31
vii

viii

About this information
Who should read this information
This information is intended for administrators of IBM Elastic Storage® System (ESS) that includes IBM
Spectrum Scale RAID.
IBM Elastic Storage System information units
IBM Elastic Storage System (ESS) 3200 documentation consists of the following information units.
Information unit Type of information Intended users
Hardware Planning
and Installation
Guide
This unit provides ESS 3200 information
including technical overview, planning,
installing, troubleshooting, and cabling.
System administrators and IBM
support team
Quick Deployment
Guide
This unit provides ESS information including the
software stack, deploying, upgrading, setting up
call home, and best practices.
System administrators,
analysts, installers, planners,
and programmers of IBM
Spectrum Scale clusters who
are very experienced with the
operating systems on which
each IBM Spectrum Scale
cluster is based
Service Guide This unit provides ESS 3200 information
including servicing and parts listings.
System administrators and IBM
support team
Problem
Determination Guide
This unit provides ESS 3200 information
including events, replacing servers, issues,
maintenance procedures, and troubleshooting.
System administrators and IBM
support team
Command Reference This unit provides information about ESS
commands and scripts.
System administrators and IBM
support team
IBM Spectrum
Scale RAID:
Administration
This unit provides IBM Spectrum Scale RAID
information including administering, monitoring,
commands, and scripts.
• System administrators of IBM
Spectrum Scale systems
• Application programmers
who are experienced with
IBM Spectrum Scale systems
and familiar with the
terminology and concepts in
the XDSM standard
Related information
Related information
For information about:
• IBM Spectrum Scale, see IBM Documentation.
• mmvdisk command, see mmvdisk documentation.
• Mellanox OFED (MLNX_OFED_LINUX-4.9-4.1.7.2) Release Notes, go to https://docs.nvidia.com/
networking/spaces/viewspace.action?key=MLNXOFEDv494080.
© Copyright IBM Corp. 2022 ix

• Mellanox OFED (MLNX_OFED_LINUX-5.4-3.0.3.0) Release Notes, go to https://docs.mellanox.com/
display/MLNXOFEDv543030/Release+Notes. (Actual version shipped is 5.5-x.)
• IBM Elastic Storage System, see IBM Documentation.
• IBM Spectrum Scale call home, see Understanding call home.
• Installing IBM Spectrum Scale and CES protocols with the installation toolkit, see Installing IBM
Spectrum Scale on Linux® nodes with the installation toolkit.
• Detailed information about the IBM Spectrum Scale installation toolkit, see Using the installation toolkit
to perform installation tasks: Explanations and examples.
• CES HDFS, see Adding CES HDFS nodes into the centralized le system.
• Installation toolkit ESS support, see ESS awareness with the installation toolkit.
• IBM POWER8® servers, see https://www.ibm.com/docs/en/power-sys-solutions/0008-ESS?
topic=P8ESS/p8hdx/5148_22l_landing.htm
• IBM POWER9™ servers, see https://www.ibm.com/docs/en/ess/6.1.0_ent?topic=guide-5105-22e-
reference-information.
For the latest support information about IBM Spectrum Scale RAID, see the IBM Spectrum Scale RAID
FAQ in IBM Documentation.
Conventions used in this information
Table 1 on page x describes the typographic conventions used in this information. UNIX le name
conventions are used throughout this information.
Table 1. Conventions
Convention Usage
bold Bold words or characters represent system elements that you must use literally,
such as commands, flags, values, and selected menu options.
Depending on the context, bold typeface sometimes represents path names,
directories, or le names.
bold
underlined
bold underlined keywords are defaults. These take effect if you do not specify a
different keyword.
constant width Examples and information that the system displays appear in constant-width
typeface.
Depending on the context, constant-width typeface sometimes represents path
names, directories, or le names.
italic Italic words or characters represent variable values that you must supply.
Italics are also used for information unit titles, for the rst use of a glossary term,
and for general emphasis in text.
<key>Angle brackets (less-than and greater-than) enclose the name of a key on the
keyboard. For example, <Enter> refers to the key on your terminal or workstation
that is labeled with the word Enter.
\In command examples, a backslash indicates that the command or coding example
continues on the next line. For example:
mkcondition -r IBM.FileSystem -e "PercentTotUsed > 90" \
-E "PercentTotUsed < 85" -m p "FileSystem space used"
{item}Braces enclose a list from which you must choose an item in format and syntax
descriptions.
x About this information

Table 1. Conventions (continued)
Convention Usage
[item]Brackets enclose optional items in format and syntax descriptions.
<Ctrl-x>The notation <Ctrl-x> indicates a control character sequence. For example,
<Ctrl-c> means that you hold down the control key while pressing <c>.
item... Ellipses indicate that you can repeat the preceding item one or more times.
|In synopsis statements, vertical lines separate a list of choices. In other words, a
vertical line means Or.
In the left margin of the document, vertical lines indicate technical changes to the
information.
How to submit your comments
To contact the IBM Spectrum Scale development organization, send your comments to the following
email address:
[email protected]
About this information xi

xii IBM Elastic Storage System 3200: Service Guide

Chapter 1. Servicing (customer tasks)
The following topics contain information about service procedures for IBM Elastic Storage System 3200.
IBM service support representatives and lab-based services personnel can access service information
through the following link.
Servicing (service personnel only)
Note: An IBM intranet connection is required to access this content.
Removing and replacing an NVMe drive - GUI (Commandless)
Refer to the service procedure to remove a faulty NVMe drive and replace it with a drive replacement part
in an IBM Elastic Storage System 3200 enclosure from GUI. This procedure is for commandless NVMe
drive replacement.
The following steps are the high-level flow of the procedure:
1. Conrm the failed drive location from the fault drive LED.
2. Prepare the failed drive for replacement.
3. Remove the drive, which has amber LED ON.
4. Insert the FRU replacement drive into the slot.
5. Run common system health check by using the SSR utility.
Note: If you encounter an error or an unexpected result, contact the next-level support.
Note:
The commandless disk replacement automates the process of replacing a failed or bad drive with a new
drive.
To check whether the commandless disk replacement option is enabled, submit the following command:
mmlsconfig | grep enableAutomaticDiskReplacement
The output appears as follows:
enableAutomaticDiskReplacement no
In this case, commandless disk replacement is not enabled. To enable commandless disk replacement,
submit the following command:
mmchconfig enableAutomaticDiskReplacement=yes -i
User needs to enable the option one time only.
This procedure requires access to management GUI.
No tools are needed to complete this task. Do not remove or loosen any screws.
When you replace this part, you must follow recommended procedures for handling
electrostatic discharge (ESD) sensitive devices.
IBM Elastic Storage System 3200 enclosure supports up to 24 U.2 dual ported NVMe SSDs and the drive
slots are numbered from 1 through 24 as shown in the following gure.
© Copyright IBM Corp. 2022 1

Figure 1. Displaying enclosure front view
A drive carrier assembly is composed of the drive or drive blank and a drive carrier, and are used to
provide for controlled insertion into, and extraction from, the storage enclosure. Drive carrier assemblies
are installed from the front of the enclosure, which simplies service access. Closing the drive carrier
handle ensures complete seating of the connectors.
All 24 drive bays must be lled with drive carrier assemblies, each containing either a drive, or drive
blank, to ensure proper airflow and system cooling. The following gure provides a close-up view of a
drive carrier assembly housing an NVMe drive.
Figure 2. Displaying drive carrier assembly
Fully assembled drive carrier assembly with one of the following options exactly matches the other drives
that are installed in the same enclosure:
• Part number - 01LL727 3.84 TB 2.5" PCIe Gen4 NVMe Flash drive
• Part number - 01LL728 7.68 TB 2.5" PCIe Gen4 NVMe Flash drive
• Part number - 01LL729 15.36 TB 2.5" PCIe Gen4 NVMe Flash drive
• Part number - 01LL831 38.4 TB FCM 2.1 PCIe Gen3 NVMe Flash drive
Figure 3. Displaying NVMe drive replacement part
Refer to the following steps to remove and replace a faulty NVMe drive:
1. Conrm the failed drive location from the fault drive LED.
You can identify the failed drive by the amber fault LED on the drive carrier. If the fault LED is lit on a
drive, it is safe to replace the drive.
2 IBM Elastic Storage System 3200: Service Guide

Alternatively, locate unhealthy drives in the management GUI, either from the Storage> Physical
Disks page as shown in Figure 4 on page 3, or from the list of events that are available under the
Monitoring > Events page.
Figure 4. Displaying physical disks state
You can also select the Display unhealthy devices option in the Display unhealthy devices >
Hardware Details page to see all the unhealthy devices with the faulty disks.
2. Prepare the failed drive for replacement.
The directed maintenance procedures (DMPs) assist you to repair a problem when you select the
action Run x procedure on a selected event from the Monitoring > Events page.
The ESS GUI detects if a disk is broken and whether it needs to be replaced. In this case, launch this
DMP to get support to replace the broken disks. You can use this DMP either to replace one disk or
multiple disks.
The following are the corresponding event details and proposed solution:
•Event name: gnr_pdisk_replaceable
•Problem: The state of a physical disk is changed to replaceable.
•Solution: Replace the disk.
The DMP automatically launches in corresponding mode depending on situation. You can launch this
DMP from the pages in the GUI and follow the wizard to release one or more disks:
•Monitoring > Hardware page: Select Replace Broken Disks from the Actions menu.
•Monitoring > Hardware page: Select the broken disk to be replaced in an enclosure and then select
Replace from the Actions menu.
•Monitoring > Events page: Select the gnr_pdisk_replaceable event from the event listing and then
select Run Fix Procedure from the Actions menu.
•Storage > Physical Disks page: Select Replace Broken Disks from the Actions menu.
•Storage > Physical Disks page: Select the disk to be replaced and then select Replace Disk from
the Actions menu.
The system uses the following command on an mmvdisk-enabled environment to release and replace
the disk:
mmvdisk pdisk replace [--prepare | --cancel] --recovery-group DiskRecoveryGroup --pdisk
DiskName
Chapter 1. Servicing (customer tasks) 3

3. Remove the drive, which has amber LED ON.
a) Press the blue touchpoint to unlock the latching handle, as shown in Figure 5 on page 4.
Unlocking the drive and release latch.
Figure 5. Unlocking the drive and release latch
b) Lift the handle and slide the drive out of the enclosure, as shown in the following gure.
Figure 6. Removing NVMe drive
4. Insert the FRU replacement drive into the slot.
a) Press the release catch on the drive carrier in the direction of the arrow to open the handle as
identied in Figure 7 on page 4. The cam releases from the locked position.
Figure 7. Releasing catch
b) Grab the frame beneath the touch point and gently push the drive carrier assembly into the drive
bay until the carrier handle engages.
c) Press the release handle downward until the release latch connects with the release catch and the
drive carrier assembly locks into place.
4 IBM Elastic Storage System 3200: Service Guide

d) Ensure that the drive carrier assembly is properly installed fully inside the drive bay so that it is
flush with the face of the enclosure.
e) Ensure that the LED indicators are at the bottom of the drive.
5. Run common system health check by using the SSR utility.
Use management GUI to check and conrm that no failed disk is reported.
Removing and replacing an NVMe drive - CLI
Refer to the service procedure to remove a faulty NVMe drive and replace it with a drive replacement part
in an IBM Elastic Storage System 3200 enclosure from command line interface.
The following steps are the high-level flow of the procedure:
1. Conrm the failed drive location from the fault drive LED.
2. Prepare the failed drive for replacement.
3. Remove the drive, which has amber LED ON.
4. Insert the FRU replacement drive into the slot.
5. Congure drive after replacement.
6. Run common system health check by using the SSR utility.
Note: If you encounter an error or an unexpected result, contact the next-level support.
This procedure requires access to CLI command as a root user. If IBM service personnel are running this
task, it requires coordination with the customer for steps that require root user access as part of the
procedure.
No tools are needed to complete this task. Do not remove or loosen any screws.
When you replace this part, you must follow recommended procedures for handling
electrostatic discharge (ESD) sensitive devices.
IBM Elastic Storage System 3200 enclosure supports up to 24 U.2 dual ported NVMe SSDs and the drive
slots are numbered from 1 through 24 as shown in the following gure.
Figure 8. Displaying enclosure front view
A drive carrier assembly is composed of the drive or drive blank and a drive carrier, and are used to
provide for controlled insertion into, and extraction from, the storage enclosure. Drive carrier assemblies
are installed from the front of the enclosure, which simplies service access. Closing the drive carrier
handle ensures complete seating of the connectors.
All 24 drive bays must be lled with drive carrier assemblies, each containing either a drive, or drive
blank, to ensure proper airflow and system cooling. The following gure provides a close-up view of a
drive carrier assembly housing an NVMe drive.
Chapter 1. Servicing (customer tasks) 5

Figure 9. Displaying drive carrier assembly
Fully assembled drive carrier assembly with one of the following options exactly matches the other drives
that are installed in the same enclosure:
• Part number - 01LL727 3.84 TB 2.5" PCIe Gen4 NVMe Flash drive
• Part number - 01LL728 7.68 TB 2.5" PCIe Gen4 NVMe Flash drive
• Part number - 01LL729 15.36 TB 2.5" PCIe Gen4 NVMe Flash drive
• Part number - 01LL831 38.4 TB FCM 2.1 PCIe Gen3 NVMe Flash drive
Figure 10. Displaying NVMe drive replacement part
Refer to the following steps to remove and replace a faulty NVMe drive:
1. Conrm the failed drive location from the fault drive LED.
You can identify the failed drive by the amber fault LED on the drive carrier. If the fault LED is lit on a
drive, it is safe to replace the drive.
Alternatively, locate unhealthy drives in the management GUI, either from the Storage> Physical
Disks page as shown in Figure 11 on page 7, or from the list of events that are available under the
Monitoring > Events page.
6 IBM Elastic Storage System 3200: Service Guide

Figure 11. Displaying physical disks state
You can also select the Display unhealthy devices option in the Display unhealthy devices >
Hardware Details page to see all the unhealthy devices with the faulty disks.
2. Prepare the failed drive for replacement.
In an IBM Elastic Storage System 3200 enclosure, a faulty drive has amber fault LED ON (non-flashing)
on the driver carrier. At the end of this step, conrm that the amber LED in the ON state for the drive to
be removed.
a) Identify drives for removal.
If one or more pdisks in the recovery group are marked for replacement, the following command
reports it with a yes in the needs service column:
mmvdisk recoverygroup list
In the following example, the BB01L recovery group needs service:
# mmvdisk recoverygroup list
needs user
recovery group active current or master server service vdisks remarks
-------------- ------- -------------------------------- ------- ------ -------
BB01L yes server01.gpfs.net yes 3
BB01R yes server02.gpfs.net no 3
It happens when the number of failed pdisks in one of the recovery group's declustered arrays
reaches or exceeds the replacement threshold for the declustered array.
The following command lists disks with the threshold for replacement:
mmvdisk pdisk list --recovery-group {all | RgName[,RgName]} --replace
# mmvdisk pdisk list --recovery-group all --replace
recovery group pdisk priority FRU (type) location
-------------- ------------ -------- --------------- --------
BB01L e2s11 1.15 00W1240 Enclosure 2 Drive 11
BB01L e3s01 1.15 00W1240 Enclosure 3 Drive 1
mmvdisk: A lower priority value means a higher need for replacement.
b) Prepare each of the pdisk name entries for replacement with the following command:
Chapter 1. Servicing (customer tasks) 7

mmvdisk pdisk replace –-prepare –-recovery-group <rg name> --pdisk <pdisk name>
To prepare pdisk e2s11 of recovery group BB01L for replacement, run the following command:
# mmvdisk pdisk replace --prepare --recovery-group BB01L --pdisk e2s11
mmvdisk: Suspending pdisk e2s11 of RG BB01L in location SX32901810-11.
mmvdisk: Location SX32901810-11 is Enclosure 2 Drive 11.
mmvdisk: Carrier released.
mmvdisk:
mmvdisk: - Remove carrier.
mmvdisk: - Replace disk in location SX32901810-11 with type '00W1240'.
mmvdisk: - Reinsert carrier.
mmvdisk: - Issue the following command:
mmvdisk:
mmvdisk: mmvdisk pdisk replace --recovery-group BB01L --pdisk 'e2s11'
The drive that is associated with the pdisk name in the previous command must have the amber
LED ON to indicate it is safe to remove this drive.
3. Remove the drive, which has amber LED ON.
a) Press the blue touchpoint to unlock the latching handle, as shown in Figure 12 on page 8.
Unlocking the drive and release latch.
Figure 12. Unlocking the drive and release latch
b) Lift the handle and slide the drive out of the enclosure, as shown in the following gure.
Figure 13. Removing NVMe drive
4. Insert the FRU replacement drive into the slot.
a) Press the release catch on the drive carrier in the direction of the arrow to open the handle as
identied in Figure 14 on page 9. The cam releases from the locked position.
8 IBM Elastic Storage System 3200: Service Guide