IBM Elastic Storage System 3500 User manual

IBM Elastic Storage System 3500
6.1.4
Service Guide
IBM
SC27-9876-00


Note
Before using this information and the product it supports, read the information in “Notices” on page
37.

This edition applies to Version 6 release 1 modication 4 of the following product and to all subsequent releases and
modications until otherwise indicated in new editions:
• IBM Spectrum® Scale Data Management Edition for IBM® ESS (product number 5765-DME)
• IBM Spectrum Scale Data Access Edition for IBM ESS (product number 5765-DAE)
IBM welcomes your comments; see the topic “How to submit your comments” on page x. When you send information
to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without
incurring any obligation to you.
© Copyright International Business Machines Corporation 2022.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM Corp.

Contents
Figures.................................................................................................................. v
Tables................................................................................................................. vii
About this information.......................................................................................... ix
Who should read this information...............................................................................................................ix
IBM Elastic Storage System information units........................................................................................... ix
Related information.....................................................................................................................................ix
Conventions used in this information......................................................................................................... ix
How to submit your comments....................................................................................................................x
Chapter1.Servicing (customer tasks).................................................................... 1
Removing and replacing an NVMe drive......................................................................................................1
Removing and replacing a drive ller.......................................................................................................... 6
Removing and replacing a power module................................................................................................... 9
Removing and replacing a bezel kit...........................................................................................................13
Miscellaneous equipment specication (MES) instructions.....................................................................17
ESS 3500 storage drives concurrent MES upgrade.............................................................................17
ESS 3500 adapter concurrent MES upgrade....................................................................................... 23
Formatting drives example........................................................................................................................29
Manually restarting GPFS on the Enterprise Storage Server 3500 canisters example........................... 31
Chapter2.Part Listings........................................................................................33
CRU part number list..................................................................................................................................33
FRU part number list..................................................................................................................................33
Accessibility features for the system.................................................................... 35
Accessibility features.................................................................................................................................35
Keyboard navigation.................................................................................................................................. 35
IBM and accessibility.................................................................................................................................35
Notices................................................................................................................37
Trademarks................................................................................................................................................ 38
Terms and conditions for product documentation................................................................................... 38
Glossary..............................................................................................................41
Index.................................................................................................................. 49
Index.................................................................................................................. 49
iii

iv

Figures
1. Displaying enclosure front view.................................................................................................................... 1
2. Displaying drive carrier assembly................................................................................................................. 2
3. Displaying NVMe drive replacement part..................................................................................................... 2
4. Displaying physical disks state..................................................................................................................... 3
5. Unlocking the drive and release latch...........................................................................................................4
6. Removing NVMe drive................................................................................................................................... 4
7. Releasing catch..............................................................................................................................................5
8. Displaying drive slots.....................................................................................................................................6
9. Displaying drive carrier assembly................................................................................................................. 7
10. Displaying drive ller replacement kit........................................................................................................ 7
11. Unlocking the drive slot and release latch................................................................................................. 7
12. Removing drive ller....................................................................................................................................8
13. Releasing catch........................................................................................................................................... 8
14. Holding the drive blank............................................................................................................................... 8
15. Inserting the drive ller...............................................................................................................................9
16. Displaying power modules........................................................................................................................10
17. Displaying fan LED.....................................................................................................................................11
18. Displaying power module jacks................................................................................................................ 12
19. Displaying power module tabs................................................................................................................. 12
20. Showing enclosure front view with shipping screws............................................................................... 14
21. Displaying left ear cap front view............................................................................................................. 14
22. Replacing left ear cap............................................................................................................................... 15
23. Displaying right ear cap front view........................................................................................................... 15
v

24. Replacing right ear cap............................................................................................................................. 16
25. Pressing release tab..................................................................................................................................16
26. Securing the enclosure............................................................................................................................. 17
27. Displaying IBM brand label location.........................................................................................................17
28. 5141 - FN2 adapter slots..........................................................................................................................24
29. 5141 - FN2 adapter slot locations............................................................................................................26
vi

Tables
1. Conventions................................................................................................................................................... x
2. IBM Elastic Storage System 3500 power module status LEDs..................................................................11
3. Adapter options...........................................................................................................................................23
4. Adapter placement priority.........................................................................................................................24
5. CRU Part Numbers...................................................................................................................................... 33
6. FRU Part Numbers.......................................................................................................................................33
vii

viii

About this information
Who should read this information
This information is intended for administrators of IBM Elastic Storage® System (ESS) that includes IBM
Spectrum Scale RAID.
IBM Elastic Storage System information units
Related information
Related information
For information about:
• IBM Spectrum Scale, see IBM Documentation.
• mmvdisk command, see mmvdisk documentation.
• Mellanox OFED (MLNX_OFED_LINUX-4.9-5.1.0.2) Release Notes, go to https://docs.nvidia.com/
networking/display/MLNXOFEDv494170/MLNX_OFED+Documentation+Rev+4.9-4.1.7.0+LTS.
• Mellanox OFED (MLNX_OFED_LINUX-5.4-3.0.3.0) Release Notes, go to https://docs.nvidia.com/
networking/display/MLNXOFEDv562090/Release+Notes. (The Mellanox OFED 5.5.x is shipped with ESS
6.1.4.)
• IBM Elastic Storage System, see IBM Documentation.
• IBM Spectrum Scale call home, see Understanding call home.
• Installing IBM Spectrum Scale and CES protocols with the installation toolkit, see Installing IBM
Spectrum Scale on Linux® nodes with the installation toolkit.
• Detailed information about the IBM Spectrum Scale installation toolkit, see Using the installation toolkit
to perform installation tasks: Explanations and examples.
• CES HDFS, see Adding CES HDFS nodes into the centralized le system.
• Installation toolkit ESS support, see ESS awareness with the installation toolkit.
• IBM POWER8® servers, see https://www.ibm.com/docs/en/power-sys-solutions/0008-ESS?
topic=P8ESS/p8hdx/5148_22l_landing.htm
• IBM POWER9™ servers, see https://www.ibm.com/docs/en/ess/6.1.0_ent?topic=guide-5105-22e-
reference-information.
For the latest support information about IBM Spectrum Scale RAID, see the IBM Spectrum Scale RAID
FAQ in IBM Documentation.
Conventions used in this information
Table 1 on page x describes the typographic conventions used in this information. UNIX le name
conventions are used throughout this information.
© Copyright IBM Corp. 2022 ix

Table 1. Conventions
Convention Usage
bold Bold words or characters represent system elements that you must use literally,
such as commands, flags, values, and selected menu options.
Depending on the context, bold typeface sometimes represents path names,
directories, or le names.
bold
underlined
bold underlined keywords are defaults. These take effect if you do not specify a
different keyword.
constant width Examples and information that the system displays appear in constant-width
typeface.
Depending on the context, constant-width typeface sometimes represents path
names, directories, or le names.
italic Italic words or characters represent variable values that you must supply.
Italics are also used for information unit titles, for the rst use of a glossary term,
and for general emphasis in text.
<key>Angle brackets (less-than and greater-than) enclose the name of a key on the
keyboard. For example, <Enter> refers to the key on your terminal or workstation
that is labeled with the word Enter.
\In command examples, a backslash indicates that the command or coding example
continues on the next line. For example:
mkcondition -r IBM.FileSystem -e "PercentTotUsed > 90" \
-E "PercentTotUsed < 85" -m p "FileSystem space used"
{item}Braces enclose a list from which you must choose an item in format and syntax
descriptions.
[item]Brackets enclose optional items in format and syntax descriptions.
<Ctrl-x>The notation <Ctrl-x> indicates a control character sequence. For example,
<Ctrl-c> means that you hold down the control key while pressing <c>.
item... Ellipses indicate that you can repeat the preceding item one or more times.
|In synopsis statements, vertical lines separate a list of choices. In other words, a
vertical line means Or.
In the left margin of the document, vertical lines indicate technical changes to the
information.
How to submit your comments
To contact the IBM Spectrum Scale development organization, send your comments to the following
email address:
[email protected]
xIBM Elastic Storage System 3500: Service Guide

Chapter 1. Servicing (customer tasks)
The following topics contain information about service procedures for IBM Elastic Storage System 3500.
IBM service support representatives and lab-based services personnel can access service information
through the following link.
Servicing (service personnel only)
Note: An IBM intranet connection is required to access this content.
Removing and replacing an NVMe drive
Refer to the service procedure to remove a faulty NVMe drive and replace it with a drive replacement part
in an IBM Elastic Storage System 3500 enclosure.
The following steps are the high-level flow of the procedure:
1. Conrm the failed drive location from the fault drive LED.
2. Prepare the failed drive for replacement.
3. Remove the drive, which has amber LED ON.
4. Insert the FRU replacement drive into the slot.
5. Congure drive after replacement.
6. Run common system health check by using the SSR utility.
Note: If you encounter an error or an unexpected result, contact the next-level support.
This procedure requires access to either management GUI or CLI command as a root user. Replacing a
drive is a task that a customer can complete following the service procedure. If IBM service personnel are
running this task, it requires coordination with the customer for steps that require root user access as part
of the procedure.
No tools are needed to complete this task. Do not remove or loosen any screws.
When you replace this part, you must follow recommended procedures for handling
electrostatic discharge (ESD) sensitive devices.
IBM Elastic Storage System 3500 enclosure supports up to 24 U.2 dual ported NVMe SSDs and the drive
slots are numbered from 1 through 24 as shown in the following gure.
Figure 1. Displaying enclosure front view
© Copyright IBM Corp. 2022 1

A drive carrier assembly is composed of the drive or drive blank and a drive carrier, and are used to
provide for controlled insertion into, and extraction from, the storage enclosure. Drive carrier assemblies
are installed from the front of the enclosure, which simplies service access. Closing the drive carrier
handle ensures complete seating of the connectors.
All 24 drive bays must be lled with drive carrier assemblies, each containing either a drive, or drive
blank, to ensure proper airflow and system cooling. The following gure provides a close-up view of a
drive carrier assembly housing an NVMe drive.
Figure 2. Displaying drive carrier assembly
Fully assembled drive carrier assembly with one of the following options exactly matches the other drives
that are installed in the same enclosure:
• Part number - 01LL727 3.84 TB 2.5" PCIe Gen4 NVMe Flash drive
• Part number - 01LL728 7.68 TB 2.5" PCIe Gen4 NVMe Flash drive
• Part number - 01LL729 15.36 TB 2.5" PCIe Gen4 NVMe Flash drive
• Part number - 01LL823 38 TB 2.5" Flash Core Module
Figure 3. Displaying NVMe drive replacement part
Refer to the following steps to remove and replace a faulty NVMe drive:
1. Conrm the failed drive location from the fault drive LED.
You can identify the failed drive by the amber fault LED on the drive carrier. If the fault LED is lit on a
drive, it is safe to replace the drive.
Alternatively, locate unhealthy drives in the management GUI, either from the Storage> Physical
Disks page as shown in Figure 4 on page 3, or from the list of events that are available under the
Monitoring > Events page.
2IBM Elastic Storage System 3500: Service Guide

Figure 4. Displaying physical disks state
You can also select the Display unhealthy devices option in the Display unhealthy devices >
Hardware Details page to see all the unhealthy devices with the faulty disks.
2. Prepare the failed drive for replacement.
In an IBM Elastic Storage System 3500 enclosure, a faulty drive has amber fault LED ON (non-flashing)
on the driver carrier. At the end of this step, conrm that the amber LED in the ON state for the drive to
be removed.
a) Identify drives for removal with the help one of the following two options:
• Option 1. Removing a drive by using the management GUI.
To replace disks by using the management GUI, follow the direct maintenance procedure (DMP)
that is available in the Maintenance procedures chapter, section Replace disks in the IBM Elastic
Storage System: Problem Determination Guide.
• Option 2. Removing a drive by using CLI commands.
If one or more pdisks in the recovery group are marked for replacement, the following command
reports it with a yes in the “needs service” column:
mmvdisk recoverygroup list
In the following example, the BB01L recovery group needs service:
# mmvdisk recoverygroup list
needs user
recovery group active current or master server service vdisks remarks
-------------- ------- -------------------------------- ------- ------ -------
BB01L yes server01.gpfs.net yes 3
BB01R yes server02.gpfs.net no 3
It happens when the number of failed pdisks in one of the recovery group's declustered arrays
reaches or exceeds the replacement threshold for the declustered array.
The following command lists disks with the threshold for replacement:
mmvdisk pdisk list --recovery-group {all | RgName[,RgName]} --replace
# mmvdisk pdisk list --recovery-group all --replace
recovery group pdisk priority FRU (type) location
-------------- ------------ -------- --------------- --------
Chapter 1. Servicing (customer tasks)3

BB01L e2s11 1.15 00W1240 Enclosure 2 Drive 11
BB01L e3s01 1.15 00W1240 Enclosure 3 Drive 1
mmvdisk: A lower priority value means a higher need for replacement.
b) Prepare each of the pdisk name entries for replacement with the following command:
mmvdisk pdisk replace –-prepare –-recovery-group <rg name> --pdisk <pdisk name>
To prepare pdisk e2s11 of recovery group BB01L for replacement, run the following command:
# mmvdisk pdisk replace --prepare --recovery-group BB01L --pdisk e2s11
mmvdisk: Suspending pdisk e2s11 of RG BB01L in location SX32901810-11.
mmvdisk: Location SX32901810-11 is Enclosure 2 Drive 11.
mmvdisk: Carrier released.
mmvdisk:
mmvdisk: - Remove carrier.
mmvdisk: - Replace disk in location SX32901810-11 with type '00W1240'.
mmvdisk: - Reinsert carrier.
mmvdisk: - Issue the following command:
mmvdisk:
mmvdisk: mmvdisk pdisk replace --recovery-group BB01L --pdisk 'e2s11'
The drive that is associated with the pdisk name in the previous command must have the amber
LED ON to indicate it is safe to remove this drive.
3. Remove the drive, which has amber LED ON.
a) Press the blue touchpoint to unlock the latching handle, as shown in Figure 5 on page 4.
Unlocking the drive and release latch.
Figure 5. Unlocking the drive and release latch
b) Lift the handle and slide the drive out of the enclosure, as shown in the following gure.
Figure 6. Removing NVMe drive
4. Insert the FRU replacement drive into the slot.
4IBM Elastic Storage System 3500: Service Guide

a) Press the release catch on the drive carrier in the direction of the arrow to open the handle as
identied in Figure 7 on page 5. The cam releases from the locked position.
Figure 7. Releasing catch
b) Grab the frame beneath the touch point and gently push the drive carrier assembly into the drive
bay until the carrier handle engages.
c) Press the release handle downward until the release latch connects with the release catch and the
drive carrier assembly locks into place.
d) Ensure that the drive carrier assembly is properly installed fully inside the drive bay so that it is
flush with the face of the enclosure.
e) Ensure that the LED indicators are at the bottom of the drive.
5. After the drive is replaced, run the following command:
mmvdisk pdisk replace –-rg <rg name> --pdisk <pdisk name>
For example, nish replacing pdisk e2s11 with the new physical disk by running the following
command:
# mmvdisk pdisk replace --recovery-group BB01L --pdisk e2s11
mmvdisk:
mmvdisk: Preparing a new pdisk for use may take many minutes.
mmvdisk:
mmvdisk: The following pdisks will be formatted on node ess01io1:
mmvdisk: /dev/sdrk
mmvdisk:
mmvdisk: Location SX32901810-11 is Enclosure 2 Drive 11.
mmvdisk: Pdisk e2s11 of RG BB01L successfully replaced.
mmvdisk: Carrier resumed.
Repeat the steps 2 through 5 for each pdisk that needs to be replaced as marked in the output
of the mmvdisk pdisk list --recovery-group {all | RgName[,RgName]} --replace
command.
Note:
You can also use the commandless disk replacement procedure. The commandless disk replacement
automates the process of replacing a failed or bad drive with a new drive.
To check whether the commandless disk replacement option is enabled, submit the following
command:
mmlsconfig | grep enableAutomaticDiskReplacement
The output appears as follows:
enableAutomaticDiskReplacement no
In this case, commandless disk replacement is not enabled. To enable commandless disk
replacement, submit the following command:
mmchconfig enableAutomaticDiskReplacement=yes -i
Chapter 1. Servicing (customer tasks)5

User needs to enable the option one time only.
For more information, see Commandless disk replacement topic of the Maintenance procedure chapter
in Problem Determination Guide.
6. Run common system health check by using the SSR utility.
Use management GUI or the mmvdisk command to check and conrm that no failed disk is reported.
Removing and replacing a drive ller
Refer to the service procedure to remove a drive slot ller in an IBM Elastic Storage System 3500
enclosure and replace it with a new one from stock.
The following steps are the high-level flow of the procedure:
1. Remove the drive ller.
2. Replace with the drive ller replacement part.
Note: If you encounter an error or an unexpected result, contact the next-level support.
Note: Drive slot llers are passive components that regulate airflow through the enclosure.
No tools are required to complete this task. Do not remove or loosen any screws.
When you replace this part, you must follow recommended procedures for handling
electrostatic discharge (ESD) sensitive devices.
IBM Elastic Storage System 3500 enclosure supports up to 24 U.2 dual ported NVMe SSDs and the drive
slots are numbered from 1 through 24 as shown in Figure 8 on page 6. All 24 drive bays must be lled
with drive carrier assemblies, each containing either a drive, or drive ller (blank), to ensure proper airflow
and system cooling.
Note: Every drive slot of an operational control enclosure must contain either a drive or a drive slot ller.
A drive slot must not be left empty for more than 10 minutes during servicing. Ensure that you have
read and understood the removal and replacement instructions and have the replacement part that is
unpacked before you remove the existing drive slot ller.
Figure 8. Displaying drive slots
A drive carrier assembly is composed of the drive or drive blank and a drive carrier, and are used to
provide for controlled insertion into, and extraction from, the storage enclosure. Drive carrier assemblies
are installed from the front of the enclosure, which simplies service access. Closing the drive carrier
handle ensures complete seating of the connectors.
6IBM Elastic Storage System 3500: Service Guide

Figure 9. Displaying drive carrier assembly
Service personnel need fully assembled drive carrier assembly housing a drive ller to complete this
service procedure:
• Part number - 01LL731 (Drive Filler)
Figure 10. Displaying drive ller replacement kit
Refer to the following steps to remove and replace a drive ller:
1. Remove the drive ller.
a) Press the blue touchpoint to unlock the latching handle, as shown in Figure 11 on page 7. Unlock
the drive slot and release latch.
Figure 11. Unlocking the drive slot and release latch
b) Raise the handle and slide the drive ller out of the enclosure, as shown in Figure 12 on page 8.
Chapter 1. Servicing (customer tasks)7

Figure 12. Removing drive ller
2. Replace with the drive ller replacement part.
a) Press the release catch on the drive carrier in the direction of the arrow to open the handle as
identied in Figure 13 on page 8. The cam releases from the locked position.
Figure 13. Releasing catch
b) Hold the drive blank the correct way up, as shown in Figure 14 on page 8.
Figure 14. Holding the drive blank
c) Ensure that the carrier handle is in the fully open position.
d) Grab the drive ller as shown in Figure 14 on page 8.
8IBM Elastic Storage System 3500: Service Guide

e) Press the release handle downward until the release latch connects with the release catch and the
drive carrier assembly locks into place.
f) Ensure that the drive carrier assembly is properly installed fully inside the drive bay so that it is
flush with the face of the enclosure.
Figure 15. Inserting the drive ller
Removing and replacing a power module
Refer to the service procedure to remove and replace a power module of an IBM Elastic Storage System
3500 enclosure.
The following steps are the high-level flow of the procedure:
1. Follow the appropriate steps to prepare for a service action to release the CMA arm.
2. Conrm the faulty power module location based on fault LED.
3. Unplug power cable to the faulty power module.
4. Remove the faulty power module.
5. Install the replacement power module part.
6. Reconnect power cable to the replaced power module.
7. Follow the appropriate steps to prepare for a service action to return the CMA arm.
8. Run common system health check by using the SSR utility.
Note: If you encounter an error or an unexpected result, contact the next-level support.
No tools are needed to complete this task. Do not remove or loosen any screws.
An IBM Elastic Storage System 3500 (MTM: 5141-FN2) is powered by two high-efciency, hot-swappable
power modules that are installed in the right and left sides at the rear of the enclosure as shown in Figure
16 on page 10.
Chapter 1. Servicing (customer tasks)9

Figure 16. Displaying power modules
Each power module includes one factory-installed 2000W, common redundant power supply (CRPS)
form factor power module, and one factory-installed power adapter board. Upgrading or replacing a
power supply requires replacement of the entire power module. Changing a power supply might require
a change to the power adapter board; therefore, the power supply and power adapter board are not
eld-serviceable individually.
Each power module has active current sharing and supports N+1 redundancy. A single power module can
power the storage enclosure indenitely; however, both power modules ensure redundancy. The power
module provides 12 V power and 3.3 V standby power to the system and includes an internal cooling fan.
Refer to Figure 16 on page 10 to view the location of each power module on the rear side of the enclosure.
When you remove or replace this part, you must follow recommended procedures for
handling ESD sensitive devices.
CAUTION: The power module contains no power switch for Off/On operations. Power Off is
achieved by removing power from the power module. Also, once power is applied to the power
module, the system will power On.
Note: Carry out installation of the device on a properly grounded electrostatic discharge
(ESD) protection surface to avoid damage to the components. Always use a properly grounded ESD wrist
strap or mat during operation, installation of the device, or when you connect power cables.
DANGER: Some factory-approved maintenance procedures on replacement parts (for example,
hot-swapping a power module, fan, NVMe drive, or server canister) do not require the user to
disconnect power to the system. However, performing other types of maintenance on systems that
are connected to an electrical power supply can result in serious injury or death. Only individuals
with knowledge and training in working with devices that contain live circuits should operate the
system. Take precautions when performing all other types of maintenance.
Use the following replacement parts to service a faulty power module in an IBM Elastic Storage System
3500 enclosure:
• Power module (Part number - 01LL893)
• Environmental notice CD (Part number - 02CM200)
Note: To complete this service procedure, go to the rear side of the enclosure to remove the faulty power
module.
Refer to the following steps to remove and replace the faulty power module:
1. Follow the appropriate steps to prepare for a service action to release the CMA arm.
10IBM Elastic Storage System 3500: Service Guide