Bull 58/88 Manual

Service procedures for the PCIe storage
enclosure
ESCALA Power7
REFERENCE
86 A1 25FK 01

Service procedures for the PCIe storage enclosure
The ESCALA Power7 publications concern the following models:
- Bull Escala E1-700 (Power 710 / 8231-E2B)
- Bull Escala E1-705 (Power 710 / 8231-E1C)
- Bull Escala E2-700 / E2-700T (Power 720 / 8202-E4B)
- Bull Escala E2-705 / E2-705T (Power 720 / 8202-E4C)
- Bull Escala E3-700 (Power 730 / 8231-E2B)
- Bull Escala E3-705 (Power 730 / 8231-E2C)
- Bull Escala E4-700 / E4-700T (Power 740 / 8205-E6B)
- Bull Escala E4-705 (Power 740 / 8205-E6C)
- Bull Escala E5-700 (Power 750 / 8233-E8B)
- Bull Escala M6-700 (Power 770 / 9117-MMB)
- Bull Escala M6-705 (Power 770 / 9117-MMC)
- Bull Escala M6-715 (Power 770 / 9117-MMD)
- Bull Escala M7-700 (Power 780 / 9179-MHB)
- Bull Escala M7-705 (Power 780 / 9179-MHC)
- Bull Escala M7-715 (Power 780 / 9179-MHD)
References to Power 755 / 8236-E8C models are irrelevant.
Hardware
November 2012
BULL CEDOC
357 AVENUE PATTON
B.P.20845
49008 ANGERS CEDEX 01
FRANCE
REFERENCE
86 A1 25FK 01

The following copyright notice protects this book under Copyright laws which prohibit such actions as, but not limited to, copying,
distributing, modifying, and making derivative works.
Copyright © Bull SAS 2012
Printed in France
Trademarks and Acknowledgements
We acknowledge the rights of the proprietors of the trademarks mentioned in this manual.
All brand names and software and hardware product names are subject to trademark and/or patent protection.
Quoting of brand and product names is for information purposes only and does not represent trademark misuse.
The information in this document is subject to change without notice. Bull will not be liable for errors contained herein, or
for incidental or consequential damages in connection with the use of this material.

Contents
Safety notices .................................v
Service information for the 58/88 or ED/R1 PCIe storage enclosure ..........1
58/88 or ED/R1 PCIe storage enclosure service overview ....................1
Introduction to the PCIe storage enclosure infrastructure ...................2
PCIe storage enclosure features .............................3
Cabling the PCIe storage enclosure ............................4
Cabling the 58/88 PCIe storage enclosure to a 31E/1C or 31E/2C system ..............4
Cabling the 58/88 PCIe storage enclosure to a 02E/4C or 05E/6C system ..............6
Cabling the ED/R1 PCIe storage enclosure to a 17M/MD or 79M/HD system ............11
Cabling the 58/87 disk drive enclosure to an ED/R1 PCIe storage enclosure ............16
Overview of problem determination ...........................18
LED usage for the 58/88 or ED/R1 PCIe storage enclosure ..................19
RAID controller location code ............................19
Determine the ERM location from the RAID controller location ................19
Determining the ERM location from the RAID controller location when using AIX ........19
Determining the ERM location from the RAID controller location when using Linux ........22
Updating microcode for PCIe storage enclosure components ..................23
FRUs for the 58/88 or ED/R1 PCIe storage enclosure ....................23
Identifying a 58/88 or ED/R1 PCIe storage enclosure SAS fabric FRU ...............24
Storage enclosure topology .............................24
Notices ...................................27
Trademarks ...................................28
Electronic emission notices ..............................28
Class A Notices .................................28
Class B Notices .................................32
Terms and conditions ................................34
iii

iv Power systems: Service procedures for the PCIe storage enclosure

Safety notices
Safety notices may be printed throughout this guide.
vDANGER notices call attention to a situation that is potentially lethal or extremely hazardous to
people.
vCAUTION notices call attention to a situation that is potentially hazardous to people because of some
existing condition.
vAttention notices call attention to the possibility of damage to a program, device, system, or data.
World Trade safety information
Several countries require the safety information contained in product publications to be presented in their
national languages. If this requirement applies to your country, a safety information booklet is included
in the publications package shipped with the product. The booklet contains the safety information in
your national language with references to the U.S. English source. Before using a U.S. English publication
to install, operate, or service this product, you must first become familiar with the related safety
information in the booklet. You should also refer to the booklet any time you do not clearly understand
any safety information in the U.S. English publications.
Laser safety information
The servers can use I/O cards or features that are fiber-optic based and that utilize lasers or LEDs.
Laser compliance
The servers may be installed inside or outside of an IT equipment rack.
v

DANGER
When working on or around the system, observe the following precautions:
Electrical voltage and current from power, telephone, and communication cables are hazardous. To
avoid a shock hazard:
vConnect power to this unit only with the provided power cord. Do not use the provided power
cord for any other product.
vDo not open or service any power supply assembly.
vDo not connect or disconnect any cables or perform installation, maintenance, or reconfiguration
of this product during an electrical storm.
vThe product might be equipped with multiple power cords. To remove all hazardous voltages,
disconnect all power cords.
vConnect all power cords to a properly wired and grounded electrical outlet. Ensure that the outlet
supplies proper voltage and phase rotation according to the system rating plate.
vConnect any equipment that will be attached to this product to properly wired outlets.
vWhen possible, use one hand only to connect or disconnect signal cables.
vNever turn on any equipment when there is evidence of fire, water, or structural damage.
vDisconnect the attached power cords, telecommunications systems, networks, and modems before
you open the device covers, unless instructed otherwise in the installation and configuration
procedures.
vConnect and disconnect cables as described in the following procedures when installing, moving,
or opening covers on this product or attached devices.
To Disconnect:
1. Turn off everything (unless instructed otherwise).
2. Remove the power cords from the outlets.
3. Remove the signal cables from the connectors.
4. Remove all cables from the devices.
To Connect:
1. Turn off everything (unless instructed otherwise).
2. Attach all cables to the devices.
3. Attach the signal cables to the connectors.
4. Attach the power cords to the outlets.
5. Turn on the devices.
(D005a)
DANGER
vi Power systems: Service procedures for the PCIe storage enclosure

Observe the following precautions when working on or around your IT rack system:
vHeavy equipment–personal injury or equipment damage might result if mishandled.
vAlways lower the leveling pads on the rack cabinet.
vAlways install stabilizer brackets on the rack cabinet.
vTo avoid hazardous conditions due to uneven mechanical loading, always install the heaviest
devices in the bottom of the rack cabinet. Always install servers and optional devices starting
from the bottom of the rack cabinet.
vRack-mounted devices are not to be used as shelves or work spaces. Do not place objects on top
of rack-mounted devices.
vEach rack cabinet might have more than one power cord. Be sure to disconnect all power cords in
the rack cabinet when directed to disconnect power during servicing.
vConnect all devices installed in a rack cabinet to power devices installed in the same rack
cabinet. Do not plug a power cord from a device installed in one rack cabinet into a power
device installed in a different rack cabinet.
vAn electrical outlet that is not correctly wired could place hazardous voltage on the metal parts of
the system or the devices that attach to the system. It is the responsibility of the customer to
ensure that the outlet is correctly wired and grounded to prevent an electrical shock.
CAUTION
vDo not install a unit in a rack where the internal rack ambient temperatures will exceed the
manufacturer's recommended ambient temperature for all your rack-mounted devices.
vDo not install a unit in a rack where the air flow is compromised. Ensure that air flow is not
blocked or reduced on any side, front, or back of a unit used for air flow through the unit.
vConsideration should be given to the connection of the equipment to the supply circuit so that
overloading of the circuits does not compromise the supply wiring or overcurrent protection. To
provide the correct power connection to a rack, refer to the rating labels located on the
equipment in the rack to determine the total power requirement of the supply circuit.
v(For sliding drawers.) Do not pull out or install any drawer or feature if the rack stabilizer brackets
are not attached to the rack. Do not pull out more than one drawer at a time. The rack might
become unstable if you pull out more than one drawer at a time.
v(For fixed drawers.) This drawer is a fixed drawer and must not be moved for servicing unless
specified by the manufacturer. Attempting to move the drawer partially or completely out of the
rack might cause the rack to become unstable or cause the drawer to fall out of the rack.
(R001)
Safety notices vii

CAUTION:
Removing components from the upper positions in the rack cabinet improves rack stability during
relocation. Follow these general guidelines whenever you relocate a populated rack cabinet within a
room or building:
vReduce the weight of the rack cabinet by removing equipment starting at the top of the rack
cabinet. When possible, restore the rack cabinet to the configuration of the rack cabinet as you
received it. If this configuration is not known, you must observe the following precautions:
– Remove all devices in the 32U position and above.
– Ensure that the heaviest devices are installed in the bottom of the rack cabinet.
– Ensure that there are no empty U-levels between devices installed in the rack cabinet below the
32U level.
vIf the rack cabinet you are relocating is part of a suite of rack cabinets, detach the rack cabinet from
the suite.
vInspect the route that you plan to take to eliminate potential hazards.
vVerify that the route that you choose can support the weight of the loaded rack cabinet. Refer to the
documentation that comes with your rack cabinet for the weight of a loaded rack cabinet.
vVerify that all door openings are at least 760 x 230 mm (30 x 80 in.).
vEnsure that all devices, shelves, drawers, doors, and cables are secure.
vEnsure that the four leveling pads are raised to their highest position.
vEnsure that there is no stabilizer bracket installed on the rack cabinet during movement.
vDo not use a ramp inclined at more than 10 degrees.
vWhen the rack cabinet is in the new location, complete the following steps:
– Lower the four leveling pads.
– Install stabilizer brackets on the rack cabinet.
– If you removed any devices from the rack cabinet, repopulate the rack cabinet from the lowest
position to the highest position.
vIf a long-distance relocation is required, restore the rack cabinet to the configuration of the rack
cabinet as you received it. Pack the rack cabinet in the original packaging material, or equivalent.
Also lower the leveling pads to raise the casters off of the pallet and bolt the rack cabinet to the
pallet.
(R002)
(L001)
(L002)
viii Power systems: Service procedures for the PCIe storage enclosure

(L003)
or
All lasers are certified in the U.S. to conform to the requirements of DHHS 21 CFR Subchapter J for class
1 laser products. Outside the U.S., they are certified to be in compliance with IEC 60825 as a class 1 laser
product. Consult the label on each part for laser certification numbers and approval information.
CAUTION:
This product might contain one or more of the following devices: CD-ROM drive, DVD-ROM drive,
DVD-RAM drive, or laser module, which are Class 1 laser products. Note the following information:
vDo not remove the covers. Removing the covers of the laser product could result in exposure to
hazardous laser radiation. There are no serviceable parts inside the device.
vUse of the controls or adjustments or performance of procedures other than those specified herein
might result in hazardous radiation exposure.
(C026)
Safety notices ix

CAUTION:
Data processing environments can contain equipment transmitting on system links with laser modules
that operate at greater than Class 1 power levels. For this reason, never look into the end of an optical
fiber cable or open receptacle. (C027)
CAUTION:
This product contains a Class 1M laser. Do not view directly with optical instruments. (C028)
CAUTION:
Some laser products contain an embedded Class 3A or Class 3B laser diode. Note the following
information: laser radiation when open. Do not stare into the beam, do not view directly with optical
instruments, and avoid direct exposure to the beam. (C030)
CAUTION:
The battery contains lithium. To avoid possible explosion, do not burn or charge the battery.
Do Not:
v___ Throw or immerse into water
v___ Heat to more than 100°C (212°F)
v___ Repair or disassemble
Exchange only with the approved part. Recycle or discard the battery as instructed by local
regulations. (C003a)
Power and cabling information for NEBS (Network Equipment-Building System)
GR-1089-CORE
The following comments apply to the servers that have been designated as conforming to NEBS
(Network Equipment-Building System) GR-1089-CORE:
The equipment is suitable for installation in the following:
vNetwork telecommunications facilities
vLocations where the NEC (National Electrical Code) applies
The intrabuilding ports of this equipment are suitable for connection to intrabuilding or unexposed
wiring or cabling only. The intrabuilding ports of this equipment must not be metallically connected to the
interfaces that connect to the OSP (outside plant) or its wiring. These interfaces are designed for use as
intrabuilding interfaces only (Type 2 or Type 4 ports as described in GR-1089-CORE) and require isolation
from the exposed OSP cabling. The addition of primary protectors is not sufficient protection to connect
these interfaces metallically to OSP wiring.
Note: All Ethernet cables must be shielded and grounded at both ends.
The ac-powered system does not require the use of an external surge protection device (SPD).
The dc-powered system employs an isolated DC return (DC-I) design. The DC battery return terminal
shall not be connected to the chassis or frame ground.
xPower systems: Service procedures for the PCIe storage enclosure

Service information for the 58/88 or ED/R1 PCIe storage
enclosure
This topic provides service information for the 58/88 or ED/R1 PCIe storage enclosure.
58/88 or ED/R1 PCIe storage enclosure service overview
Find information to help you service the 58/88 or ED/R1 PCIe storage enclosure.
The (feature code 58/88 and ED/R1), are 19 inch, rack-mountable 1.8-inch solid-state drive (SSD)
optimized PCIe storage enclosures. They attach to supported POWER7®servers by using PCI Express
generation 2 (PCIe) adapters and cables. The 58/88 or ED/R1 PCIe storage enclosure include the
following components:
vRedundant power supplies
vFans
vTwo large cache SAS RAID controllers along with redundant SAS port expanders
vUp to 30 1.8 inch dual ported SAS SSDs
The 58/88 and ED/R1 PCIe storage enclosures contain two redundant enclosure RAID modules (ERMs),
which contain a SAS RAID controller, a SAS port expander, and a replaceable fan assembly. Two power
supplies provide an N+1 redundant feature that allows one supply to power the entire box if the other is
failed or missing.
vThe two power supplies have built in fans.
vFan speed is automatically adjusted as needed to ensure proper enclosure cooling temperatures even
when there is a fan failure.
The SAS RAID controllers are always configured in a dual controller (dual storage IOA) configuration.
This configuration provides redundant access paths to the SAS devices and mirrored copies of cache data
and parity update footprints. The dual controller configuration also allows for increased performance
capability when multiple RAID arrays are used in an Active/Active configuration. The SAS RAID
controllers support RAID 0, 10, 5, and 6 and hot spare.
The 58/88 or ED/R1 PCIe storage enclosure is a one EIA unit (1U) drawer that mounts in a 19-inch rack.
It is designed for light path service with light-emitting diode (LED) indicators on the front and rear of the
enclosure chassis. These chassis LED indicators provide status and fault information for:
vEnclosure identification
vEnclosure power
vEnclosure fans
vERMs
vEnclosure faults
vSSD activity and SSD fault or identification
The PCIe storage enclosure is designed for front and rear service in place (no service position) except for
the midplane FRU replacement.
1

Introduction to the PCIe storage enclosure infrastructure
Learn how the 58/88 or ED/R1 PCIe storage enclosure is different from previous storage enclosures.
With other IBM®SAS storage enclosure models, connecting disk drives was accomplished by using
individual components externally connected to each other:
vStorage enclosures
vDisk drive controllers
vSAS cables for multiple paths to devices and dedicated adapter to adapter connections.
Figure 1 shows an example of a dual storage I/O adapter (IOA) solution by using externally connected
components.
With the 58/88 or ED/R1 PCIe storage enclosures, all of the necessary components are integrated within
the storage enclosure.
Figure 2 on page 3 shows a dual storage IOA solution integrated within the 58/88 or ED/R1 PCIe storage
enclosure.
Figure 1. Dual storage IOA solution by using externally connected components
2Power systems: Service procedures for the PCIe storage enclosure

The following components and functions are integrated within the 58/88 or ED/R1 PCIe storage
enclosure:
vThe enclosure RAID module (ERM) contains the RAID adapter function and the expander function.
vThe storage enclosure uses internal wiring to connect components with multiple redundant paths,
eliminating the need for external SAS cables.
vIncreased number of 1.5-inch solid-state drives (SSDs) (up to 30).
PCIe storage enclosure features
This table shows the main features of the 58/88 and ED/R1 PCI Express (PCIe) storage enclosures.
Table 1. PCIe storage enclosure features
Feature code 58/88 ED/R1
Description IBM EXP30 Ultra solid-state drive
(SSD) I/O drawer
IBM EXP30 Ultra solid-state drive
(SSD) I/O drawer
System connection x8 x8
Systems supported 31E/2B, 31E/1C, or 31E/2C, 02E/4C,
05E/6C
31E/2B, 31E/1C, or 31E/2C, 02E/4C,
05E/6C
Operating systems supported1AIX®, Linux AIX, Linux
High availability (HA) two-system
redundant array of independent
disks (RAID)
Yes Yes
HA single-system RAID Yes Yes
Requires HA RAID configuration Yes Yes
JBOD drives supported No (requires drives formatted 528
bytes/block)
No (requires drives formatted 528
bytes/block)
Internal SSD drive type 1.8-inch small form factor (SFF) 6 GB
Dual Ported serial-attached SCSI
(SAS) SSD
1.8-inch small form factor (SFF) 6 GB
Dual Ported serial-attached SCSI
(SAS) SSD
SSD drive feature codes supported ES02 ES02
Internal SSD drives Six minimum to 30 maximum Six minimum to 30 maximum
External HDD drives None 48 maximum
Figure 2. Dual storage IOA solution integrated within the 58/88 or ED/R1 PCIe storage enclosure.
Service information 3

Table 1. PCIe storage enclosure features (continued)
Feature code 58/88 ED/R1
External SAS enclosures None Two 58/87 disk drive enclosures
maximum
Drive redundant SAS paths Two paths from each enclosure RAID
module (ERM) (one path through
each SAS expander)
Two paths from each enclosure RAID
module (ERM) (one path through
each SAS expander)
Drive concurrent maintenance Yes Yes
Enclosure RAID Module (ERM) 3.1-GB cache RAID SAS enclosure 6
GB x8
3.1-GB cache RAID SAS enclosure 6
GB x8
ERM custom card identification
number (CCIN)
57C3 57C3
ERM RAID levels RAID 0, 5, 6, 10 with hot spare for 5,
6, 10
RAID 0, 5, 6, 10 with hot spare for 5,
6, 10
ERM write cache size 3.1 GB 3.1 GB
ERM external SAS links None Two mini SAS HD 4x connectors that
contain three physical links each
ERM concurrent maintenance No Yes
ERM fan assembly Two redundant fans per ERM Two redundant fans per ERM
ERM fan concurrent maintenance No Yes
Redundant power supplies Two Two
Power supply concurrent
maintenance
Yes Yes
Note: 1See PCI adapter information by feature type for the 31E/2B, 31E/1C, or 31E/2C
Cabling the PCIe storage enclosure
Use this information as a guide for cabling your storage enclosure.
Cabling the 58/88 PCIe storage enclosure to a 31E/1C or 31E/2C
system
Use this as a guide for cabling your PCIe storage enclosure to a 31E/1C or 31E/2C system.
Figure 3 on page 5 shows an example of connecting a 58/88 PCIe storage enclosure to a 31E/1C or
31E/2C system using two FC EJ0H GX adapters.
4Power systems: Service procedures for the PCIe storage enclosure

Figure 4 on page 6 shows one 58/88 PCIe storage enclosure connected to two 31E/1C or 31E/2C systems
using two FC EJ0H GX adapters.
Figure 3. One 58/88 PCIe storage enclosure connected to a 31E/1C or 31E/2C system using two FC EJ0H GX
adapters
Service information 5

Figure 5 shows two 58/88 PCIe storage enclosures connected to two 31E/1C or 31E/2C systems using
four FC EJ0H GX adapters.
Cabling the 58/88 PCIe storage enclosure to a 02E/4C or 05E/6C
system
Use this reference information as a guide for cabling your PCIe storage enclosure to a 02E/4C or 05E/6C
system.
Figure 6 on page 7 shows one 58/88 PCIe storage enclosure connected to a single 02E/4C or 05E/6C
system by using one FC EJ03 GX adapter.
Figure 4. One 58/88 PCIe storage enclosure connected to two 31E/1C or 31E/2C systems using two FC EJ0H GX
adapters
Figure 5. Two 58/88 PCIe storage enclosures connected to two 31E/1C or 31E/2C systems using four FC EJ0H GX
adapters
6Power systems: Service procedures for the PCIe storage enclosure

Figure 7 on page 8 shows one 58/88 PCIe storage enclosure connected to a 02E/4C or 05E/6C system by
using two FC EJ03 GX adapters.
Figure 6. One 58/88 PCIe storage enclosure connected to a single 02E/4C or 05E/6C system by using one FC EJ03
GX adapter
Service information 7

Figure 8 shows two 58/88 PCIe storage enclosures connected to a single 02E/4C or 05E/6C system by
using two FC EJ03 GX adapters.
Figure 9 on page 9 shows one 58/88 PCIe storage enclosure connected to two 02E/4C or 05E/6C systems
by using two FC EJ03 GX adapters.
Figure 7. One 58/88 PCIe storage enclosure connected to a single 02E/4C or 05E/6C system by using two FC EJ03
GX adapters
Figure 8. Two 58/88 PCIe storage enclosures connected to a single 02E/4C or 05E/6C system by using two FC EJ03
GX adapters.
8Power systems: Service procedures for the PCIe storage enclosure