IBM System Cluster 1350 Installation and maintenance instructions

IBM System Cluster 1350
Installation and Service Guide


IBM System Cluster 1350
Installation and Service Guide


Note: Before using this information and the product it supports, read the general information in “Safety” on page vand
Appendix C, “Notices,” on page 35.
Fourteenth edition, May 2007
©Copyright International Business Machines Corporation 2007. All rights reserved.
US Government Users Restricted Rights –Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.

Contents
Safety ....................................v
Safety Information .................................v
Handling static-sensitive devices ............................ xiii
Notices and statements used in this document ....................... xiv
Chapter 1. System overview ...........................1
Cluster components .................................4
Cluster nodes ..................................5
Management node ................................5
Storage nodes ..................................5
Storage servers .................................6
Storage expansion units...............................6
Fibre Channel adapters, switches, and expansion cards ....................6
Hard disk drives .................................6
Transceivers ..................................7
High-speed adapters................................7
Console ....................................7
KVM switches ..................................8
InfiniBand switches and devices ............................8
InfiniBand host channel adapters ...........................8
10/100/1000 Ethernet switches ............................8
10 Gigabit Ethernet switches .............................9
I/O modules ..................................9
Terminal servers .................................9
High-speed Myrinet switches and line cards ........................9
Myrinet BladeCenter connectivity ...........................11
Power distribution unit (PDU) ............................11
Related documentation ...............................11
Chapter 2. Cabinet placement and intercabinet cabling ...............13
Minimum service clearances ..............................13
Types of intercabinet cabling..............................15
Cabinet placement guidelines .............................15
Installing astability kit ...............................16
Engaging the side stabilizers..............................16
Floor loading ...................................16
Preparing aRear Door Heat eXchanger ..........................18
Chapter 3. Cabling the Cluster 1350 .......................19
Connecting the cables ................................20
1Gigabit Ethernet cabling .............................20
Myrinet high-speed switch cabling ..........................20
10/100/1000 Ethernet cabling ............................20
Fibre-channel cabling ...............................21
KVM cabling ..................................21
Remote console manager cabling ...........................21
Local console manager cabling ............................21
Voltaire InfiniBand switch router cabling ........................22
Replacing adefective cable in aharness ..........................22
Chapter 4. Checking the Cluster 1350 cabling process ...............23
Installing cables and checking connections in the expansion cabinets ................23
Installing cables and checking connections in the primary cabinet .................24
Turning on the power to the expansion cabinets .......................24
©Copyright IBM Corp. 2007 iii

Turning on the power to the primary cabinet ........................24
Verifying the hardware installation ...........................25
Diagnosing alights out or brown out event ........................25
Related topics ..................................26
Chapter 5. Accessing the cluster from aremote location ..............27
Using the remote power command ...........................27
Remote console ..................................27
Displaying node configuration information .........................27
Related topics ..................................28
Appendix A. Error and event logs ........................29
Appendix B. Getting help and technical assistance ................31
Before you call ..................................31
Using the documentation ...............................32
Getting help and information from the World Wide Web ....................32
Software service and support .............................32
Hardware service and support .............................32
IBM Taiwan product service ..............................33
Appendix C. Notices ..............................35
Trademarks ...................................35
Important notes ..................................36
Product recycling and disposal .............................37
Battery return program ...............................37
Electronic emission notices ..............................38
Federal Communications Commission (FCC) statement ....................38
Industry Canada Class Aemission compliance statement ...................39
Australia and New Zealand Class Astatement ......................39
United Kingdom telecommunications safety requirement ...................39
European Union EMC Directive conformance statement ...................39
Taiwanese Class Awarning statement .........................39
Chinese Class Awarning statement ..........................40
Japanese Voluntary Control Council for Interference (VCCI) statement...............40
Appendix D. Declaration of IBM product noise emission values ...........41
Appendix E. International License Agreement for Non-Warranted Programs ......43
Part 1-General Terms................................43
Part 2-Country-unique Terms .............................45
Index ....................................47
iv IBM System Cluster 1350 Installation and Service Guide

Safety
For general information concerning safety, refer to Electrical Safety for IBM Customer
Engineers,S229-8124. For acopy of the publication, contact your IBM®account
representative or the IBM branch office serving your locality.
Enterprise rack safety information: Read the safety notices in the manual
provided with the Enterprise rack before beginning work. Keep the Enterprise
Rack manual near the rack for fast reference.
Safety Information
Before installing this product, read the Safety Information.
Antes de instalar este produto, leia as Informações de Segurança.
Pred instalací tohoto produktu si prectete prírucku bezpecnostních instrukcí.
Læs sikkerhedsforskrifterne, før du installerer dette produkt.
Lees voordat udit product installeert eerst de veiligheidsvoorschriften.
Ennen kuin asennat tämän tuotteen, lue turvaohjeet kohdasta Safety Information.
Avant d’installer ce produit, lisez les consignes de sécurité.
Vor der Installation dieses Produkts die Sicherheitshinweise lesen.
Prima di installare questo prodotto, leggere le Informazioni sulla Sicurezza.
Les sikkerhetsinformasjonen (Safety Information) før du installerer dette produktet.
©Copyright IBM Corp. 2007 v

Antes de instalar este produto, leia as Informações sobre Segurança.
Antes de instalar este producto, lea la información de seguridad.
Läs säkerhetsinformationen innan du installerar den här produkten.
Important
Note:
All caution and danger statements in this documentation begin with a
number. This number is used to cross reference an English caution or danger
statement with translated versions of the caution or danger statement in the
IBM Safety Information book that comes with your server.
For example, if acaution statement begins with anumber 1, translations for
that caution statement appear in the IBM Safety Information book under
statement 1.
Be sure to read all caution and danger statements in this documentation
before performing the instructions. Read any additional safety information
that comes with your server or optional device before you install the device.
vi IBM System Cluster 1350 Installation and Service Guide

Statement 1:
DANGER
Electrical current from power, telephone, and communication cables is
hazardous.
To avoid ashock hazard:
vDo not connect or disconnect any cables or perform installation,
maintenance, or reconfiguration of this product during an electrical storm.
vConnect all power cords to aproperly wired and grounded electrical outlet.
vConnect to properly wired outlets any equipment that will be attached to
this product.
vWhen possible, use one hand only to connect or disconnect signal cables.
vNever turn on any equipment when there is evidence of fire, water, or
structural damage.
vDisconnect the attached power cords, telecommunications systems,
networks, and modems before you open the device covers, unless
instructed otherwise in the installation and configuration procedures.
vConnect and disconnect cables as described in the following table when
installing, moving, or opening covers on this product or attached devices.
To Connect: To Disconnect:
1. Turn everything OFF.
2. First, attach all cables to devices.
3. Attach signal cables to connectors.
4. Attach power cords to outlet.
5. Turn device ON.
1. Turn everything OFF.
2. First, remove power cords from outlet.
3. Remove signal cables from connectors.
4. Remove all cables from devices.
Statement 3:
CAUTION:
When laser products (such as CD-ROMs, DVD drives, fiber optic devices, or
transmitters) are installed, note the following:
vDo not remove the covers. Removing the covers of the laser product could
result in exposure to hazardous laser radiation. There are no serviceable parts
inside the device.
vUse of controls or adjustments or performance of procedures other than those
specified herein might result in hazardous radiation exposure.
Safety vii

DANGER
Some laser products contain an embedded Class 3A or Class 3B laser diode.
Note the following.
Laser radiation when open. Do not stare into the beam, do not view directly
with optical instruments, and avoid direct exposure to the beam.
Class 1 Laser Product
Laser Klasse 1
Laser Klass 1
Luokan 1 Laserlaite
Appareil A Laser de Classe 1
`
Statement 5:
CAUTION:
The power control button on the device and the power switch on the power
supply do not turn off the electrical current supplied to the device. The device
also might have more than one power cord. To remove all electrical current from
the device, ensure that all power cords are disconnected from the power source.
1
2
Statement 6:
CAUTION:
If you install astrain-relief bracket option over the end of the power cord that is
connected to the device, you must connect the other end of the power cord to an
easily accessible power source.
viii IBM System Cluster 1350 Installation and Service Guide

Statement 7:
CAUTION:
If the device has doors, be sure to remove or secure the doors before moving or
lifting the device to avoid personal injury. The doors will not support the
weight of the device.
Statement 8:
CAUTION:
Never remove the cover on apower supply or any part that has the following
label attached.
Hazardous voltage, current, and energy levels are present inside any component
that has this label attached. There are no serviceable parts inside these
components. If you suspect aproblem with one of these parts, contact aservice
technician.
Statement 9:
CAUTION:
To avoid personal injury, disconnect the hot-swap fan cables before removing
the fan from the device.
Safety ix

Statement 13:
DANGER
Overloading abranch circuit is potentially afire hazard and ashock hazard
under certain conditions. To avoid these hazards, ensure that your system
electrical requirements do not exceed branch circuit protection requirements.
Refer to the information that is provided with your device for electrical
specifications.
Statement 14:
CAUTION:
Hazardous voltage, current, and energy levels might be present. Only aqualified
service technician is authorized to remove the covers where the following label
is attached.
Statement 15:
CAUTION:
Make sure that the rack is secured properly to avoid tipping when the server
unit is extended.
Statement 16:
CAUTION:
Some accessory or option board outputs exceed Class 2or limited power source
limits and must be installed with appropriate interconnecting cabling in
accordance with the national electric code.
xIBM System Cluster 1350 Installation and Service Guide

Statement 19:
CAUTION:
The power-control button on the device does not turn off the electrical current
supplied to the device. The device also might have more than one connection to
dc power. To remove all electrical current from the device, ensure that all
connections to dc power are disconnected at the dc power input terminals.
OFF
Statement 20:
CAUTION:
To avoid personal injury, before lifting the unit, remove all the blades to reduce
the weight.
Statement 21:
CAUTION:
Hazardous energy is present when the blade is connected to the power source.
Always replace the blade cover before installing the blade.
Statement 23:
CAUTION:
Do not place any object weighing more than 50 kg (110 lb.) on top of
rack-mounted devices.
>50 kg (110 lb.)
Safety xi

CAUTION:
Removing components from the upper positions in the Enterprise Rack cabinet
improves rack stability during relocation. Follow these general guidelines
whenever you relocate apopulated rack cabinet within aroom or building:
vReduce the weight of the rack cabinet by removing equipment starting at the
top of the rack cabinet. When possible, restore the rack cabinet to the
configuration of the rack cabinet as you received it. If this configuration is not
known, you must do the following:
–Remove all devices in the 32-U position and above.
–Ensure that the heaviest devices are installed in the bottom of the rack
cabinet.
–Ensure that there are no empty U-levels between devices installed in the
rack cabinet below the 32-U level.
v
If the rack cabinet you are relocating is part of asuite of rack cabinets, detach
the rack cabinet from the suite.
vInspect the route that you plan to take to eliminate potential hazards.
vMake sure that the route that you choose can support the weight of the loaded
rack cabinet. Refer to the documentation that comes with your rack cabinet for
the weight of aloaded rack cabinet.
vMake sure that all door openings are at least 760 x2030 MM. (30 x80 in.)
vEnsure that all devices, shelves, drawers, doors, and cables are secure.
vEnsure that the four leveling pads are raised to their highest position.
vEnsure that there is no stabilizer bracket installed on the rack cabinet.
vDo not use aramp inclined at more than ten degrees.
vOnce the rack cabinet is in the new location, do the following:
–Lower the four leveling pads.
–Install stabilizer brackets on the rack cabinet.
–If you removed any devices from the rack cabinet, repopulate the rack
cabinet from the lowest position to the highest position.
If along distance relocation is required, restore the rack cabinet to the
configuration of the rack cabinet as you received it. Pack the rack cabinet in the
original packaging material, or equivalent. Also, lower the leveling pads to raise
the casters off of the pallet and bolt the rack cabinet to the pallet.
xii IBM System Cluster 1350 Installation and Service Guide

DANGER
v
Do not extend more than one sliding device at atime.
vThe maximum allowable weight for devices on slide rails is 80 kg
(176 lb.). Do not install sliding devices that exceed this weight.
vAlways lower the leveling pads on the rack cabinet.
vAlways install stabilizer brackets on the rack cabinet.
vAlways install servers and optional devices starting from the bottom of the
rack cabinet.
vAlways install the heaviest devices in the bottom of the rack cabinet.
Handling static-sensitive devices
Attention:Static electricity can damage electronic devices, including your server.
To avoid damage, keep static-sensitive devices in their static-protective packages
until you are ready to install them.
To reduce the possibility of damage from electrostatic discharge, observe the
following precautions:
vWhen you work on aBladeCenter unit that has an electrostatic discharge (ESD)
connector, use awrist strap when you handle modules, optional devices, or
blade servers. To work correctly, the wrist strap must have agood contact at
both ends (touching your skin at one end and firmly connected to the ESD
connector on the front or back of the BladeCenter unit).
vLimit your movement. Movement can cause static electricity to build up around
you.
vHandle the device carefully, holding it by its edges or its frame.
vDo not touch solder joints, pins, or exposed circuitry.
vDo not leave the device where others can handle and damage it.
vWhile the device is still in its static-protective package, touch it to any unpainted
metal surface of the BladeCenter chassis or any unpainted metal surface on any
other grounded component in the rack you are installing the device in for at
least 2seconds. (This drains static electricity from the package and from your
body.)
vRemove the device from its package and install it directly into the server
without setting down the device. If it is necessary to set down the device, put it
back into its static-protective package. Do not place the device on your server
cover or on ametal surface.
vTake additional care when handling devices during cold weather. Heating
reduces indoor humidity and increases static electricity.
Safety xiii

Notices and statements used in this document
The caution and danger statements that appear in this document are also in the
multilingual Safety Information document on the IBM BladeCenter Documentation CD.
Each statement is numbered for reference to the corresponding statement in the
Safety Information document.
The following notices and statements are used in the documentation:
vNote: These notices provide important tips, guidance, or advice.
vImportant: These notices provide information or advice that might help you
avoid inconvenient or problem situations.
vAttention: These notices indicate possible damage to programs, devices, or data.
An attention notice is placed just before the instruction or situation in which
damage could occur.
vCaution: These statements indicate situations that can be potentially hazardous
to you. Acaution statement is placed just before the description of apotentially
hazardous procedure step or situation.
vDanger: These statements indicate situations that can be potentially lethal or
extremely hazardous to you. Adanger statement is placed just before the
description of apotentially lethal or extremely hazardous procedure step or
situation.
xiv IBM System Cluster 1350 Installation and Service Guide

Chapter 1. System overview
This chapter provides an overview of the IBM®System Cluster 1350 software
components, hardware components, and related documentation.
The Cluster 1350 uses two types of cabinets: primary and expansion. Acabinet is
called primary if it contains the management node and console monitor. An
expansion cabinet can contain storage nodes or mass-storage devices as well as
computing nodes, called cluster nodes; it does not contain amanagement node or
aconsole.
The cluster may consists of:
vOne or more racks.
vFrom four to 1024 cluster nodes. The nodes are configured to run customer
applications or provide other services required by the customer, such as file
server, network gateway, or storage server.
vOne management node (System x3650 or System x3655) for cluster systems
management (CSM) and administration.
vAmanagement Ethernet VLAN that is used for secure traffic for hardware
control.
The management Ethernet VLAN is used for management traffic only. It is
logically isolated for security using the VLAN capability of the Cisco Ethernet
switches, and it is only accessible from the management node. The cluster VLAN
and management VLANs share the same physical switches.
vAcluster VLAN used for other management traffic and user traffic. Switches
integrated with the cluster are used for the management Ethernet VLAN and the
cluster Ethernet VLAN.
vService processor networks. All nodes in the cluster are connected through serial
service processors (such as the System x3650). The first node in aserial
connection must have amanagement node, which is connected through the
Ethernet to the management-Ethernet VLAN.
vAterminal server network for remote or local console. Optionally, the customer
might elect to include an additional network.
vAhigh-performance Myrinet 2000 cluster interconnect, an additional Gigabit
Ethernet, or an InfiniBand switch.
vThe customer can elect to configure asubset of cluster nodes with additional
external storage. This can be aFibre Channel solution.
vAsupported distribution of the Linux operating system.
vCluster systems-management software, such as CSM.
The Cluster 1350 supports amaximum of 1024 nodes in addition to the one
required System x3650 (or the System x3655 management node for 64-bit
processing environments) management node. All nodes must run one of the
following Linux versions:
vRed Hat Enterprise Linux (RHEL), version 4for AMD or EMT (32-bit or 64-bit)
vRHEL, version 4for AMD or EMT (64-bit)
vSUSE LINUX Enterprise Server (SLES), version 9for AMD or EMT (64-bit)
vSLES, version 10
vLinux Cluster Installation Tool (LCIT), version 5.0
©Copyright IBM Corp. 2007 1

The Cluster 1350 uses aprimary cabinet and an expansion cabinet. The primary
cabinet contains the management node and console monitor. An expansion cabinet
does not contain amanagement node or console. An expansion cabinet can contain
the following components:
vCluster or compute nodes
vStorage nodes
vMass storage devices
vExpansion units
vNetworking switches
The following figures illustrate aprimary cabinet, an expansion cabinet containing
cluster nodes, and an expansion cabinet containing storage controllers and mass
storage.
1 Gigabit Ethernet switch
KVM switch
Management
Node System x3650
Monitor
Gigabit Ethernet switch
Serial Terminal Servers
12
11
10
9
8
7
6
5
4
3
2
1
16
15
14
13
Cluster Nodes
System x3550
Storage
Node System x3650
Storage Expansion Unit
(DS 4000 EXP810)
Storage server
(DS4800)
20
19
18
17
Cluster Nodes
System x3550
1-U blank
1-U blank
1-U blank
1-U blank
1-U blank
1-U blank
21
2IBM System Cluster 1350 Installation and Service Guide

The following figure shows how the node numbering scheme maps to other
expansion cabinets.
Serial Terminal Server
1-U blank
1-U blank
14
13
12
11
10
9
8
7
6
5
4
3
32
31
30
29
28
27
26
25
24
23
22
21
20
19
1
2
18
17
16
15
Cluster
Nodes
System x3550
Cabinet 1
56
55
54
53
52
51
50
49
48
47
73
72
71
70
69
68
67
66
39
40
41
42
43
44
45
46
Cabinet 2
88
87
86
85
84
83
82
81
80
79
78
77
111
110
109
108
107
106
105
104
103
102
101
100
99
98
97
96
91
90
89
Cabinet 3
240
239
238
237
236
235
234
233
232
231
230
229
255
254
253
252
251
250
249
248
244
243
242
241
Cabinet 7
92
33
34
36
37
38
35
93
94
245
246
76
75
74
65
64
63
62
61
60
59
58
57 95
112
113
114
247
256
blank
blank
blank
blank
blank
blank
blank
blank
blank
blank
blank
............
Cluster
nodes
System x3550
Chapter 1. System overview 3

The following figure shows an expansion cabinet containing storage controllers
and mass storage.
24
26
23
12
11
10
9
8
7
6
5
4
3
2
1
16
15
14
13
Storage nodes
System x3650
36
30
29
28
27
Serial Terminal Servers
32
33
34
31
Storage servers
(DS4800)
37
38
39
40
41
42
Storage servers
(DS4800)
Storage Expansion Units
(DS4000 EXP810)
Gigabit Ethernet switch
1-U blank
1-U blank
1-U blank
1-U blank
35
25
Cluster components
This section describes the components in the Cluster 1350. Cluster components
include:
vCluster nodes
vManagement node
vStorage nodes
vStorage servers
vStorage expansion units
vFibre Channel adapters, switches, and expansion cards
vHard disk drives
4IBM System Cluster 1350 Installation and Service Guide