IBM System Cluster 1350 Manual

eServer Cluster 1350
Cluster 1350 Installation and Service
IBM

eServer Cluster 1350
Cluster 1350 Installation and Service
IBM

Note!
Before using this information and the product it supports, please review the information “Safety and environmental
notices” on page xi.
June 2003 edition
This book is the current version of IBM eServer Cluster 1350 installation and service documentation.
IBM welcomes your comments. You may address your comments to the following address:
International Business Machines Corporation
Department 55JA, Mail Station P384
2455 South Road
Poughkeepsie, NY 12601-5400
United States of America
FAX (United States and Canada): 1+845+432-9405
FAX (Other Countries):
(Your international Access Code)+1+845+432+9405
IBMLink (United States customers only): IBMUSM10(MHVRCFS)
Internet e-mail: [email protected]
If you would like a reply, be sure to include your name, address, telephone number, or FAX number.
Make sure to include the following in your comment or note:
vTitle of this book
vPage number or topic related to your comment
When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any
way it believes appropriate without incurring any obligation to you.
© Copyright International Business Machines Corporation 2003. All rights reserved.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.

Contents
Figures ..............vii
Tables ...............ix
Safety and environmental notices . . . xi
Safety notices ..............xi
Safety Information ...........xi
Environmental notices...........xv
Product recycling and disposal .......xv
Part 1. Introduction to Cluster 1350 1
Chapter 1. System overview ......3
Related Topics ..............9
Part 2. Installation .........11
Chapter 2. Unpacking the eServer
Cluster 1350 ............13
Chapter 3. Placing the cabinets ....15
Customer responsibilities ..........15
Installer responsibilities ..........15
Cabinet placement ...........15
Chapter 4. Cabling..........17
Overview of Cluster 1350 cabling .......17
VLAN options ............17
General information ...........24
Attaching the cables ...........24
1-GB Ethernet cabling ...........24
High-speed (Myrinet) switch cabling ......25
10/100/1000 Ethernet cabling ........26
Fibre Channel cabling ...........27
KVM cabling ..............28
Remote Console Manager (RCM) cabling ....29
Replacing a defective cable in a harness .....29
Chapter 5. Power up the cluster ....31
Initial Cluster 1350 power-on procedure .....31
Checking connections in the Expansion cabinets 31
Checking connections in the Primary cabinet . . 31
Power-on the Expansion cabinets ......32
Power-on the Primary cabinet .......32
Lights out or brown out ..........33
Related topics .............33
Chapter 6. Software installation ....35
Software Version Matrix ..........35
Download link for drivers and firmware . . . 36
Install a supported distribution of Linux.....36
Installing Red Hat Linux 7.3 or 8 ......37
Installing SuSE Linux 8 or 8.1 .......39
Installing SuSE Linux Enterprise Server (SLES) 7
or8................42
Install Cluster Systems Management (CSM) . . . 44
Pre-installation tasks ..........44
Installing CSM on the Management Node . . . 46
Configure the Storage nodes.........46
Prerequisites .............46
Issues ...............47
Copy the system image out to all nodes in the
cluster ................48
Test the configuration ...........48
Chapter 7. Cluster management ....51
Chapter 8. Remote access ......53
Remote power .............53
Remote console .............53
Chapter 9. Cluster power down ....55
Power down the system ..........55
Lights out or brown out ..........55
Related topics .............55
Part 3. Service...........57
Chapter 10. Hardware/software problem
determination............59
How to use this information.........5960
Isolating network, node, and Linux problems . . 60
Isolating hardware problems .......63
Isolating software problems ........69
SNMP monitoring ...........71
Setting up SNMP alerts from Myrinet ....71
Resetting RSA cards ..........72
Checking service processor logs ......72
Problem determination references ......72
Chapter 11. Management, Cluster and
Storage Nodes ...........75
Disk drive failure on the Management Node . . . 75
Disk drive failure on a Cluster Node ......75
Disk drive failure on a Storage Node ......76
Systemboard failures ...........76
Other Cluster Node Problems ........76
x335 problems ............76
BladeCenter problems ..........77
Additional Information ..........77
Chapter 12. Power problems .....79
No power to multiple devices ........79
No power to an individual device ......79
© Copyright IBM Corp. 2003 iii

Chapter 13. KVM Switch replacement
and configuration ..........81
Replacement of NetBAY 2x8 console switch....81
Uncable the console switch ........81
Remove the rails from the cabinet ......81
Remove the console switch from the rails . . . 81
Install the console switch and rails into the
cabinet ...............81
Configure and setup the console switch after device
replacement ..............84
Upgrading the console switch FLASH level. . . 84
Replacement of NetBAY Advanced Connectivity
Technology RCM ............85
Chapter 14. KVM control .......87
Saving the KVM Switch settings .......87
Connecting components with KVM Switch power
on.................87
Switching between nodes and the console ....87
Security features.............88
Resetting the mouse and keyboard ......88
Chapter 15. Port server replacement
and configuration ..........89
Replacement ..............89
Uncable the port server .........89
Remove the tray from the cabinet ......89
Remove the port server from the tray.....89
Install the port server and rails .......89
Configuration and setup after device replacement 90
Chapter 16. Cisco 10/100 Switch
replacement and configuration ....93
Replacement of the 24-port switch .......93
Remove the rails from the cabinet ......93
Remove the Cisco 24-port 10/100 Switch from
the rails ..............93
Install the Cisco 24-port 10/100 Switch and rails 93
Replacement of the 48-port switch .......96
Remove the rails from the cabinet ......96
Remove the Cisco 48-Port 10/100 Switch from
the rails ..............96
Install the Cisco 48-Port 10/100 Switch and rails 96
Configuration and setup after device replacement 99
Setup troubleshooting .........100
Additional information ..........100
Chapter 17. Cisco Gigabit Switch
replacement and configuration ....101
Installation procedure ..........101
Remove the rails from the cabinet .....101
Remove the Cisco Gigabit Switch from the rails 101
Install the Cisco Gigabit Switch and rails into
the cabinet .............101
Configure and setup after device replacement . . 104
Setup troubleshooting .........105
Additional information ..........105
Chapter 18. Cisco 4000 Series switch
replacement ............107
Installation, removal, replacement, and
troubleshooting procedures.........107
Additional information ..........107
Chapter 19. Myrinet 2000 ......109
Myrinet PCI board ...........109
Myrinet switch chassis ..........109
Configure and setup after device replacement . . 110
Additional information ..........110
Chapter 20. Power Management
Module replacement and configuration 111
Replacement..............111
Uncable the Power Management Module . . . 111
Remove the tray from the cabinet......111
Remove the Power Management Module from
the tray ..............112
Install the Power Management Module and tray
into the cabinet ............113
Configure and setup after device replacement . . 117
Installation of power bricks for RSAs on tray . . . 117
Related topics .............118
Chapter 21. Power Distribution Unit
removal and replacement ......119
Removal and replacement .........119
Part 4. Appendixes ........121
Appendix A. Frequently Asked
Questions .............123
Appendix B. Error Logs .......125
Appendix C. Known problems ....127
Node ................127
Amber light on node ..........127
COM port settings in BIOS ........127
CSM................127
Stale NFS mounts ...........127
rpower hard shut down .........127
Storage ...............127
Driver module ordering .........127
KVM................128
GUI does not appear on first node .....128
2x8 Switch powers on with console port B. . . 128
Cluster port 1 reboots .........128
Subsequent KVMs unresponsive ......129
RSA and Service Processor .........129
RSA unable to load firmware .......129
RSA/Service Processor invalid naming ....129
Light path points to PCI LED .......129
Myrinet ...............129
Myrinet communication fails .......129
iv Installation and Service

Appendix D. Setting up network
switches .............131
General networking notes .........131
Switch commands ...........131
Appendix E. International License
Agreement for Non-Warranted
Programs .............137
Part 1 - General Terms ..........137
Part 2 - Country-unique Terms .......139
License Information ...........140
Program: Embedded Software from Cisco
Systems, Inc. ............140
Notices ..............143
Trademarks ..............143
Electronic emissions notices ........144
Federal Communications Commission (FCC)
statement..............144
Industry Canada Class A emission compliance
statement..............144
Australia and New Zealand Class A statement 144
United Kingdom telecommunications safety
requirement .............144
European Union EMC Directive conformance
statement..............144
Taiwanese Class A warning statement ....145
Chinese Class A warning statement .....145
Japanese Voluntary Control Council for
Interference (VCCI) statement .......145
Regulatory and Compliance Requirements....146
Telecom ..............146
Safety Compliance...........146
Batteries ..............147
Environmental Statement ........147
Index ...............149
Contents v

vi Installation and Service

Figures
1. Example of an Eserver Cluster 1350 Primary
cabinet ..............4
2. Example of an Eserver Cluster 1350 Expansion
Cabinet with Cluster Nodes .......5
3. Example of an Eserver Cluster 1350 Expansion
cabinet containing storage controllers and mass
storage ..............6
4. Frame stabilizer (tilt) foot installation ....16
5. Intercabinet cabling for the 1-GB Ethernet
connections (VLAN types 1, 3 and 4) ....25
6. Intercabinet cabling for Myrinet connections
(VLAN types 4 and 6) .........26
7. Intercabinet cabling for Gbit Ethernet
connections (VLAN types 2,3,5 and 6)....27
8. Intercabinet cabling for Fibre Channel
connections. ............28
9. Intercabinet cabling for KVM connections 29
10. Mount switch rails to the console switch 82
11. Install switch rails into cabinet from the rear of
the cabinet .............83
12. Mount switch rails to the Cisco 24-port 10/100
Switch ..............94
13. Install switch rails from the rear of the cabinet 95
14. Mount switch rails to the Cisco 48-port 10/100
Switch ..............97
15. Install switch rails into cabinet from the rear of
the cabinet .............98
16. Mount switch rails to the Cisco Gigabit
Switch ..............102
17. Install switch rails into cabinet from the rear
of the cabinet ...........103
18. Thermal duct used with Cisco 4000 switches 107
19. Retract sliding bracket at the back of the tray
(front of the cabinet) .........112
20. Mount Power Management Module onto
Power Management Module tray .....113
21. Mount power bricks onto Power Management
Module tray ............114
22. Install Power Management Module tray into
cabinet from the rear of the cabinet, part 1 . . 115
23. Install Power Management Module tray into
cabinet from the rear of the cabinet, part 2 . . 116
24. Mount power bricks onto Power Management
Module tray ............118
© Copyright IBM Corp. 2003 vii

viii Installation and Service

Tables
1. Type 1 VLAN. 10/100 Ethernet ......18
2. Type 2 VLAN.10/100/1000 Ethernet ....18
3. Type 3 VLAN. 10/100 Ethernet with
10/100/1000 public high speed VLAN . . . 19
4. Type 4 VLAN. 10/100/1000 Ethernet with 2
Gbit public high speed VLAN ......19
5. Type 5 VLAN. 10/100/1000 Ethernet with
10/100/1000 Ethernet public high speed
VLAN ..............20
6. Type 6 VLAN.10/100/1000 Ethernet with 2
Gbit Myrinet public high speed VLAN . . . 21
7. Type 2 VLAN with multiple Cisco 400x
switches..............22
8. Type 5 VLAN with multiple Cisco 400x
switches..............22
9. Type 6 VLAN with multiple Cisco 400x
switches..............23
10. Cluster 1350 supported software and firmware
versions - June 2003..........35
11. Troubleshooting the shared VLAN .....60
12. Network troubleshooting for a cluster with one
network..............61
13. ................62
14. Network troubleshooting for a cluster with
two networks ............63
15. Troubleshooting the remote console network 63
16. Troubleshooting the service processor network 65
17. Troubleshooting the Fibre Storage network 67
18. Troubleshooting the terminal server network
for the Remote Console ........68
19. Troubleshooting the KVM network.....69
20. IBM components used for Management,
Cluster, and Storage Nodes .......75
21. Myrinet Chassis Sizes and Capacities 109
© Copyright IBM Corp. 2003 ix

xInstallation and Service

Safety and environmental notices
For general information concerning safety, refer to Electrical Safety for IBM Customer
Engineers, S229-8124. For a copy of the publication, contact your IBM account
representative or the IBM branch office serving your locality.
Enterprise Rack Safety Information: Read the safety notices in the manual
provided with the Enterprise Rack before beginning work. Keep the Enterprise
Rack Manual near the rack for fast reference.
Safety notices
The procedures described in this document must be performed by qualified service
personnel. Safety warnings are contained within these procedures. If you cannot
read the language of this document, do not perform any procedures until you
receive a translated copy. IBM does not accept responsibility or liability for failure
to follow these procedures correctly.
Safety Information
Before installing this product, read the Safety Information.
Antes de instalar este produto, leia as Informações de Segurança.
Pred instalací tohoto produktu si prectete prírucku bezpecnostních instrukcí.
Læs sikkerhedsforskrifterne, før du installerer dette produkt.
Lees voordat u dit product installeert eerst de veiligheidsvoorschriften.
Ennen kuin asennat tämän tuotteen, lue turvaohjeet kohdasta Safety Information.
Avant d’installer ce produit, lisez les consignes de sécurité.
Vor der Installation dieses Produkts die Sicherheitshinweise lesen.
Prima di installare questo prodotto, leggere le Informazioni sulla Sicurezza.
© Copyright IBM Corp. 2003 xi

Les sikkerhetsinformasjonen (Safety Information) før du installerer dette produktet.
Antes de instalar este produto, leia as Informações sobre Segurança.
Antes de instalar este producto, lea la información de seguridad.
Läs säkerhetsinformationen innan du installerar den här produkten.
Important:
All caution and danger statements in this documentation begin with a
number. This number is used to cross reference an English caution or
danger statement with translated versions of the caution or danger
statement in the IBM NetBAY Rack Safety Information book.
For example, if a caution statement begins with a number 1,
translations for that caution statement appear in the IBM NetBAY Rack
Safety Information book under statement 1.
Be sure to read all caution and danger statements in this
documentation before performing the instructions. Read any additional
safety information that comes with your server or optional device
before you install the device.
Statement 2:
DANGER
vAlways lower the leveling pads on the rack cabinet.
vAlways install stabilizer brackets on the rack cabinet.
vAlways install servers and optional devices starting from the bottom of the
rack cabinet.
vAlways install the heaviest devices in the bottom of the rack cabinet.
Statement 3:
xii Installation and Service

DANGER
vDo not extend more than one sliding device at a time.
vThe maximum allowable weight for devices on slide rails is 80 kg
(176 lb). Do not install sliding devices that exceed this weight.
Statement 4:
DANGER
Electrical current from power, telephone, and communication cables is
hazardous.
To avoid a shock hazard:
vDo not connect or disconnect any cables or perform installation,
maintenance, or reconfiguration of this product during an electrical storm.
vConnect all power cords to a properly wired and grounded electrical outlet.
vConnect to properly wired outlets any equipment that will be attached to
this product.
vWhen possible, use one hand only to connect or disconnect signal cables.
vNever turn on any equipment when there is evidence of fire, water, or
structural damage.
vDisconnect the attached power cords, telecommunications systems,
networks, and modems before you open the device covers, unless
instructed otherwise in the installation and configuration procedures.
vConnect and disconnect cables as described in the following table when
installing, moving, or opening covers on this product or attached devices.
To Connect: To Disconnect:
1. Turn everything OFF.
2. First, attach all cables to devices.
3. Attach signal cables to connectors.
4. Attach power cords to outlet.
5. Turn device ON.
1. Turn everything OFF.
2. First, remove power cords from outlet.
3. Remove signal cables from connectors.
4. Remove all cables from devices.
Statement 7:
Safety and environmental notices xiii

CAUTION:
The power control button on the device and the power switch on the power
supply do not turn off the electrical current supplied to the device. The device
also might have more than one power cord. To remove all electrical current from
the device, ensure that all power cords are disconnected from the power source.
1
2
Statement 8:
DANGER
vPlug power cords from devices in the rack cabinet into electrical outlets
that are located near the rack cabinet and are easily accessible.
vEach rack cabinet might have more than one power cord. Be sure to
disconnect all power cords in the rack cabinet before servicing any device
in the rack cabinet.
vInstall an emergency-power-off switch if more than one power device
(power distribution unit or uninterruptible power supply) is installed in
the same rack cabinet.
vConnect all devices installed in a rack cabinet to power devices installed in
the same rack cabinet. Do not plug a power cord from a device installed in
one rack cabinet into a power device installed in a different rack cabinet.
Statement 10:
xiv Installation and Service

CAUTION:
Removing components from the upper positions in the Enterprise Rack cabinet
improves rack stability during relocation. Follow these general guidelines
whenever you relocate a populated rack cabinet within a room or building:
vReduce the weight of the rack cabinet by removing equipment starting at the
top of the rack cabinet. When possible, restore the rack cabinet to the
configuration of the rack cabinet as you received it. If this configuration is not
known, you must do the following:
– Remove all devices in the 32U position and above.
– Ensure that the heaviest devices are installed in the bottom of the rack
cabinet.
– Ensure that there are no empty U-levels between devices installed in the
rack cabinet below the 32U level.
vIf the rack cabinet you are relocating is part of a suite of rack cabinets, detach
the rack cabinet from the suite.
vInspect the route that you plan to take to eliminate potential hazards.
vVerify that the route that you choose can support the weight of the loaded
rack cabinet. Refer to the documentation that comes with your rack cabinet for
the weight of a loaded rack cabinet.
vVerify that all door openings are at least 760 x 2030 mm (30 x 80 in.)
vEnsure that all devices, shelves, drawers, doors, and cables are secure.
vEnsure that the four leveling pads are raised to their highest position.
vEnsure that there is no stabilizer bracket installed on the rack cabinet.
vDo not use a ramp inclined at more than ten degrees.
vOnce the rack cabinet is in the new location, do the following:
– Lower the four leveling pads.
– Install stabilizer brackets on the rack cabinet.
– If you removed any devices from the rack cabinet, repopulate the rack
cabinet from the lowest position to the highest position.
If a long distance relocation is required, restore the rack cabinet to the
configuration of the rack cabinet as you received it. Pack the rack cabinet in the
original packaging material, or equivalent. Also, lower the leveling pads to raise
the casters off of the pallet and bolt the rack cabinet to the pallet.
The following is a list of safety notices (in English only) pertaining to hardware
maintenance tasks:
DANGER notices warn you of conditions or procedures that can result in death
or severe personal injury.
CAUTION notices warn you of conditions or procedures that can cause
personal injury that is neither lethal nor extremely hazardous.
ATTENTION emphasizes certain explanatory information and calls attention to
statements in text, figures, and tables. Attention notices also highlight situations
that could potentially cause damage to the equipment or loss of data.
Environmental notices
Product recycling and disposal
This product contains materials such as circuit boards, cables, electromagnetic
compatibility gaskets, and connectors which might contain lead and
Safety and environmental notices xv

copper/beryllium alloys that require special handling and disposal at end of life.
Before this unit is disposed of, these materials must be removed and recycled or
discarded according to applicable regulations. IBM offers product return programs
in several countries. You can find country-specific instructions at
www.ibm.com/ibm/environment/products/prp.phtml.
This product might contain nickel-cadmium or lithium batteries in communication
adapters. The batteries must be recycled or disposed of properly. Recycling
facilities might not be available in your area. In the United States, IBM has
established a collection process for reuse, recycling, or proper disposal of used
sealed lead-acid, nickel-cadmium and nickel metal hydride batteries and battery
packs from IBM equipment. For information on proper disposal of batteries in this
product, please contact IBM at 1-800-426-4333. For information on disposal of
batteries outside the United States, contact your local waste disposal or recycling
facility.
xvi Installation and Service

Part 1. Introduction to Cluster 1350
Chapter 1. System overview ........3
Related Topics ..............9
© Copyright IBM Corp. 2003 1

2Installation and Service