IBM eserver Cluster 1350 Installation and maintenance instructions

IBM Eserver Cluster 1350
Installation and Service Guide
ERserver


IBM Eserver Cluster 1350
Installation and Service Guide
ER s e r v e r


Note: Before using this information and the product it supports, read the general information in “Safety”
on page vii and Appendix G, “Notices,” on page 93.
Fourth edition, February 2004
© Copyright International Business Machines Corporation 2004. All rights reserved.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.

Contents
Safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Safety Information . . . . . . . . . . . . . . . . . . . . . . . . vii
Notices and statements used in this document . . . . . . . . . . . . . xiii
Chapter 1. System overview . . . . . . . . . . . . . . . . . . . .1
Cluster components . . . . . . . . . . . . . . . . . . . . . . .4
Cluster nodes . . . . . . . . . . . . . . . . . . . . . . . . .5
Management node . . . . . . . . . . . . . . . . . . . . . . .5
Storage nodes . . . . . . . . . . . . . . . . . . . . . . . .5
Storage servers . . . . . . . . . . . . . . . . . . . . . . . .6
Storage expansion units . . . . . . . . . . . . . . . . . . . . .6
SCSI/RAID storage controller adapters . . . . . . . . . . . . . . .6
Console . . . . . . . . . . . . . . . . . . . . . . . . . . .6
KVM switch . . . . . . . . . . . . . . . . . . . . . . . . .6
10/100 Mb Ethernet switch . . . . . . . . . . . . . . . . . . . .6
1 Gb Ethernet switch . . . . . . . . . . . . . . . . . . . . . .7
SMC mini-GBIC expansion module . . . . . . . . . . . . . . . . .7
Terminal server . . . . . . . . . . . . . . . . . . . . . . . .7
High-speed Myrinet switch . . . . . . . . . . . . . . . . . . . .7
Myrinet cluster expansion card . . . . . . . . . . . . . . . . . .7
High-speed Cisco switch . . . . . . . . . . . . . . . . . . . . .7
Power management module . . . . . . . . . . . . . . . . . . .7
Power distribution unit (PDU) . . . . . . . . . . . . . . . . . . .7
Related publications . . . . . . . . . . . . . . . . . . . . . . .8
Chapter 2. Unpacking the Cluster 1350 . . . . . . . . . . . . . . .9
Chapter 3. Placing the cabinets . . . . . . . . . . . . . . . . . .11
Customer responsibilities . . . . . . . . . . . . . . . . . . . . .11
Installer responsibilities . . . . . . . . . . . . . . . . . . . . . .11
Chapter 4. Cabling the Cluster 1350 . . . . . . . . . . . . . . . .13
VLAN options . . . . . . . . . . . . . . . . . . . . . . . . .14
Connecting the cables . . . . . . . . . . . . . . . . . . . . . .20
1 Gb Ethernet cabling . . . . . . . . . . . . . . . . . . . . .20
High-speed (Myrinet) switch cabling . . . . . . . . . . . . . . . .20
10/100/1000 Ethernet cabling . . . . . . . . . . . . . . . . . .21
Fibre Channel cabling . . . . . . . . . . . . . . . . . . . . .21
KVM cabling . . . . . . . . . . . . . . . . . . . . . . . . .21
Remote Console Manager cabling . . . . . . . . . . . . . . . . .21
Replacing a defective cable in a harness . . . . . . . . . . . . . . .22
Chapter 5. Checking the Cluster 1350 cabling process . . . . . . . . .23
Checking connections in the expansion cabinets . . . . . . . . . . . .23
Checking connections in the primary cabinet . . . . . . . . . . . . . .23
Turning on the power to the expansion cabinets . . . . . . . . . . . . .24
Turning on the power to the primary cabinet . . . . . . . . . . . . . .24
Verifying the installation of the Linux Cluster Installation Tool . . . . . . . .25
Lights out or brown out event . . . . . . . . . . . . . . . . . . .26
Related topics . . . . . . . . . . . . . . . . . . . . . . . . .26
Chapter 6. Installing the software . . . . . . . . . . . . . . . . .27
Software version matrix . . . . . . . . . . . . . . . . . . . . . .27
© Copyright IBM Corp. 2004 iii

Downloading device drivers and firmware . . . . . . . . . . . . . . .29
Installing a supported version of Linux . . . . . . . . . . . . . . . .29
Installing the Cluster System Management software . . . . . . . . . . .29
Installing the General Parallel File System system-management software . . .29
Configuring the storage nodes . . . . . . . . . . . . . . . . . . .30
Installing the storage node software . . . . . . . . . . . . . . . .30
Defining nodes . . . . . . . . . . . . . . . . . . . . . . . .31
Distributing the system image to all nodes in the cluster . . . . . . . . .33
Verifying the configuration . . . . . . . . . . . . . . . . . . . . .33
Chapter 7. Administering the Cluster 1350 . . . . . . . . . . . . . .35
Using the remote power command . . . . . . . . . . . . . . . . .35
Remote console . . . . . . . . . . . . . . . . . . . . . . . .35
Shutting down the system components . . . . . . . . . . . . . . . .35
Finding information about a lights out or brown out event . . . . . . . . .36
Related topics . . . . . . . . . . . . . . . . . . . . . . . . .36
Chapter 8. Troubleshooting hardware and software problems . . . . . .37
Isolating network, node, and Linux problems . . . . . . . . . . . . . .38
Clustering with one network . . . . . . . . . . . . . . . . . . .39
Clustering with two networks . . . . . . . . . . . . . . . . . . .39
Isolating hardware problems . . . . . . . . . . . . . . . . . . . .42
Node checks . . . . . . . . . . . . . . . . . . . . . . . . .42
Service processor network . . . . . . . . . . . . . . . . . . .44
Checking storage . . . . . . . . . . . . . . . . . . . . . . .45
Checking the terminal server . . . . . . . . . . . . . . . . . . .46
Troubleshooting the KVM network . . . . . . . . . . . . . . . . .47
File-system failure . . . . . . . . . . . . . . . . . . . . . . .47
PFA alert indicates internal disk . . . . . . . . . . . . . . . . . .47
I/O errors in syslog . . . . . . . . . . . . . . . . . . . . . .47
Isolating software problems . . . . . . . . . . . . . . . . . . . .48
Operating-system checks . . . . . . . . . . . . . . . . . . . .48
CSM checks . . . . . . . . . . . . . . . . . . . . . . . . .48
GPFS checks . . . . . . . . . . . . . . . . . . . . . . . .49
SNMP monitoring . . . . . . . . . . . . . . . . . . . . . . . .49
Configuring SNMP alerts from Myrinet . . . . . . . . . . . . . . . .49
Resetting the Remote Supervisor Adapter card . . . . . . . . . . . . .50
Checking service processor logs . . . . . . . . . . . . . . . . . .50
Management, cluster, and storage node problems . . . . . . . . . . . .50
Disk drive failures . . . . . . . . . . . . . . . . . . . . . . .51
System board failures . . . . . . . . . . . . . . . . . . . . .52
xSeries 335 problems . . . . . . . . . . . . . . . . . . . . .52
BladeCenter problems . . . . . . . . . . . . . . . . . . . . .52
Power problems . . . . . . . . . . . . . . . . . . . . . . .53
Related publications . . . . . . . . . . . . . . . . . . . . . . .54
Chapter 9. Configuring the KVM switch . . . . . . . . . . . . . . .55
Configuring the console switch after device replacement . . . . . . . . .55
Upgrading the console switch FLASH level . . . . . . . . . . . . . .55
Replacement of NetBAY Advanced Connectivity Technology Remote Console
Manager . . . . . . . . . . . . . . . . . . . . . . . . . .56
Chapter 10. Using the KVM switch . . . . . . . . . . . . . . . . .57
Saving the KVM switch settings . . . . . . . . . . . . . . . . . . .57
Connecting components with the KVM switch power turned on . . . . . . .57
Switching between nodes and the console . . . . . . . . . . . . . . .57
iv IBM Eserver Cluster 1350 Installation and Service Guide

Security features . . . . . . . . . . . . . . . . . . . . . . . .58
Resetting the mouse and keyboard . . . . . . . . . . . . . . . . .58
Chapter 11. Configuring the terminal server after device replacement . . .59
Chapter 12. Configuring and replacing the Cisco 10/100 switch . . . . .61
Configuration and setup after device replacement . . . . . . . . . . . .61
Setup troubleshooting . . . . . . . . . . . . . . . . . . . . . .62
Additional information . . . . . . . . . . . . . . . . . . . . . .62
Chapter 13. Configuring and replacing the Cisco Gigabit switch . . . . .63
Replacement procedure . . . . . . . . . . . . . . . . . . . . .63
Configuring after device replacement . . . . . . . . . . . . . . . . .63
Setup troubleshooting . . . . . . . . . . . . . . . . . . . . . .64
Additional information . . . . . . . . . . . . . . . . . . . . . .64
Chapter 14. Replacing the Cisco 4000 series switch . . . . . . . . . .65
Installation, removal, replacement, and troubleshooting procedures . . . . . .65
Additional information . . . . . . . . . . . . . . . . . . . . . .65
Chapter 15. Replacing the Myrinet 2000 Gigabit switch . . . . . . . . .67
Myrinet PCI board . . . . . . . . . . . . . . . . . . . . . . . .67
Myrinet switch chassis . . . . . . . . . . . . . . . . . . . . . .67
Configure and setup after device replacement . . . . . . . . . . . . .68
Additional information . . . . . . . . . . . . . . . . . . . . . .68
Chapter 16. Configuring and replacing the Power Management Module 69
Replacing the Power Management Module . . . . . . . . . . . . . .69
Configuring after device replacement . . . . . . . . . . . . . . . . .69
Related topics . . . . . . . . . . . . . . . . . . . . . . . . .69
Chapter 17. Removing and replacing the Power Distribution Unit . . . . .71
Appendix A. Getting help and technical assistance . . . . . . . . . .73
Before you call . . . . . . . . . . . . . . . . . . . . . . . . .73
Hardware service and support . . . . . . . . . . . . . . . . . . .73
Appendix B. Frequently asked questions . . . . . . . . . . . . . .75
Appendix C. Error and event logs . . . . . . . . . . . . . . . . .77
Appendix D. Known problems . . . . . . . . . . . . . . . . . .79
Node . . . . . . . . . . . . . . . . . . . . . . . . . . . .79
Amber light on node . . . . . . . . . . . . . . . . . . . . . .79
COM port settings in BIOS . . . . . . . . . . . . . . . . . . .79
CSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79
Stale NFS mounts . . . . . . . . . . . . . . . . . . . . . . .79
rpower hard shut down . . . . . . . . . . . . . . . . . . . . .79
Storage . . . . . . . . . . . . . . . . . . . . . . . . . . .79
Driver module ordering . . . . . . . . . . . . . . . . . . . . .79
KVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80
GUI does not appear on first node . . . . . . . . . . . . . . . . .80
2x8 Switch powers on with console port B . . . . . . . . . . . . . .80
Cluster port 1 reboots . . . . . . . . . . . . . . . . . . . . .80
Subsequent KVMs unresponsive . . . . . . . . . . . . . . . . .80
RSA and Service Processor . . . . . . . . . . . . . . . . . . . .81
Contents v

RSA unable to load firmware . . . . . . . . . . . . . . . . . . .81
RSA/Service Processor invalid naming . . . . . . . . . . . . . . .81
Light path points to PCI LED . . . . . . . . . . . . . . . . . . .81
Myrinet communication fails . . . . . . . . . . . . . . . . . . . .81
Appendix E. Configuring network switches . . . . . . . . . . . . .83
General networking notes . . . . . . . . . . . . . . . . . . . . .83
Switch commands . . . . . . . . . . . . . . . . . . . . . . . .83
Switch commands for the Cisco Catalyst Ethernet 3508/3550 switch running
IOS . . . . . . . . . . . . . . . . . . . . . . . . . . .83
Switch commands for the Cisco Gigabit 4006 switch running IOS . . . . .84
Switch commands for the Cisco Gigabit 4006 switch running Cisco Catalyst
Operating System . . . . . . . . . . . . . . . . . . . . . .85
Miscellaneous CISCO switch commands for the Cisco Catalyst Operating
System . . . . . . . . . . . . . . . . . . . . . . . . . .86
Miscellaneous Cisco switch commands for IOS . . . . . . . . . . . .87
Appendix F. International License Agreement for Non-Warranted Programs 89
Part 1 - General Terms . . . . . . . . . . . . . . . . . . . . . .89
Part 2 - Country-unique Terms . . . . . . . . . . . . . . . . . . .91
License Information . . . . . . . . . . . . . . . . . . . . . . .92
Appendix G. Notices . . . . . . . . . . . . . . . . . . . . . .93
Edition notice . . . . . . . . . . . . . . . . . . . . . . . . .93
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . .94
Important notes . . . . . . . . . . . . . . . . . . . . . . . . .94
Product recycling and disposal . . . . . . . . . . . . . . . . . . .95
Battery return program . . . . . . . . . . . . . . . . . . . . . .95
Electronic emission notices . . . . . . . . . . . . . . . . . . . .96
Federal Communications Commission (FCC) statement . . . . . . . . .96
Industry Canada Class A emission compliance statement . . . . . . . .96
Australia and New Zealand Class A statement . . . . . . . . . . . .96
United Kingdom telecommunications safety requirement . . . . . . . . .96
European Union EMC Directive conformance statement . . . . . . . . .96
Taiwanese Class A warning statement . . . . . . . . . . . . . . .97
Chinese Class A warning statement . . . . . . . . . . . . . . . .97
Japanese Voluntary Control Council for Interference (VCCI) statement . . .97
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . .99
vi IBM Eserver Cluster 1350 Installation and Service Guide

Safety
For general information concerning safety, refer to Electrical Safety for IBM
Customer Engineers, S229-8124. For a copy of the publication, contact your IBM
®
account representative or the IBM branch office serving your locality.
Enterprise rack safety information: Read the safety notices in the manual
provided with the enterprise rack before beginning work. Keep the Enterprise Rack
manual near the rack for fast reference.
The procedures described in this document must be performed by qualified service
personnel. Safety warnings are contained within these procedures. If you cannot
read the language of this document, do not perform any procedures until you
receive a translated copy. IBM does not accept responsibility or liability for failure to
follow these procedures correctly.
Safety Information
Before installing this product, read the Safety Information.
Antes de instalar este produto, leia as Informações de Segurança.
Pred instalací tohoto produktu si prectete prírucku bezpecnostních instrukcí.
Læs sikkerhedsforskrifterne, før du installerer dette produkt.
Lees voordat u dit product installeert eerst de veiligheidsvoorschriften.
Ennen kuin asennat tämän tuotteen, lue turvaohjeet kohdasta Safety Information.
Avant d’installer ce produit, lisez les consignes de sécurité.
Vor der Installation dieses Produkts die Sicherheitshinweise lesen.
Prima di installare questo prodotto, leggere le Informazioni sulla Sicurezza.
© Copyright IBM Corp. 2004 vii

Les sikkerhetsinformasjonen (Safety Information) før du installerer dette produktet.
Antes de instalar este produto, leia as Informações sobre Segurança.
Antes de instalar este producto, lea la información de seguridad.
Läs säkerhetsinformationen innan du installerar den här produkten.
Important:
All caution and danger statements in this documentation begin with a
number. This number is used to cross reference an English caution or
danger statement with translated versions of the caution or danger
statement in the IBM NetBAY Rack Safety Information book.
For example, if a caution statement begins with a number 1,
translations for that caution statement appear in the IBM NetBAY Rack
Safety Information book under statement 1.
Be sure to read all caution and danger statements in this
documentation before performing the instructions. Read any additional
safety information that comes with your server or optional device before
you install the device.
Statement 2:
DANGER
v
Always lower the leveling pads on the rack cabinet.
v Always install stabilizer brackets on the rack cabinet.
v Always install servers and optional devices starting from the bottom of
the rack cabinet.
v Always install the heaviest devices in the bottom of the rack cabinet.
viii IBM Eserver Cluster 1350 Installation and Service Guide