Bull Escala M6-700 User manual

ESCALA Power7
High performance clustering
REFERENCE
86 A1 93FF 03

ESCALA Power7
High performance clustering
The ESCALA Power7 publications concern the following models:
- Bull Escala E5-700 (Power 750 / 8233-E8B)
- Bull Escala M6-700 (Power 770 / 9117-MMB)
- Bull Escala M6-705 (Power 770 / 9117-MMC)
- Bull Escala M7-700 (Power 780 / 9179-MHB)
- Bull Escala M7-705 (Power 780 / 9179-MHC)
- Bull Escala E1-700 (Power 710 / 8231-E2B)
- Bull Escala E1-705 (Power 710 / 8231-E1C)
- Bull Escala E2-700 / E2-700T (Power 720 / 8202-E4B)
- Bull Escala E2-705 / E2-705T (Power 720 / 8202-E4C)
- Bull Escala E3-700 (Power 730 / 8231-E2B)
- Bull Escala E3-705 (Power 730 / 8231-E2C)
- Bull Escala E4-700 / E4-700T (Power 740 / 8205-E6B)
- Bull Escala E4-705 (Power 740 / 8205-E6C)
References to Power 755 / 8236-E8C models are irrelevant.
Hardware
October 2011
BULL CEDOC
357 AVENUE PATTON
B.P.20845
49008 ANGERS CEDEX 01
FRANCE
REFERENCE
86 A1 93FF 03

The following copyright notice protects this book under Copyright laws which prohibit such actions as, but not limited to, copying,
distributing, modifying, and making derivative works.
Bull SAS 2011
Copyright
Printed in France
Suggestions and criticisms concerning the form, content, and presentation of this book are
invited. A form is provided at the end of this book for this purpose.
To order additional copies of this book or other Bull Technical Publications, you are invited
to use the Ordering Form also provided at the end of this book.
Trademarks and Acknowledgements
We acknowledge the right of proprietors of trademarks mentioned in this book.
The information in this document is subject to change without notice. Bull will not be liable for errors contained herein, or
r incidental or consequential damages in connection with the use of this material.
fo

Contents
Safety notices .................................ix
High-performance computing clusters using InfiniBand hardware...........1
Clustering systems by using InfiniBand hardware .......................2
Cluster information resources.............................2
Fabric communications ...............................6
IBM GX+ or GX++ host channel adapter ........................7
Logical switch naming convention .........................9
Host channel adapter statistics counter .......................10
Vendor and IBM switches ............................10
QLogic switches supported by IBM .........................10
Cables ...................................10
Subnet Manager ................................11
POWER Hypervisor ..............................12
Device drivers ................................12
IBM host stack ................................12
Management subsystem function overview ........................13
Management subsystem integration recommendations ...................13
Management subsystem high-level functions ......................14
Management subsystem overview ..........................15
xCAT..................................17
Fabric manager ...............................17
Hardware Management Console .........................18
Switch chassis viewer .............................19
Switch command-line interface ..........................19
Server Operating system ............................20
Network Time Protocol ............................20
Fast Fabric Toolset ..............................20
Flexible Service processor............................21
Fabric viewer................................21
Email notifications ..............................22
Management subsystem networks .........................22
Vendor log flow to xCAT event management ......................23
Supported components in an HPC cluster ........................24
Cluster planning..................................26
Cluster planning overview .............................27
Required level of support, firmware, and devices......................28
Server planning .................................29
Server types .................................29
Planning InfiniBand network cabling and configuration ...................30
Topology planning ...............................30
Example configurations using only 9125-F2A servers ..................33
Example configurations: 9125-F2A compute servers and 8203-E4Astorage servers .........43
Configurations with IO router servers .......................47
Cable planning ................................48
Planning QLogic or IBM Machine Type InfiniBand switch configuration .............49
Planning maximum transfer unit (MTU).......................51
Planning for global identifier prefixes........................52
Planning an IBM GX HCA configuration .......................53
IP subnet addressing restriction with RSCT......................53
Management subsystem planning ...........................54
Planning your Systems Management application .....................55
Planning xCAT as your Systems Management application .................55
Planning for QLogic fabric management applications ...................56
Planning the fabric manager and fabric Viewer ....................56
© Copyright IBM Corp. 2011 iii

Planning Fast Fabric Toolset ...........................63
Planning for fabric management server ........................64
Planning event monitoring with QLogic and management server ...............66
Planning event monitoring with xCAT on the cluster management server ...........66
Planning to run remote commands with QLogic from the management server ...........67
Planning to run remote commands with QLogic from xCAT/MS ..............67
Frame planning .................................68
Planning installation flow .............................68
Key installation points..............................68
Installation responsibilities by organization .......................68
Installation responsibilities of units and devices .....................69
Order of installation ..............................70
Installation coordination worksheet .........................73
Planning for an HPC MPI configuration .........................74
Planning 12x HCA connections ............................75
Planning aids..................................75
Planning checklist ................................75
Planning worksheets ...............................76
Cluster summary worksheet ............................77
Frame and rack planning worksheet .........................79
Server planning worksheet ............................81
QLogic and IBM switch planning worksheets ......................83
Planning worksheet for 24-port switches.......................84
Planning worksheet for switches with more than 24 ports .................85
xCAT planning worksheets ............................89
QLogic fabric management worksheets ........................92
Installing a high-performance computing (HPC) cluster with an InfiniBand network ...........96
IBM Service representative installation responsibilities ....................97
Cluster expansion or partial installation .........................97
Site setup for power, cooling, and floor .........................98
Installing and configuring the management subsystem ....................98
Installing and configuring the management subsystem for a cluster expansion or addition ......101
Installing and configuring service VLAN devices ....................102
Installing the Hardware Management Console .....................102
Installing the xCAT management server .......................104
Installing operating system installation servers .....................105
Installing the fabric management server .......................105
Set up remote logging .............................112
Remote syslogging to an xCAT/MS ........................112
Using syslog on RedHat Linux-based xCAT/MS ...................120
Set up remote command processing .........................120
Setting up remote command processing from the xCAT/MS................120
Installing and configuring servers with management consoles ................122
Installing and configuring the cluster server hardware....................123
Server installation and configuration information for expansion ...............123
Server hardware installation and configuration procedure .................124
Installing the operating system and configuring the cluster servers ...............127
Installing the operating system and configuring the cluster servers information for expansion .....127
Installing the operating system and configuring the cluster servers ..............128
Installation sub procedure for AIX only.......................134
RedHat rpms required for InfiniBand .......................135
Installing and configuring vendor or IBM InfiniBand switches .................137
Installing and configuring InfiniBand switches when adding or expanding an existing cluster .....137
Installing and configuring the InfiniBand switch.....................138
Attaching cables to the InfiniBand network .......................143
Cabling the InfiniBand network information for expansion .................144
InfiniBand network cabling procedure ........................144
Verifying the InfiniBand network topology and operation ..................145
Installing or replacing an InfiniBand GX host channel adapter .................147
Deferring replacement of a failing host channel adapter ..................149
Verifying the installed InfiniBand network (fabric) in AIX ..................150
iv Power Systems: High performance clustering

Fabric verification ................................150
Fabric verification responsibilities .........................150
Reference documentation for fabric verification procedures .................150
Fabric verification tasks .............................150
Fabric verification procedure ...........................151
Runtime errors .................................151
Cluster Fabric Management .............................152
Cluster fabric management flow ...........................152
Cluster Fabric Management components and their use ...................152
xCAT Systems Management ...........................152
QLogic subnet manager .............................153
QLogic fast fabric toolset ............................154
QLogic performance manager ...........................155
Managing the fabric management server .......................155
Cluster fabric management tasks ...........................155
Monitoring the fabric for problems ..........................156
Monitoring fabric logs from the xCAT Cluster Management server ..............156
Health checking ...............................157
Setting up periodic fabric health checking ......................158
Output files for health check ..........................164
Interpreting health check .changes files .......................167
Interpreting health check .diff files ........................172
Querying status ...............................174
Remotely accessing QLogic management tools and commands from xCAT/MS ...........174
Remotely accessing the Fabric Management Server from xCAT/MS ..............175
Remotely accessing QLogic switches from the xCAT/MS ..................175
Updating code .................................176
Updating Fabric Manager code ..........................176
Updating switch chassis code ...........................179
Finding and interpreting configuration changes ......................180
Hints on using iba_report .............................180
Cluster service ..................................183
Service responsibilities ..............................183
Fault reporting mechanisms ............................183
Fault diagnosis approach .............................185
Types of events................................185
Isolating link problems .............................186
Restarting or repowering on scenarios ........................187
The importance of NTP .............................187
Table of symptoms ...............................187
Service procedures ...............................191
Capturing data for fabric diagnosis ..........................193
Using script command to capture switch CLI output ...................196
Capture data for Fabric Manager and Fast Fabric problems ..................196
Mapping fabric devices ..............................197
General mapping of IBM HCA GUIDs to physical HCAs ..................197
Finding devices based on a known logical switch ....................199
Finding devices based on a known logical HCA.....................201
Finding devices based on a known physical switch port ..................203
Finding devices based on a known ib interface (ibX/ehcaX) .................205
IBM GX HCA Physical port mapping based on device number .................207
Interpreting switch vendor log formats .........................207
Log severities ................................207
Switch chassis management log format ........................208
Subnet Manager log format............................209
Diagnosing link errors ..............................210
Diagnosing and repairing switch component problems ...................213
Diagnosing and repairing IBM system problems......................213
Diagnosing configuration changes ..........................213
Checking for hardware problems affecting the fabric ....................214
Checking for fabric configuration and functional problems ..................214
Contents v

Checking InfiniBand configuration in AIX ........................215
Checking system configuration in AIX .........................217
Verifying the availability of processor resources .....................217
Verifying the availability of memory resources .....................217
Checking InfiniBand configuration in Linux .......................218
Checking system configuration in Linux ........................220
Verifying the availability of processor resources .....................220
Verifying the availability of memory resources .....................220
Checking multicast groups .............................221
Diagnosing swapped HCA ports ...........................221
Diagnosing swapped switch ports ..........................222
Diagnosing events reported by the operating system ....................223
Diagnosing performance problems ..........................224
Diagnosing and recovering ping problems........................225
Diagnosing application crashes ...........................226
Diagnosing management subsystem problems ......................226
Problem with event management or remote syslogging ..................226
Event not in xCAT/MS:/tmp/systemEvents .....................227
Event not in xCAT/MS: /var/log/xcat/syslog.fabric.notices................228
Event not in xCAT/MS: /var/log/xcat/syslog.fabric.info.................230
Event not in log on fabric management server ....................231
Event not in switch log ............................232
Reconfiguring xCAT event management .......................232
Reconfiguring xCAT on the AIX operating system ...................232
Reconfiguring xCAT on the Linux operating system ..................233
Recovering from an HCA preventing a logical partition from activating ..............235
Recovering ibX interfaces .............................235
Recovering a single ibX interface in AIX .......................235
Recovering all of the ibX interfaces in an LPAR in the AIX .................236
Recovering an ibX interface tcp_sendspace and tcp_recvspace ................237
Recovering ml0 in AIX .............................237
Recovering icm in AIX .............................237
Recovering ehcaX interfaces in Linux .........................237
Recovering a single ibX interface in Linux .......................237
Recovering all of the ibX interfaces in an LPAR in the Linux ................238
Recovering to 4K maximum transfer units in the AIX ....................238
Recovering to 4K maximum transfer units in the Linux ...................241
Recovering the original master SM ..........................243
Re-establishing Health Check baseline .........................244
Verifying link FRU replacements ...........................244
Verifying repairs and configuration changes .......................245
Restarting the cluster ...............................246
Restarting or powering off an IBM system........................247
Counting devices ................................248
Counting switches...............................248
Counting logical switches ............................249
Counting host channel adapters ..........................249
Counting end ports ..............................249
Counting ports ................................249
Counting Subnet Managers............................250
Counting devices example ............................250
Handling emergency power off situations ........................251
Monitoring and checking for fabric problems.......................252
Retraining 9125-F2A links .............................252
How to retrain 9125-F2A links...........................252
When to retrain 9125-F2A links ..........................254
Error counters ..................................254
Interpreting error counters .............................255
Interpreting link Integrity errors ..........................256
Interpreting remote errors ............................260
Example PortXmitDiscard analyses ........................261
vi Power Systems: High performance clustering

Example PortRcvRemotePhysicalErrors analyses....................262
Interpreting security errors ............................264
Diagnose a link problem based on error counters .....................264
Error counter details ...............................265
Categorizing Error Counters ...........................265
Link Integrity Errors ..............................266
LinkDownedCounter .............................266
LinkErrorRecoveryCounter ...........................266
LocalLinkIntegrityErrors............................267
ExcessiveBufferOverrunErrors ..........................267
PortRcvErrors ...............................268
SymbolErrorCounter .............................269
Remote Link Errors (including congestion and link integrity) ................271
PortRcvRemotePhysicalErrors ..........................271
PortXmitDiscards ..............................271
Security errors ................................273
PortXmitConstraintErrors ...........................273
PortRcvConstraintErrors............................273
Other error counters ..............................273
VL15Dropped ...............................273
PortRcvSwitchRelayErrors ...........................274
Clearing error counters ..............................274
Example health check scripts .............................275
Configuration script ...............................276
Error counter clearing script ............................276
Healthcheck control script .............................277
Cron setup on the Fabric MS ............................279
Improved healthcheck ..............................279
Notices ...................................283
Trademarks ...................................284
Electronic emission notices ..............................285
Class A Notices.................................285
Terms and conditions................................288
Contents vii

viii Power Systems: High performance clustering