IBM RS/6000 SP User manual

SG24-4778-00
RS/6000 SP: Problem Determination Guide
December 1996
This soft copy for use by IBM employees only.

International Technical Support Organization
RS/6000 SP: Problem Determination Guide
December 1996
SG24-4778-00
IBML
This soft copy for use by IBM employees only.

This soft copy for use by IBM employees only.
Take Note!
Before using this information and the product it supports, be sure to read the general information in
Appendix D, “Special Notices” on page 281.
First Edition (December 1996)
This edition applies to Version 2, Release 1 of POWERparallel System Support Programs for use with the AIX 4.1.4
Comments may be addressed to:
IBM Corporation, International Technical Support Organization
Dept. HYJ Mail Station P099
522 South Road
Poughkeepsie, New York 12601-5400
When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any
way it believes appropriate without incurring any obligation to you.
Copyright International Business Machines Corporation 1996. All rights reserved.
Note to U.S. Government Users — Documentation related to restricted rights — Use, duplication or disclosure is
subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.

This soft copy for use by IBM employees only.
Contents
Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
How This Redbook Is Organized ........................... xv
The Team That Wrote This Redbook ........................ xvii
Comments Welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii
Chapter 1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 RS/6000 SP: Hardware and Software ...................... 1
1.1.1 The Control Workstation ........................... 1
1.1.2 The Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.3 The High Performance Switch ........................ 3
1.2 Problem Determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Chapter 2. The Installation Process ......................... 7
2.1 Major Steps in Installing and Configuring RS/6000 SP Software ...... 7
2.2 Prepare the Control Workstation ......................... 8
2.2.1 PSSP Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.2 AIX Software Components .......................... 10
2.2.3 Disk Space Requirements .......................... 11
2.2.4 RS-232 Control Lines Diagnostics ..................... 11
2.2.5 Changing IP Addresses and Hostnames ................. 12
2.2.6 Maximum Number of Processes ...................... 14
2.2.7 Number of Licensed Users ......................... 14
2.2.8 Tunable Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.9 /spdata Directory Structure ......................... 15
2.2.10 Install PSSP on the Control Workstation ................. 16
2.2.11 PSSP 1.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.12 PSSP 2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.13 Changes from PSSP 1.2 to PSSP 2.1 ................... 19
2.2.14 PSSP Software Strategy .......................... 20
2.2.15 Obtaining PSSP PTFs Using FixDist .................... 21
2.2.16 Determining Software Service Level ................... 22
2.2.17 Working with PSSP PTFs .......................... 25
2.3 Authentication Services Diagnostics ...................... 28
2.4 install_cw Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.1 Problems Solving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.5 Problems with spmon ............................... 31
2.5.1 Problems with System Monitor Commands ................ 32
2.5.2 Problems Accessing Pulldown Menus of spmon ............. 33
2.5.3 System Monitor Displays Blanks Instead of the Expected Information 33
2.5.4 Problems with System Monitor GUI .................... 36
2.5.5 Logging Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.5.6 Example — Some Nodes Are Missing ................... 37
2.5.7 Example — spmon Will Not Start ...................... 38
2.6 Problems with Some PSSP Scripts ....................... 40
2.6.1 setup_authent Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.6.2 install_cw Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.7 Restore Control Workstation mksysb ...................... 43
Copyright IBM Corp. 1996 iii

This soft copy for use by IBM employees only.
2.8 NIM Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.9 NIM Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.9.1 NIM Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.9.2 Setup_server Fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.9.3 Debugging NIM Installations ......................... 60
2.10 Network Booting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
2.10.1 Scripts Involved in Network Boot ..................... 62
2.10.2 Node Installation Diagnostics ....................... 66
2.11 Node Customization Problems ......................... 69
2.11.1 Why a Node Is Not Being Customized .................. 69
Chapter 3. Kerberos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.1.1 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.1.2 Authorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.1.3 Distributed Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.1.4 Remote Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.1.5 .rhosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2.1 Principal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2.2 Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2.3 Realm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.2.4 Ticket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.2.5 Key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.2.6 Ticket-Granting Ticket . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.3 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.3.1 ssp.authent 2.1.0.2 ............................... 74
3.3.2 ssp.clients 2.1.0.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.4 Install Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.4.1 setup_authent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.4.2 install_cw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.4.3 setup_server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.4.4 Network Boot .................................. 75
3.5 Daemons and Databases ............................. 75
3.5.1 Kerberos Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.5.2 kadmind Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.5.3 kpropd Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.5.4 /etc/services File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.5.5 Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.6 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.6.1 /.k . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.6.2 $HOME/.klogin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.6.3 /tmp/tkt<uid> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.6.4 /etc/krb-srvtab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.6.5 /etc/krb.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.6.6 /etc/krb.realms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.6.7 /var/adm/SPlogs/kerberos . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.7 Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.7.1 kinit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.7.2 klist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.7.3 kdestroy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.7.4 kstash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.7.5 dsh and p* Commands ............................ 82
3.7.6 rsh and rcp ................................... 82
3.7.7 sysctl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
iv SP PD Guide

This soft copy for use by IBM employees only.
3.8 Solving Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.8.1 Daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.8.2 Tickets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.8.3 PATH Variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.8.4 Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.8.5 TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.8.6 Remote Principals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.8.7 Service Key Files ............................... 84
3.8.8 PTF Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.8.9 Rebuild the Kerberos Database ...................... 85
Chapter 4. The Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.1.1 Software Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.1.2 High Performance Switch and SP Switch Coexistence ......... 91
4.1.3 Comparing HiPS and SP Switch ...................... 91
4.1.4 Improvements of SP Switch over HiPS .................. 92
4.2 Reviewing Switch Boards ............................. 93
4.3 Switch Topology and Clock Subsystem ..................... 95
4.3.1 Switch Topology Files ............................ 95
4.3.2 Topology File Nomenclature ......................... 99
4.3.3 HiPS Clock Subsystem ........................... 100
4.3.4 SP Switch Clock Subsystem ........................ 101
4.3.5 Switch Clock Files .............................. 102
4.4 Node Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.4.1 Primary Node Behavior .......................... 103
4.4.2 Primary Backup Node Behavior ..................... 103
4.4.3 Secondary Node Behavior ......................... 104
4.4.4 Recovery from a Switch Failure ..................... 104
4.5 Switch Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.5.1 The rc.switch Script ............................. 108
4.5.2 Switch Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4.6 Reviewing Switch Processes .......................... 110
4.6.1 Switch Responds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.7 Switch Log Files ................................. 113
4.7.1 The out.top Log File ............................. 116
4.7.2 Patterns in out.top File ........................... 117
4.7.3 The flt Log File ................................ 119
4.7.4 Reading the flt File ............................. 120
4.7.5 Fault Syndrome on the flt File ....................... 123
4.8 Additional Problem Determination ....................... 124
4.8.1 Debugging Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
4.8.2 AIX Error Logging .............................. 124
Chapter 5. System Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.1 Scope, Rules, and Limitations ......................... 125
5.1.1 Scope of System Partitioning ....................... 125
5.1.2 Some Rules for Partitioning ........................ 126
5.1.3 Some Limitations of System Partitioning ................ 127
5.2 Partitioning and the High Performance Switch ............... 127
5.2.1 Partitioning a Single Switch/Frame System .............. 128
5.2.2 Partitioning a Two Switch/Frame System ................ 130
5.2.3 Partitioning a Three or More Switch/Frame System ......... 130
5.2.4 Switch Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
5.2.5 Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Contents v

This soft copy for use by IBM employees only.
5.3 Creating Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.3.1 Prerequisites to System Partitioning ................... 137
5.3.2 Process Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.3.3 Archiving the SDR .............................. 140
5.3.4 Customizing the Partitions ......................... 141
5.3.5 Verify the Configuration .......................... 142
5.3.6 Applying the System Partitioning ..................... 144
5.3.7 Validating the Partitions and Restoring the SDR ............ 144
5.4 SDR Reorganization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5.4.1 SDR Daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5.4.2 SDR Directory Structure .......................... 147
5.4.3 New Object Classes ............................ 149
5.4.4 SDR Locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
5.4.5 The Restructured SDR ........................... 150
5.5 Heartbeat Reorganization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
5.5.1 The Heartbeat before System Partitioning ............... 152
5.5.2 The Heartbeat after System Partitioning ................ 154
5.5.3 Daemons and Scripts ............................ 156
5.6 Resource Manager Reorganization ...................... 157
Chapter 6. Error Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
6.1 Error Logging Overview ............................. 159
6.1.1 Terms Used by the Error Logging Facility ................ 160
6.1.2 Error Logging Commands ......................... 161
6.1.3 Error Log Files ................................ 161
6.2 SP Error Logging ................................. 162
6.2.1 Install and Configure Error Log ...................... 163
6.2.2 AIX Error Log Facility ............................ 164
6.2.3 BSD syslog Facility ............................. 165
6.2.4 Trimming syslog Files ........................... 168
6.2.5 Maintain Error Logs ............................. 170
6.2.6 Collecting System Data .......................... 173
6.2.7 SP Error Log ................................. 174
6.3 Error Notification Facility ............................ 175
6.3.1 Error Notification Objects ......................... 178
6.3.2 Error Notification Object Class ...................... 180
6.3.3 Adding an Error Notification Object ................... 181
6.3.4 Mailing Error Reports to the Control Workstation ........... 183
6.3.5 Notification on Boot Device ........................ 185
6.3.6 Notification Power Loss and PANIC ................... 186
6.4 Error Daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
6.4.1 AIX Error Daemons ............................. 187
6.4.2 SP Log Daemons .............................. 191
Chapter 7. Isolating Problems on the SP System ................ 195
7.1 Isolating Booting Problems ........................... 195
7.1.1 Booting Process Overview ......................... 195
7.1.2 Booting Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
7.2 Isolating System Monitor Problems ...................... 197
7.3 Isolating Switch Problems ............................ 199
7.4 Isolating System Partitioning Problems .................... 200
Chapter 8. Producing a System Dump ....................... 201
8.1 Handling Systems Dumps ............................ 201
8.1.1 How to Start a Dump ............................ 204
vi SP PD Guide

This soft copy for use by IBM employees only.
8.1.2 Copying a System Dump .......................... 206
8.1.3 Sending the Dump to the Support Center ................ 208
8.1.4 Using crash to Analyze a Dump ..................... 210
Chapter 9. User and Services Management ................... 213
9.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
9.1.1 User Accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
9.1.2 File Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
9.1.3 Auto Mount Daemon (Amd) ........................ 213
9.1.4 Print Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
9.1.5 Network Time Protocol (NTP) ....................... 213
9.2 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
9.2.1 ssp.basic 2.1.0.10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
9.2.2 ssp.sysman 2.1.0.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
9.3 Managing User Accounts ............................ 214
9.3.1 Adding an SP User ............................. 216
9.3.2 Changing User Attributes ......................... 217
9.3.3 Login Control ................................. 218
9.4 Managing File Collections ............................ 218
9.4.1 Using File Collections ........................... 219
9.4.2 supper Hints and Tips ........................... 219
9.5 Managing Amd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
9.5.1 Using Amd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
9.5.2 Amd Hints and Tips ............................. 222
9.6 Managing Print Services ............................ 225
9.6.1 Using Print Services ............................ 225
9.7 Managing NTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
9.7.1 Using NTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
9.7.2 NTP Hints and Tips ............................. 226
Appendix A. RS/6000 SP Script Files ....................... 229
A.1 The setup_authent Script ............................ 229
A.2 The install_cw Script .............................. 236
A.3 The setup_server Script ............................. 239
A.4 The rc.switch Script ............................... 262
Appendix B. The SDR Structure .......................... 271
Appendix C. IP Address and Hostname Changes for the SP ......... 273
C.1 SDR Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
C.2 RS/6000 SP System Files ............................ 273
C.3 Procedures Used When Changing IP Addresses/Hostnames ....... 275
C.4 Updating the RS/6000 SP Node Interfaces .................. 275
C.5 Control Workstation IP Address/Hostname Changes ........... 276
C.6 Execution on RS/6000 SP Nodes ........................ 279
Appendix D. Special Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
Appendix E. Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . 283
E.1 International Technical Support Organization Publications ........ 283
E.2 Redbooks on CD-ROMs ............................. 283
E.3 Other Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
How To Get ITSO Redbooks ............................. 285
How IBM Employees Can Get ITSO Redbooks .................. 285
Contents vii

This soft copy for use by IBM employees only.
How Customers Can Get ITSO Redbooks ..................... 286
IBM Redbook Order Form .............................. 287
List of Abbreviations ................................. 289
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
viii SP PD Guide

This soft copy for use by IBM employees only.
Figures
1. Control Workstation Connected to the SP Frame . . . . . . . . . . . . . . 2
2. Supervisor Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. The High Performance Switch . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. SP Error Log Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
5. Sample of a Portion of .profile File . . . . . . . . . . . . . . . . . . . . . . . 9
6. Minimum Required BOS and Runtime Environment Components . . . . 10
7. Minimum Required Components of Network Install Manager (NIM) . . . 10
8. Minimum Required Components of Base Networking Software . . . . . 11
9. Output of lsvg rootvg Command to Check Disk Space . . . . . . . . . . . 11
10. /etc/hosts File on the Control Workstation .................. 13
11. Output of a Successful Ping ........................... 14
12. /spdata Directory Structure ........................... 15
13. Obtaining PSSP PTFs Using FixDist ...................... 21
14. Output of Running install_cw Script ...................... 30
15. Portion of /etc/inittab That Gets Added from Running install_cw ..... 30
16. Portion of /etc/services That Gets Added from Running install_cw ... 30
17. Snapshot of System Monitor .......................... 32
18. The hardmon Daemon Is Not Running .................... 39
19. The sdr Daemon Is Not Running ........................ 39
20. The hmacls File .................................. 40
21. /etc/inittab File after Running install_cw Script ............... 43
22. /etc/services File after Running install_cw Script .............. 43
23. Output from setup_server ............................ 68
24. Extract from the /etc/services File ....................... 76
25. Example Using netstat Command ....................... 76
26. Example of /etc/krb-srvtab from the Control Workstation ......... 78
27. Example of /etc/krb-srvtab from a Node ................... 78
28. Example of a /etc/krb.conf File ......................... 78
29. Example of a /etc/krb.realms File ....................... 79
30. Example of kerberos.log File .......................... 80
31. Example of admin_server.syslog File ..................... 80
32. Example of klist Output ............................. 81
33. Example of klist -srvtab Output ......................... 81
34. inittab Entries for Kerberos Daemons ..................... 82
35. Example of PATH Statement from .profile File ................ 83
36. Sample /etc/krb.conf File ............................ 83
37. Sample /etc/krb.realms File ........................... 83
38. Example of Host Name Resolution ....................... 84
39. Example of klist -srvtab Command on the Control Workstation ...... 84
40. How to Rebuild the Kerberos Database .................... 85
41. The HiPS Showing the External Connections ................ 88
42. Example of the Cabling for a 128-Way System ................ 89
43. Example of the Cabling on a 48-Way System ................ 90
44. The HiPS Board .................................. 93
45. The SP Switch Board ............................... 95
46. Topology File Sample .............................. 97
47. Topology File - Example ............................. 98
48. Topology File Nomenclature .......................... 99
49. The HiPS Clock Subsystem .......................... 100
50. The HiPS Chip Clock Tree ........................... 101
51. The High Performance Switch Board .................... 102
Copyright IBM Corp. 1996 ix

This soft copy for use by IBM employees only.
52. Diagram of the Different Phases on the High Performance Switch ... 107
53. The High Performance Switch Board .................... 110
54. Switch Responds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
55. The flt File .................................... 121
56. Fault Syndrome on flt File ........................... 123
57. Message in Error Report while Estart Has Been Executed ....... 124
58. Topology File Insert from a Single Switch 16 Wide Node System ... 129
59. Example of an 8_8 Configuration ....................... 131
60. Insert of Topology File for the First 8-Way Partition ............ 132
61. Insert of Topology File for the Second Partition .............. 133
62. Directory Structure for the Topology Files in PSSP 2.1 .......... 135
63. SMIT Menu for the Fastpath ″syspar″.................... 139
64. Flow Chart of the System Partitioning Process .............. 140
65. Example of SMIT Screen for Customizing Layouts ............ 141
66. Example of SMIT Screen for Verifying the Configuration ......... 143
67. Data Organization of the SDR in PSSP 2.1 ................. 148
68. PSSP 2.1 SDR Directory Structure ...................... 149
69. Example of the Heartbeat Subsystem after System Partitioning .... 155
70. Error Logging Components .......................... 160
71. /etc/syslog.conf file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
72. /etc/syslog.pid file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
73. /spdata/sys1/amd.tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
74. /spdata/sys1/err_methods/EN_pend . . . . . . . . . . . . . . . . . . . . . 177
75. /spdata/sys1/err_methods/EN_pend.envs . . . . . . . . . . . . . . . . . . 178
76. Booting Process Overview .......................... 195
77. Booting Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
78. System Monitor Problems ........................... 197
79. Switch Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
80. System Partitioning Problems ........................ 200
81. Copying a System Dump ........................... 206
82. Using crash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
83. Default Values for the Site Environment Information Panel ....... 215
84. Output from splstdata -e ............................ 216
85. Example of chsh Command on a Node ................... 217
86. Extract from /etc/security/user File ..................... 218
87. Example of Login by Blocked User ID .................... 218
88. Output from lssrc -s supfilesrv Command ................. 219
89. supper Error Messages When PSSP Code Is Mismatched ........ 219
90. Extract from /var/adm/SPlogs/filec/sup5.13.96.12.02r Log File ..... 220
91. supper Errors When Bypassing PTF Set 8 ................. 221
92. Default Options Starting the Amd Daemon ................. 221
93. Example of Output from df Command Showing Multiple Mounts .... 222
94. Sample Amd Map File ............................. 223
95. Error Messages When the Filesystem Is Not Exported .......... 223
96. Output after Exporting /tony on the Control Workstation ......... 223
97. Example of Output from lssrc -g nfs Command .............. 224
98. Extract from amd.log File after amd_start -f Command ......... 224
99. Extract from amd.log When Filesystem Is Not Exported ......... 224
100. Example of /.rhosts File ............................ 226
101. setup_authent Script Flow Chart (1/7) .................... 229
102. setup_authent Script Flow Chart (2/7) .................... 230
103. setup_authent Script Flow Chart (3/7) .................... 231
104. setup_authent Script Flow Chart (4/7) .................... 232
105. setup_authent Script Flow Chart (5/7) .................... 233
106. setup_authent Script Flow Chart (6/7) .................... 234
xSP PD Guide

This soft copy for use by IBM employees only.
107. setup_authent Script Flow Chart (7/7) .................... 235
108. install_cw Script Flow Chart (1/3) ...................... 236
109. install_cw Script Flow Chart (2/3) ...................... 237
110. install_cw Script Flow Chart (3/3) ...................... 238
111. setup_server Script Flow Chart (1/23) .................... 239
112. setup_server Script Flow Chart (2/23) .................... 240
113. setup_server Script Flow Chart (3/23) .................... 241
114. setup_server Script Flow Chart (4/23) .................... 242
115. setup_server Script Flow Chart (5/23) .................... 243
116. setup_server Script Flow Chart (6/23) .................... 244
117. setup_server Script Flow Chart (7/23) .................... 245
118. setup_server Script Flow Chart (8/23) .................... 246
119. setup_server Script Flow Chart (9/23) .................... 247
120. setup_server Script Flow Chart (10/23) ................... 248
121. setup_server Script Flow Chart (11/23) ................... 249
122. setup_server Script Flow Chart (12/23) ................... 250
123. setup_server Script Flow Chart (13/23) ................... 251
124. setup_server Script Flow Chart (14/23) ................... 252
125. setup_server Script Flow Chart (15/23) ................... 253
126. setup_server Script Flow Chart (16/23) ................... 254
127. setup_server Script Flow Chart (17/23) ................... 255
128. setup_server Script Flow Chart (18/23) ................... 256
129. setup_server Script Flow Chart (19/23) ................... 257
130. setup_server Script Flow Chart (20/23) ................... 258
131. setup_server Script Flow Chart (21/23) ................... 259
132. setup_server Script Flow Chart (22/23) ................... 260
133. setup_server Script Flow Chart (23/23) ................... 261
134. rc.switch Script Flow Chart (1/8) ....................... 262
135. rc.switch Script Flow Chart (2/8) ....................... 263
136. rc.switch Script Flow Chart (3/8) ....................... 264
137. rc.switch Script Flow Chart (4/8) ....................... 265
138. rc.switch Script Flow Chart (5/8) ....................... 266
139. rc.switch Script Flow Chart (6/8) ....................... 267
140. rc.switch Script Flow Chart (7/8) ....................... 268
141. rc.switch Script Flow Chart (8/8) ....................... 269
142. The SDR Structure ............................... 271
Figures xi

This soft copy for use by IBM employees only.
xii SP PD Guide

This soft copy for use by IBM employees only.
Tables
1. /spdata Filesystem Directories . . . . . . . . . . . . . . . . . . . . . . . . . 16
2. Components of the PSSP Install Image . . . . . . . . . . . . . . . . . . . . 17
3. Output Terms of the lslpp Command . . . . . . . . . . . . . . . . . . . . . 24
4. Components and Sizes of the PSSP PTFset11 . . . . . . . . . . . . . . . . 28
5. NIM Client Definition Information . . . . . . . . . . . . . . . . . . . . . . . . 55
6. Cabling - SP Switch versus HiPS . . . . . . . . . . . . . . . . . . . . . . . . 98
7. Configuration File Field Definitions . . . . . . . . . . . . . . . . . . . . . 190
Copyright IBM Corp. 1996 xiii

This soft copy for use by IBM employees only.
xiv SP PD Guide

This soft copy for use by IBM employees only.
Preface
Problem determination and problem solving on the RS/6000 SP can be difficult
because the malfunction may imply the involvement of several components
within AIX and the POWERparallel System Support Programs.
This redbook describes each SP component and helps you understand how they
are related to one another. It gives a comprehensive explanation of processes
occurring within the SP system and provides an approach to diagnosis and
problem solving.
For example, we describe how you can use the error log facility to handle
AIX-related problems. For RS/6000 SP-specific problems, we show how you can
use the PSSP component log files to solve them. All SP log information is
located at /var/adm/SPlogs, and all PSSP components use this directory
structure to store debugging information; the internal file structure and syntax
are specific to each component.
The redbook also includes appendices with useful reference material about
RS/6000 SP script files, the SDR structure, and how to change IP addresses and
hostnames for the SP.
This redbook is a valuable tool for system administrators and other technical
support personnel who deal with SP problems. It is also useful for those who
want a more comprehensive understanding of RS/6000 SP components.
How This Redbook Is Organized
This redbook contains 295 pages. Most of the chapters can be read individually,
except for Chapter 5, “System Partitioning,” for which Chapter 4, “The Switch”
is a prerequisite.
This redbook is organized as follows:
•Chapter 1, “Overview”
This chapter gives a brief overview of the main SP components. It explains
how these components make up the system and which of them are specific
to the RS/6000 SP. Experienced SP readers do not need to read this chapter.
•Chapter 2, “The Installation Process”
This chapter covers the kind of problems that are frequently found when the
system administrator is installing or customizing PSSP components. Typical
problems with PSSP scripts, NIM components, and customization steps are
covered here.
•Chapter 3, “Kerberos”
Kerberos provides authentication services that allow certain distributed
services within the SP system, or between it and other workstations, to
secure control access to their facilities. This chapter explains the Kerberos
concepts and how the components work together. It also covers the
common problems that the Kerberos administrator faces, and gives some
procedures to deal with them.
Copyright IBM Corp. 1996 xv

This soft copy for use by IBM employees only.
•Chapter 4, “The Switch”
Most of the Switch-related problems require a clear understanding of the
Switch components. This chapter explains those components and gives
examples about configuration and topology files. Using these examples, the
chapter explains how to handle Switch problems and how to interpret the
Switch log files. This chapter also prepares the reader for Chapter 5, which
requires a good understanding of the Switch components.
•Chapter 5, “System Partitioning”
This chapter covers System Partitioning in a way that will enable the reader
to understand how all the key components fit together, and give an
understanding of how to carry out problem determination based on this
information. It is essential that the reader fully understands the information
laid out in Chapter 4 before beginning to read these details about System
Partitioning.
•Chapter 6, “Error Logging”
This chapter describes the error logging facilities provided by the PSSP
components, and how these facilities interface with the standard AIX error
logging “errlog” mechanism. This chapter also includes a description of
how to customize the
errlog to provide specific notifications that are not part
of the standard set of notifications provided for the PSSP components.
•Chapter 7, “Isolating Problems on the SP System”
The RS/6000 SP provides several tools that help to identify and isolate
problems. In this chapter, these tools will be used, along with the Symptom
Index in Chapter 3 of the
Diagnosis and Messages Guide , GC23-3899 to help
to isolate problems. Once the problems are isolated, the procedures
explained in the previous chapters can be applied.
•Chapter 8, “Producing a System Dump”
AIX allows you to generate Memory System Images, called
dumps , that can
be analyzed at a later time. Sometimes, under error conditions, these
dumps are generated automatically by the operating system. This facility is
present in the Control Workstation and in the nodes. This chapter explains
how to generate system dumps and how to handle them using standard SP
tools.
•Chapter 9, “User and Services Management”
This chapter briefly describes some of the components of User and Service
Management, such as File Collection, Amd, Print Services, and NTP. It also
describes how to diagnose and handle common problems with these
components.
•Appendix A, “RS/6000 SP Script Files”
Many of the SP tasks are carried out by script files. This appendix provides
flow charts for the main PSSP scripts, which can be used as a reference for
problem determination.
•Appendix B, “The SDR Structure”
The SDR is the core of the SP configuration data management. The
structure consists of files, processes, and commands that allow you to
retrieve information. Its structure is shown in this appendix as a reference
for those who want to know more about its components and how they are
interrelated.
xvi SP PD Guide

This soft copy for use by IBM employees only.
•Appendix C, “IP Address and Hostname Changes for the SP”
The IP and Hostname changing on the nodes or on the Control Workstation
is one of the most difficult tasks faced by the administrator. This appendix
provides a procedure that allows you to perform painless and successful IP
and Hostname changing by giving some advice and an easy-to-follow
procedure.
The Team That Wrote This Redbook
This redbook was produced by a team of specialists from around the world
working at the International Technical Support Organization Poughkeepsie
Center.
Marcelo R. Barrios is an SP Project Leader at the International Technical
Support Organization, Poughkeepsie Center. He writes extensively and teaches
IBM classes worldwide on all areas of RS/6000 SP. Before joining the ITSO one
year ago, Mr. Barrios worked as an assistant professor in the Electronic
Department of Santa Maria University, Valparaiso, Chile. In 1993, he joined to
IBM as a Marketing Specialist in the RS/6000 Unit, IBM Chile.
Richard Wagner is a AIX Software Specialist in the RS/6000 Software Support
Center Mainz, IBM Germany. He joined IBM in 1989 and worked in RS/6000 SP
support supporting the RS/6000 BOS software. During the last 10 months he has
focused on the RS/6000 SP products. His expertise has been gained by working
closely with customers resolving installation and post-installation issues. He has
written extensively on the SP Switch. A special “thanks” is extended to
Richards′s colleagues in the AIX Support Center, Mainz, for covering the work
while Richard was writing the redbook.
Andy Hoiles is a Senior AIX Systems Specialist in the RS/6000 Support Center,
IBM UK Ltd. He joined IBM in 1989 and began supporting the RS/6000 software
in 1992. During the last 18 months he has focused on RS/6000 SP products and
his expertise has been gained by working closely with customers resolving
installation and post-installation issues.
Sunil Jain is an Advisory IT Specialist in Sydney, Australia. He is a certified AIX
Support Professional and CNE. He has 18 years experience in the computer
industry. He holds a Post Graduate (M.Sc) degree in Operations Research from
University of Delhi. His area of expertise includes AIX (on RS/6000 and SP),
LAN/WAN and communications/networking products. He has written extensively
on LAN backup/recovery, EDI Implementation and SP Problem Determination.
Tony Schlee is an Advisory Systems Engineer at IBM South Africa. He has 8
years UNIX experience. He has worked for IBM since 1992 with AIX on the
RS/6000.
Thanks to the following people for their invaluable contributions to this project:
Endy Chiakpo
International Technical Support Organization, Poughkeepsie Center
John Lobbes
RS/6000 SP Support Center, Poughkeepsie
Preface xvii

This soft copy for use by IBM employees only.
Comments Welcome
We want our redbooks to be as helpful as possible. Should you have any
comments about this or other redbooks, please send us a note at the following
address:
[email protected]
Your comments are important to us!
xviii SP PD Guide