Silicon Graphics Altix ICE 8000 Series User manual

Scali ManageTM On SGI®Altix®ICE
System Quick Reference Guide
007–5450–001

COPYRIGHT
© 2008, SGI. All rights reserved; provided portions may be copyright in third parties, as indicated elsewhere herein. No permission is
granted to copy, distribute, or create derivative works from the contents of this electronic documentation in any manner, in whole or in
part, without the prior written permission of SGI.
LIMITED RIGHTS LEGEND
The software described in this document is "commercial computer software" provided with restricted rights (except as to included
open/free source) as specified in the FAR 52.227-19 and/or the DFAR 227.7202, or successive sections. Use beyond license provisions is
a violation of worldwide intellectual property laws, treaties and conventions. This document is provided with limited rights as defined
in 52.227-14.
TRADEMARKS AND ATTRIBUTIONS
SGI, the SGI logo, and Altix are registered trademarks and SGI ProPack is a trademark of SGI in the United States and/or other
countries worldwide.
Altair is a registered trademark and PBS Professional is a trademark of Altair Engineering, Inc. Intel, Xeon, and Itanium are
trademarks or registered trademarks of Intel Corporation. InfiniBand is a trademark of the InfiniBand Trade Association. Linux is a
registered trademark of Linus Torvalds. Novell is a registered trademark and SUSE is a trademark of Novell, Inc., in the United States
and other countries. Red Hat and all Red Hat-based trademarks are trademarks or registered trademarks of Red Hat, Inc. in the
United States and other countries. Scali Manage is a trademark of Platform Computing, Inc.
All other trademarks mentioned herein are the property of their respective owners.

Record of Revision
Version Description
001 April 2008
Original publication.
007–5450–001 iii

Contents
About This Guide ..................... xv
Related Publications . . . ................... xv
Obtaining Publications . . . ................... xvi
Conventions . ........................xvii
Reader Comments .......................xvii
1. SGI Altix ICE 8000 Series System Overview .......... 1
Hardware Overview . . . ................... 1
Basic System Building Blocks ................... 1
InfiniBand Fabric . . . ................... 3
Gigabit Ethernet Network ................... 4
Individual Rack Unit . ................... 4
Power Supply ....................... 4
Four-tier, Hierarchical Framework . . .............. 5
Chassis Manager . . . ................... 6
System Nodes ....................... 7
System Admin Controller ................... 8
Rack Leader Controller . ................... 8
Chassis Management Control (CMC) Blade .............. 9
Compute Node . . . ................... 9
Individual Rack Unit . ................... 10
Login Service Node . . ................... 10
Batch Service Node . . ................... 10
Gateway Service Node . ................... 11
007–5450–001 v

Contents
Storage Service Node . ................... 11
Networks . . ........................ 12
Networks Overview . . . ................... 13
Gigabit Ethernet (GigE) and 10/100 Ethernet Connections . ......... 14
VLANs . . ........................ 16
InfiniBand Fabric . . . ................... 21
Network Interface Naming Conventions . . .............. 22
Ethernet Networks . . . ................... 23
InfiniBand Networks . . ................... 23
System Admin Controller . ................... 24
Service Nodes ....................... 24
Rack Leader Controllers . ................... 25
Chassis Management Control (CMC) Blade .............. 25
Compute Nodes ....................... 25
2. Getting Started with Scali Manage . . . . .......... 27
Installing or Updating Software ................... 27
Administrative Tips ....................... 28
System Password Information ................... 28
Power on or Power off System Components or Obtain Status ......... 28
Scali Manage Installer Directory .................. 29
Scali Manage Command CLI Help .................. 30
Configuring the Scali Manage Server . . . .............. 31
Defining New Racks or Service Nodes . . . .............. 31
Discovering Service and Leader Nodes . . . .............. 32
Installing Service and Leader Nodes . . . .............. 32
Discovering CMCs and Compute Nodes . . .............. 33
Installing Compute Nodes . ................... 33
vi 007–5450–001

Scali ManageTM On SGI®Altix®ICE System Quick Reference Guide
Configuration Session Example ................... 37
Using the Scali Manage GUI . ................... 37
Displaying Cluster Components ................... 38
Scali Manage Troubleshooting Tips .................. 39
Compute Node RPMs . . . ................... 40
Compute Node RPMs on SLES .................. 40
Compute Node RPMs on RHEL .................. 41
3. System Fabric Management . . . . . . .......... 43
Appendix A. InfiniBand Fabric Details . . . .......... 45
InfiniBand Fabric Management Configuration and Operation Overview ....... 45
Configuring and Initializing the InfiniBand Fabric Manually . ......... 51
Appendix B. InfiniBand Fabric Troubleshooting .......... 55
Useful Utilities and Diagnostics ................... 55
ibstat Command . . . ................... 56
ibstatus Command . . ................... 57
perfquery Command . . ................... 58
ibnetdiscover Command . ................... 59
ibdiagnet Command . . ................... 60
Index . ......................... 65
007–5450–001 vii

Figures
Figure 1-1 Basic System Building Blocks . .............. 2
Figure 1-2 Chassis Manager Cabling . . . .............. 7
Figure 1-3 Service Nodes . ................... 12
Figure 1-4 Network Connections In a System With Two IRUs ......... 14
Figure 1-5 Chassis Manager ................... 15
Figure 1-6 VLAN_GBE and VLAN_BMC Network Connections - IRU View . . . . . . 18
Figure 1-7 VLAN_GBE and VLAN_BMC Network Connections – Rack View . . . . . 19
Figure 1-8 VLAN_HEAD Network Connections .............. 20
Figure 1-9 Two InfiniBand Fabrics in a System with Two IRUs ......... 22
Figure 2-1 Example Starting Screen for the Scali Manage GUI ......... 38
Figure 2-2 Cluster Components Selection Screen Example . ......... 39
Figure A-1 Two InfiniBand Fabrics in a System with Two IRUs ........ 50
007–5450–001 ix

Examples
Example A-1 opensm-ib0.conf and opensm-ib.conf Configuration Files . . . . 45
007–5450–001 xi

Procedures
Procedure A-1 Configuring and Initializing the InfiniBand Fabric Manually . . . . 51
007–5450–001 xiii

About This Guide
This guide is a reference document for people who manage the operation of SGI Altix
ICE 8000 series systems running SUSE Linux Enterprise Server 10 Service Pack 1 or
Red Hat Enterprise Linux 5.1 (RHEL5.1) with SGI ProPack 5 for Linux Service Pack 4
(or later). It describes how to use Scali Manage version 5.6.1–1 (or later) management
software to perform general system discovery, installation, configuration, and
operations on SGI Altix ICE 8000 series systems.
This manual contains the following chapters:
• Chapter 1, "SGI Altix ICE 8000 Series System Overview" on page 1
• Chapter 2, "Getting Started with Scali Manage" on page 27
• Chapter 3, "System Fabric Management" on page 43
• Appendix A, "InfiniBand Fabric Details" on page 45
• Appendix B, "InfiniBand Fabric Troubleshooting" on page 55
Related Publications
This section describes documentation you may find useful, as follows:
•SGI Altix ICE 8000 System User’s Guide
This is the hardware users guide for the SGI Alitx 8000 series systems. It describes
the features of the SGI Altix ICE 8000 series system, as well as, troubleshooting,
upgrading, and repairing.
The following manuals for Scali Manage are available from Platform Computing, Inc:
•Scali Manage User’s Guide, (2006)
This document provides an overview of a Scali system in terms of instructions for
building a Scali system. Configuration guidelines for hardware and software are
covered along with instructions on use and general management of the cluster
system.
•Scali Quick Start Guide, (2007)
007–5450–001 xv

About This Guide
This document is for system administrators and provides an overview of how to
use Scali Manage to operate your system.
•Scali Manage Installation Guide, (2006)
This document describes the Scali Manage software installer that helps the user
with installation of the OS, Scali software and third-party applications that are
installed as RPMs.
For a list of manuals supporting SGI ProPack for Linux releases covering the
following topics, see the SGI ProPack 5 for Linux Service Pack 4 Start Here:
• SGI documentation supporting SGI Altix ICE systems
• Novell documentation for SUSE Linux Enterprise Server 10 (SLES10)
• Intel Compiler Documentation
• Intel documentation about Xeon architecture
Obtaining Publications
You can obtain SGI documentation in the following ways:
• See the SGI Technical Publications Library at: http://docs.sgi.com. Various formats
are available. This library contains the most recent and most comprehensive set of
online books, release notes, man pages, and other information.
• Online versions of the SGI ProPack 5 for Linux Service Pack 4 Start Here, the SGI
ProPack 5 SP4 release notes, which contain the latest information about software
and documentation in this release, the list of RPMs distributed with SGI ProPack 5
SP4, and a useful migration guide, which contains helpful hints and advice for
customers moving from earlier versions of SGI ProPack to SGI ProPack 5, can be
found in the /docs directory on the SGI ProPack 5 Open/Free Source CD.
The SGI ProPack 5 for Linux SP4 release notes get installed to the following
location on a system running SGI ProPack 5:
/usr/share/doc/sgi-propack-5/README.txt.
• You can view man pages by typing man title on a command line.
xvi 007–5450–001

Scali ManageTM On SGI®Altix®ICE System Quick Reference Guide
Conventions
The following conventions are used throughout this document:
Convention Meaning
command This fixed-space font denotes literal items such as
commands, files, routines, path names, signals,
messages, and programming language structures.
variable Italic typeface denotes variable entries and words or
concepts being defined.
user input This bold, fixed-space font denotes literal items that the
user enters in interactive sessions. (Output is shown in
nonbold, fixed-space font.)
[ ] Brackets enclose optional portions of a command or
directive line.
... Ellipses indicate that a preceding element can be
repeated.
Reader Comments
If you have comments about the technical accuracy, content, or organization of this
publication, contact SGI. Be sure to include the title and document number of the
publication with your comments. (Online, the document number is located in the
front matter of the publication. In printed publications, the document number is
located at the bottom of each page.)
You can contact SGI in any of the following ways:
• Send e-mail to the following address:
[email protected]
• Contact your customer service representative and ask that an incident be filed in
the SGI incident tracking system.
• Send mail to the following address:
SGI
Technical Publications
1140 East Arques Avenue
007–5450–001 xvii

About This Guide
Sunnyvale, CA 94085–4602
SGI values your comments and will respond to them promptly.
xviii 007–5450–001

Chapter 1
SGI Altix ICE 8000 Series System Overview
An SGI Altix ICE 8000 series system is an integrated blade environment that can scale
to thousands of nodes. The Scali Manage management software enables you to
provision, install, configure, and manage your system. This chapter provides an
overview of the SGI Altix ICE 8000 series system and covers the following topics:
• "Hardware Overview" on page 1
• "Networks" on page 12
• "Network Interface Naming Conventions" on page 22
Hardware Overview
This section provides a brief overview of the SGI Altix ICE 8000 series system
hardware and covers the following topics:
• "Basic System Building Blocks" on page 1
• "System Nodes" on page 7
For a detailed hardware description, see the SGI Altix ICE 8000 Series System Hardware
User’s Guide.
Basic System Building Blocks
The SGI Altix ICE 8000 series system is a blade-based, scalable, high density compute
system. The basic building block is the individual rack unit (IRU). The IRU provides
power, cooling, system control, and the network fabric for 16 compute blades, as
shown in Figure 1-1 on page 2. Each compute blade supports two either dual–core or
quad-core Xeon processor sockets and eight fully-buffered, double-data-rate two
(DDR2) memory dual in-line memory module (DIMMs). Four IRUs can reside in a
custom designed 42U high rack.
One rack supports a maximum of 512 processor cores and 2TB of memory.
007–5450–001 1

1: SGI Altix ICE 8000 Series System Overview
Power supplies
InfiniBand
switch blade InfiniBand switch blade
Chassis manager
Independent Rack Unit (IRU)
Power supplies
Rack leader controller
42U High Rack
IRU
IRU
Admin server
Figure 1-1 Basic System Building Blocks
This hardware overview section covers the following topics:
• "InfiniBand Fabric" on page 3
• "Gigabit Ethernet Network" on page 4
• "Individual Rack Unit" on page 4
2 007–5450–001