Silicon Graphics ICE XA Installation manual

SGI®ICE™XA System 
Hardware User Guide 
Document Number 007-6399-002

COPYRIGHT
© 2015 Silicon Graphics International Corporation. All rights reserved; provided portions may be copyright in third parties, as indicated elsewhere herein. No
permission is granted to copy, distribute, or create derivative works from the contents of this electronic documentation in any manner, in whole or in part,
without the prior written permission of SGI.
LIMITED RIGHTS LEGEND
The software described in this document is "commercial computer software" provided with restricted rights (except as to included open/free source) as specified
in the FAR 52.227-19 and/or the DFAR 227.7202, or successive sections. Use beyond license provisions is a violation of worldwide intellectual property laws,
treaties and conventions. This document is provided with limited rights as defined in 52.227-14.
The electronic (software) version of this document was developed at private expense; if acquired under an agreement with the USA government or any
contractor thereto, it is acquired as “commercial computer software” subject to the provisions of its applicable license agreement, as specified in (a) 48 CFR
12.212 of the FAR; or, if acquired for Department of Defense units, (b) 48 CFR 227-7202 of the DoD FAR Supplement; or sections succeeding thereto.
Contractor/manufacturer is SGI, 900 North McCarthy Blvd. Milpitas, CA 95035.
TRADEMARKS AND ATTRIBUTIONS
SGI, and the SGI logo are registered trademarks and Rackable, SGI Lustre and SGI ICE are trademarks of, Silicon Graphics International, in the United States
and/or other countries worldwide.
Intel, Intel QuickPath Interconnect (QPI), Itanium and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States
and other countries. 
UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company, Ltd. 
Infiniband is a trademark of the InfiniBand Trade Association. 
Red Hat and all Red Hat-based trademarks are trademarks or registered trademarks of Red Hat, Inc. in the United States and other countries.
Linux is a registered trademark of Linus Torvalds.
All other trademarks mentioned herein are the property of their respective owners.

007-6399-002 iii
Record of Revision
Version Description
-001 May, 2015
First release
-002 July, 2015
Second release covering updated cpower commands

007-6399-002 v
Contents
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . ix
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . xi
Audience. . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Important Information . . . . . . . . . . . . . . . . . . . . . . xiii
Chapter Descriptions . . . . . . . . . . . . . . . . . . . . . . xiv
Related Publications . . . . . . . . . . . . . . . . . . . . . . . xv
Conventions . . . . . . . . . . . . . . . . . . . . . . . . .xvii
Product Support . . . . . . . . . . . . . . . . . . . . . . . .xvii
Reader Comments . . . . . . . . . . . . . . . . . . . . . . xviii
1. Operation Procedures . . . . . . . . . . . . . . . . . . . . . . 1
Precautions . . . . . . . . . . . . . . . . . . . . . . . . . 1
ESD Precaution . . . . . . . . . . . . . . . . . . . . . . . 1
Safety Precautions . . . . . . . . . . . . . . . . . . . . . . 2
Console Connections . . . . . . . . . . . . . . . . . . . . . . 3
Powering the System On and Off . . . . . . . . . . . . . . . . . . . 4
Preparing to Power On . . . . . . . . . . . . . . . . . . . . . 5
Powering On and Off . . . . . . . . . . . . . . . . . . . . . 8
Console Management Power (cpower) Commands . . . . . . . . . . . . 8
Monitoring Your Server . . . . . . . . . . . . . . . . . . . . . . 12
Chassis Management Monitoring and Control . . . . . . . . . . . . . . . 12
PCIe Subsystem Control . . . . . . . . . . . . . . . . . . . . 13
2. System Management . . . . . . . . . . . . . . . . . . . . . . 15
Using the 1U Console Option . . . . . . . . . . . . . . . . . . . . 17
Levels of System and Chassis Control . . . . . . . . . . . . . . . . . . 17

vi 007-6399-002
Contents
Chassis Controller Interaction . . . . . . . . . . . . . . . . . . . . 17
Chassis Manager Interconnects . . . . . . . . . . . . . . . . . . . 18
Chassis Management Control (CMC) Functions . . . . . . . . . . . . . . 19
System Power Status . . . . . . . . . . . . . . . . . . . . . . . 19
3. System Overview . . . . . . . . . . . . . . . . . . . . . . . 21
System Models . . . . . . . . . . . . . . . . . . . . . . . . 22
SGI ICE XA System and Blade Architectures . . . . . . . . . . . . . . . . 26
IP125 Blade Architecture Overview . . . . . . . . . . . . . . . . . 26
IP139 Blade Architecture Overview . . . . . . . . . . . . . . . . . 27
QuickPath Interconnect Features. . . . . . . . . . . . . . . . . . . . 28
QPI Bandwidth Overview . . . . . . . . . . . . . . . . . . . . 28
Blade Memory Features . . . . . . . . . . . . . . . . . . . . . 29
Blade DIMM Memory Features . . . . . . . . . . . . . . . . . 29
DIMM Bandwidth Factors . . . . . . . . . . . . . . . . . . . 29
Memory Channel Recommendation . . . . . . . . . . . . . . . . 30
System InfiniBand Switch Blades . . . . . . . . . . . . . . . . . . 30
System Features and Major Components . . . . . . . . . . . . . . . . . 31
Modularity and Scalability . . . . . . . . . . . . . . . . . . . . 31
System Administration Server . . . . . . . . . . . . . . . . . . 32
Rack Leader Controller . . . . . . . . . . . . . . . . . . . . 33
Multiple Chassis Manager Connections . . . . . . . . . . . . . . . 33
The RLC as Fabric Manager . . . . . . . . . . . . . . . . . . 34
Service Nodes . . . . . . . . . . . . . . . . . . . . . . . 35
Login Server Function . . . . . . . . . . . . . . . . . . . . 35
Batch Server Node . . . . . . . . . . . . . . . . . . . . . 36
I/O Gateway Node . . . . . . . . . . . . . . . . . . . . . 36
Optional Lustre Nodes Overview . . . . . . . . . . . . . . . . . . 37
MDS Node . . . . . . . . . . . . . . . . . . . . . . . 37
OSS Node . . . . . . . . . . . . . . . . . . . . . . . 38
Reliability, Availability, and Serviceability (RAS) . . . . . . . . . . . . . . . 38

Contents
007-6399-002 vii
Optional SGI Remote Services (SGI RS) . . . . . . . . . . . . . . . . . 39
SGI Remote Services Primary Capabilities . . . . . . . . . . . . . . . 39
SGI Remote Services Benefits . . . . . . . . . . . . . . . . . . . 40
SGI Remote Service Operations Overview. . . . . . . . . . . . . . . . 40
SGI Knowledgebase . . . . . . . . . . . . . . . . . . . . . 41
SGI Warranty Levels . . . . . . . . . . . . . . . . . . . . . 42
System Components . . . . . . . . . . . . . . . . . . . . . . . 43
E-Cell Rack Numbering . . . . . . . . . . . . . . . . . . . . 43
Optional System Components . . . . . . . . . . . . . . . . . . . 44
4. Rack Information . . . . . . . . . . . . . . . . . . . . . . . 45
Overview . . . . . . . . . . . . . . . . . . . . . . . . . 45
SGI ICE XA Series D-Rack (42U) . . . . . . . . . . . . . . . . . . . 46
ICE XA D-Rack Technical Specifications . . . . . . . . . . . . . . . . . 49
SGI ICE XA E-Cell Rack Assemblies . . . . . . . . . . . . . . . . . . 50
E-Cell Functional Overview . . . . . . . . . . . . . . . . . . . 51
E-Cell Compute Rack Specifications . . . . . . . . . . . . . . . . 51
E-Cell Cooling Rack Specifications . . . . . . . . . . . . . . . . 52
Cooling Distribution Unit (CDU) Rack Specifications . . . . . . . . . . . 53
5. SGI ICE XA Administration/Leader Servers . . . . . . . . . . . . . . . 55
Overview . . . . . . . . . . . . . . . . . . . . . . . . . 56
System Hierarchy . . . . . . . . . . . . . . . . . . . . . . 56
Communication Hierarchy. . . . . . . . . . . . . . . . . . . 57
1U Rack Leader Controller and Administration Server . . . . . . . . . . . . . . 59
1U Service Nodes . . . . . . . . . . . . . . . . . . . . . . . 60
C1104-GP1 1U Service Node . . . . . . . . . . . . . . . . . . . 60
2U Service Nodes . . . . . . . . . . . . . . . . . . . . . . . 62
C2112-GP2 Service Node . . . . . . . . . . . . . . . . . . . . 62
SGI UV 30 2U Service Node . . . . . . . . . . . . . . . . . . . 63

viii 007-6399-002
Contents
A. Technical Specifications and Pinouts . . . . . . . . . . . . . . . . . . 65
System-level Specifications . . . . . . . . . . . . . . . . . . . . . 65
D-Rack Physical and Power Specifications . . . . . . . . . . . . . . . . 66
D-Rack System Environmental Specifications . . . . . . . . . . . . . . . 67
ICE XA E-Rack Technical Specifications . . . . . . . . . . . . . . . . 68
Ethernet Port Specification . . . . . . . . . . . . . . . . . . . . 70
B. Safety Information and Regulatory Specifications . . . . . . . . . . . . . . 71
Safety Information . . . . . . . . . . . . . . . . . . . . . . . 71
Regulatory Specifications . . . . . . . . . . . . . . . . . . . . . 73
CMN Number . . . . . . . . . . . . . . . . . . . . . . . 73
CE Notice and Manufacturer’s Declaration of Conformity . . . . . . . . . . . . 73
Electromagnetic Emissions . . . . . . . . . . . . . . . . . . . . 73
FCC Notice (USA Only) . . . . . . . . . . . . . . . . . . . 74
Industry Canada Notice (Canada Only) . . . . . . . . . . . . . . . 74
VCCI Notice (Japan Only) . . . . . . . . . . . . . . . . . . . 75
Chinese Class A Regulatory Notice . . . . . . . . . . . . . . . . 75
Korean Class A Regulatory Notice . . . . . . . . . . . . . . . . . 75
Shielded Cables . . . . . . . . . . . . . . . . . . . . . . . 76
Electrostatic Discharge and Laser Compliance . . . . . . . . . . . . . . . 76
Lithium Battery Statements . . . . . . . . . . . . . . . . . . . . 77

007-6399-002 ix
List of Figures
Figure 1-1 Flat Panel Rackmount Console Option . . . . . . . . . . 3
Figure 1-2 Administrative Controller Video Console Connection Points . . . . 4
Figure 1-3 Eight-Outlet Single-Phase PDU Example . . . . . . . . . 6
Figure 1-4 Three-Phase PDU Example . . . . . . . . . . . . . 7
Figure 2-1 SGI ICE XA System Admin and Service Node Network Access Example . 16
Figure 2-2 E-rack System Chassis Management Diagram (With CDU). . . . . 18
Figure 3-1 SGI ICE XA Series System E-Cell Assembly Example . . . . . . 23
Figure 3-2 D-rack Standard and Optional Components Example . . . . . . 25
Figure 3-3 SGI ICE XA System and Network Components Overview . . . . . 32
Figure 3-4 Administration and RLC Cabling to Chassis Managers . . . . . . 34
Figure 3-5 Example Rear View of a 1U Service Node . . . . . . . . . 35
Figure 3-6 2U Service Node Front and Rear Panel Example . . . . . . . 36
Figure 3-7 SGI Remote Solutions Process Overview . . . . . . . . . 41
Figure 4-1 D-Rack Front Lock Example. . . . . . . . . . . . . 47
Figure 4-2 Optional Water-Chilled Door Panels on Rear of ICE XA D-Rack . . . 48
Figure 4-3 E-Cell Rack Configuration Example. . . . . . . . . . . 50
Figure 4-4 E-Cell Rack Assembly Example (CDU on Left). . . . . . . . 54
Figure 5-1 SGI ICE XA System Administration Hierarchy Example Diagram . . . 58
Figure 5-2 1U Rack Leader Controller (RLC)/Administration Server Front Panel . . 59
Figure 5-3 1U RLC/Administration Server Rear Components Example . . . . 59
Figure 5-4 SGI Rackable C1104-GP1 1U Service Node Front and Rear Panel Example 61
Figure 5-5 SGI Rackable C1104-GP1 System Control Panel and LEDs . . . . 61
Figure 5-6 SGI Rackable C2112-GP2 Service Node Front and Rear Panel Example . 62
Figure 5-7 SGI UV 30 Service Node Front and Rear Panel Example . . . . . 63

x007-6399-002
List of Figures
Figure A-1 Ethernet Port . . . . . . . . . . . . . . . . . 70
Figure B-1 VCCI Notice (Japan Only) . . . . . . . . . . . . . 75
Figure B-2 Chinese Class A Regulatory Notice . . . . . . . . . . . 75
Figure B-3 Korean Class A Regulatory Notice . . . . . . . . . . . 75

007-6399-002 xi
List of Tables
Table 1-1 cpower option, action, target type and target list descriptions . . . . 8
Table 1-2 cpower example command strings . . . . . . . . . . . 11
Table 4-1 Tall SGI ICE XA D-Rack Technical Specifications . . . . . . . 49
Table 4-2 SGI ICE XA E-Rack Technical Specifications . . . . . . . . 51
Table 4-3 SGI ICE XA E-Cell Cooling-Rack Specifications . . . . . . . 52
Table 4-4 SGI ICE XA Cooling Distribution Unit (CDU) Specifications . . . . 53
Table A-1 SGI ICE XA Series Configuration Ranges . . . . . . . . . 65
Table A-2 ICE XA System D-Rack Physical Specifications . . . . . . . 66
Table A-3 Environmental Specifications (Single D-Rack) . . . . . . . . 67
Table A-4 SGI ICE XA E-Rack Physical Specifications . . . . . . . . 68
Table A-5 Environmental Specifications (Single E-Rack) . . . . . . . . 69
Table A-6 Ethernet Pinouts . . . . . . . . . . . . . . . . 70

007-6399-002 xiii
About This Guide
This guide provides an overview of the architecture, general operation and descriptions of the
major components that compose the SGI®Integrated Compute Environment (ICE™) XA series
blade enclosure systems. It also provides the standard procedures for powering on and powering
off the system, basic troubleshooting information, customer maintenance procedures and
important safety and regulatory specifications.
Audience
This guide is written for owners, system administrators, and users of SGI ICE XA series computer
systems.
It is written with the assumption that the reader has a good working knowledge of computers and
computer systems.
Important Information
Warning: To avoid problems that could void your warranty, your SGI or other approved
service technician should perform all the setup, addition, or replacement of parts, cabling,
and service of your SGI ICE XA series system, with the exception of the following items that
you can perform yourself:
• Using your system console or network access workstation to enter commands and perform
system functions such as powering on and powering off, as described in this guide.
• Removing and replacing service nodes in the air-cooled D-rack.
• Adding and replacing disk drives in optional storage systems and using the operator’s panel
on optional mass storage.

xiv 007-6399-002
About This Guide
Chapter Descriptions
The following topics are covered in this guide:
•Chapter 1, “Operation Procedures,” provides instructions for powering on and powering off
your system.
•Chapter 2, “System Management,” describes the function of the chassis management
controllers (CMC) and provides overview instructions for operating the controllers.
•Chapter 3, “System Overview,” provides environmental and technical information needed to
properly set up and configure the blade systems.
•Chapter 4, “Rack Information,” describes the system’s rack features.
•Chapter 5, “SGI ICE XA Administration/Leader Servers” describes all the controls,
connectors and LEDs located on the front of the stand-alone administrative, rack leader and
other support server nodes. An outline of the server functions is also provided.
•Chapter 6, “Basic Troubleshooting,” provides recommended actions if problems occur on
your system.
•Chapter 7, “Maintenance Procedures,” covers end-user service procedures that do not
require special skills or tools to perform. Procedures not covered in this chapter should be
referred to SGI customer support specialists or in-house trained service personnel.
•Appendix A, “Technical Specifications and Pinouts‚" provides physical, environmental, and
power specifications for your system. Also included are the pinouts for the non-proprietary
connectors.
•Appendix B, “Safety Information and Regulatory Specifications‚" lists regulatory
information related to use of the blade cluster system in the United States and other
countries. It also provides a list of safety instructions to follow when installing, operating, or
servicing the product.

Related Publications
007-6399-002 xv
Related Publications
The following documents are relevant to and can be used with the ICE XA series of computer
systems:
•SGI Rackable C1104-GP2 and C1110-GP2 System User Guide, (P/N 007-6388-00x)
This guide discuses the use, maintenance and operation of the 1U server primarily used as the
system’s rack leader controller (RLC) server node. This stand-alone 1U compute node is also used
as the default administrative server on the ICE XA system. It may also be ordered configured as
a login, or batch server, or other type of support server used with the ICE XA series of computer
systems. The C1104-GP2 variation uses four 3.5-inch internal drives and the C1110-GP2 uses ten
internal 2.5-inch drives.
•SGI Rackable C1104-GP1 System User Guide, (P/N 007-6364-00x)
This user guide covers an overview of the installation, architecture, general operation, and
descriptions of the major components in the SGI Rackable C1104-GP1 server. It also provides
basic troubleshooting and maintenance information, and important safety and regulatory
specifications. This 1U server is used only as an optional service node for login, batch, MDS or
other service node purposes. This server is not used as a system RLC or administrative server.
•SGI Rackable C2112-GP2 System User Guide (P/N 007-6362-00x)
This guide covers general operation, installation, configuration, and servicing of the 2U Rackable
C2112-GP2 server node used in the SGI ICE XA system. The 2U server can be used as a service
node for login, batch, I/O gateway, MDS, or other service node purposes.
•SGI UV 30 System User Guide, (P/N 007-6419-00x)
This user guide covers general operation, configuration, and troubleshooting. Also included is a
description of major components of the optional 2U-high SGI UV 30 four-socket server node unit
used in SGI ICE XA systems. The UV 30 server cannot be used as an administrative server or rack
leader controller. Uses for the system include configuration as an I/O gateway, a mass storage
resource, a general service node for login or batch services or some combination of the previous
functions.
•SGI Management Center Installation and Configuration Guide for Clusters, 
(P/N 007-6359-00x)
This guide discuses software installation and system configuration operations used with the SGI
ICE XA series servers. The management center software is also used to provision other non-ICE
clusters or other SGI systems.

xvi 007-6399-002
About This Guide
•SGI Management Center Administration Guide for Clusters, (P/N 007-6358-00x)
This document is intended for people who manage and administer the operation of SGI ICE XA
systems. The management center software is also used to administer other non-ICE SGI clusters
or systems.
• Man pages (online)
Man pages locate and print the titled entries from the online reference manuals.
You can obtain SGI documentation, release notes, or man pages in the following ways:
• See the SGI Technical Publications Library at http://docs.sgi.com. 
Various formats are available. This library contains the most recent and most comprehensive
set of online books, release notes, man pages, and other information.
• The release notes, which contain the latest information about software and documentation in
this release, are in a file named README.SGI in the root directory of the SGI ProPack for
Linux distribution media.
• You can also view man pages by typing man <title> on a command line.
SGI systems include a set of Linux man pages, formatted in the standard UNIXA “man page”
style. Important system configuration files and commands are documented on man pages. These
are found online on the internal system disk (or DVD) and are displayed using the man command.
For example, to display a man page, type the request on a command line:
man commandx
References in the documentation to these pages include the name of the command and the section
number in which the command is found. For additional information about displaying man pages
using the man command, see man(1). In addition, the apropos command locates man pages
based on keywords. For example, to display a list of man pages that describe disks, type the
following on a command line:
apropos disk

Conventions
007-6399-002 xvii
Conventions
The following conventions are used throughout this document:
Convention Meaning
Command This fixed-space font denotes literal items such as commands, files,
routines, path names, signals, messages, and programming language
structures.
variable The italic typeface denotes variable entries and words or concepts being
defined. Italic typeface is also used for book titles.
user input This bold fixed-space font denotes literal items that the user enters in
interactive sessions. Output is shown in nonbold, fixed-space font.
[ ] Brackets enclose optional portions of a command or directive line.
... Ellipses indicate that a preceding element can be repeated.
man page(x)Man page section identifiers appear in parentheses after man page names.
GUI element This font denotes the names of graphical user interface (GUI) elements such
as windows, screens, dialog boxes, menus, toolbars, icons, buttons, boxes,
fields, and lists.
Product Support
SGI provides a comprehensive product support and maintenance program for its products, as
follows:
• If you are in North America, contact the Technical Assistance Center at 
[phone]SGI or contact your authorized service provider.
• If you are outside North America, contact the SGI subsidiary or authorized distributor in
your country. International customers can visit http://www.sgi.com/support/ 
Click on the “Support Centers” link under the “Online Support” heading for information on
how to contact your nearest SGI customer support center.

xviii 007-6399-002
About This Guide
Reader Comments
If you have comments about the technical accuracy, content, or organization of this document,
contact SGI. Be sure to include the title and document number of the manual with your comments.
(Online, the document number is located in the front matter of the manual. In printed manuals, the
document number is located at the bottom of each page.)
You can contact SGI in the following ways:
• Send e-mail to the following address: techpu[email protected]
• Contact your customer service representative and ask that an incident be filed in the SGI
incident tracking system.
SGI values your comments and will respond to them promptly.

007-6399-002 1
Chapter 1
1. Operation Procedures
This chapter explains how to operate your new system in the following sections:
•“Precautions” on page 1
•“Console Connections” on page 3
•“Powering the System On and Off” on page 4
•“Monitoring Your Server” on page 12
Precautions
Before operating your system, familiarize yourself with the safety information in the following
sections:
•“ESD Precaution” on page 1
•“Safety Precautions” on page 2
ESD Precaution
Caution: Observe all electro-static discharge (ESD) precautions. Failure to do so can result in
damage to the equipment.
Wear an approved ESD wrist strap when you handle any ESD-sensitive device to eliminate
possible damage to equipment. Connect the wrist strap cord directly to earth ground.

2007-6399-002
1: Operation Procedures
Safety Precautions
Warning: Before operating or servicing any part of this product, read the “Safety
Information” on page 69.
Danger: Keep fingers and conductive tools away from high-voltage areas. Failure to
follow these precautions will result in serious injury or death. The high-voltage areas of the
system are indicated with high-voltage warning labels.
Caution:
!
Power off the system only after the system software has been shut down in an orderly
manner. If you power off the system before you halt the operating system, data may be corrupted.
Warning: If a lithium battery is installed in your system as a soldered part, only qualified
SGI service personnel should replace this lithium battery. For a battery of another type,
replace it only with the same type or an equivalent type recommended by the battery
manufacturer, or an explosion could occur. Discard used batteries according to the
manufacturer’s instructions.