Silicon Graphics UV 2000 User manual

SGI®UV 2000 System User Guide
Document Number 007-5832-002

COPYRIGHT
© 2013 SGI. All rights reserved; provided portions may be copyright in third parties, asindicated elsewhere herein. No permission is granted to copy, distribute,
or create derivative works from the contents of this electronic documentation in any manner, in whole or in part, without the prior written permission of SGI.
LIMITED RIGHTS LEGEND
The software described in this document is "commercial computer software" provided with restricted rights (except as to included open/freesource) as specified
in the FAR 52.227-19 and/or the DFAR 227.7202, or successive sections. Use beyond license provisions is a violation of worldwide intellectual property laws,
treaties and conventions. This document is provided with limited rights as defined in 52.227-14.
The electronic (software) version of this document was developed at private expense; if acquired under an agreement with the USA government or any
contractor thereto, it is acquired as “commercial computer software” subject to the provisions of its applicable license agreement, as specified in (a) 48 CFR
12.212 of the FAR; or, if acquired for Department of Defense units, (b) 48 CFR 227-7202 of the DoD FAR Supplement; or sections succeeding thereto.
Contractor/manufacturer is SGI, 46600 Landing Parkway, Fremont, CA 94538.
TRADEMARKS AND ATTRIBUTIONS
Silicon Graphics, SGI, the SGI logo, NUMAlink and NUMAflex are trademarks or registered trademarks of Silicon Graphics International Corp. or its
subsidiaries in the United States and/or other countries worldwide.
Intel, Itanium and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company, Ltd.
Infiniband is a trademark of the InfiniBand Trade Association.
LSI, MegaRAID, and MegaRAID Storage Manager are trademarks or registered trademarks of LSI Corporation.
Linux is a registered trademark of Linus Torvalds in the U.S. and other countries.
Red Hat and all Red Hat-based trademarks are trademarks or registered trademarks of Red Hat, Inc. in the United States and other countries.
SUSE LINUX is a registered trademark of Novell Inc.
Windows is a registered trademark of Microsoft Corporation in the United States and other countries.
All other trademarks mentioned herein are the property of their respective owners.

007-5832-002 iii
Record of Revision
Version Description
001 June, 2012
First Release
002 May, 2013
MIC/GPU Blade and RAID updates

007-5832-002 v
Contents
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . xi
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . xiii
Audience. . . . . . . . . . . . . . . . . . . . . . . . . . xv
Important Information . . . . . . . . . . . . . . . . . . . . . . xv
Chapter Descriptions . . . . . . . . . . . . . . . . . . . . . . xvi
Related Publications. . . . . . . . . . . . . . . . . . . . . . .xvii
Conventions . . . . . . . . . . . . . . . . . . . . . . . . . xix
Product Support . . . . . . . . . . . . . . . . . . . . . . . . xx
Reader Comments . . . . . . . . . . . . . . . . . . . . . . . xx
1. Operation Procedures . . . . . . . . . . . . . . . . . . . . . . 1
Precautions . . . . . . . . . . . . . . . . . . . . . . . . . 1
ESD Precaution . . . . . . . . . . . . . . . . . . . . . . . 1
Safety Precautions . . . . . . . . . . . . . . . . . . . . . . 2
Power Connections Overview . . . . . . . . . . . . . . . . . . . . 2
System Connections Overview . . . . . . . . . . . . . . . . . . . . 7
Connecting to the UV System Control Network . . . . . . . . . . . . . . 7
System Controller Access . . . . . . . . . . . . . . . . . . . . 8
Serial Console Hardware Requirements . . . . . . . . . . . . . . . . 9
Establishing a Serial Connection to the CMC on SGI UV 2000 . . . . . . . . . . 9
Establishing CMC IP Hardware Connections . . . . . . . . . . . . . . . 9
Using DHCP to Establish an IP Address . . . . . . . . . . . . . . . 10
Using a Static IP Address . . . . . . . . . . . . . . . . . . . 10

vi 007-5832-002
Contents
System Control Overview . . . . . . . . . . . . . . . . . . . . . 12
Communicating with the System . . . . . . . . . . . . . . . . . . 13
The SGI Management Center Graphical User Interface . . . . . . . . . . . . 13
Powering-On and Off From the SGI Management Center Interface . . . . . . . . . 13
The Command Line Interface . . . . . . . . . . . . . . . . . . . 14
Powering On and Off from the Command Line Interface . . . . . . . . . . . 14
Command Options for Power On . . . . . . . . . . . . . . . . . 15
Power On the System From the SMN Command Line. . . . . . . . . . . . 15
Specific CLI Commands Used With the SMN . . . . . . . . . . . . . . 16
Optional Power On From the CMC Command Line . . . . . . . . . . . . . 17
Booting Directly From a CMC . . . . . . . . . . . . . . . . . . . 17
Power On the System Using the CMC Network . . . . . . . . . . . . . 17
Optional Power On Using the SMN to Connect to the CMC . . . . . . . . . . 18
Monitoring Power On . . . . . . . . . . . . . . . . . . . . . 19
Power off a UV System . . . . . . . . . . . . . . . . . . . . . 20
Additional CLI Power Command Options . . . . . . . . . . . . . . . 20
Using Embedded Support Partner (ESP). . . . . . . . . . . . . . . . . . 22
Optional Components . . . . . . . . . . . . . . . . . . . . . . 23
PCIe Cards . . . . . . . . . . . . . . . . . . . . . . . . 23
PCIe Drive Controllers in BaseIO Blade . . . . . . . . . . . . . . . . 25
RAID PCIe Disk Controller . . . . . . . . . . . . . . . . . . 25
Non-RAID PCIe Disk Controller . . . . . . . . . . . . . . . . . 26
2. System Control . . . . . . . . . . . . . . . . . . . . . . . . 27
Levels of System Control . . . . . . . . . . . . . . . . . . . . . 27
System Management Node (SMN) Overview . . . . . . . . . . . . . . . 28
CMC Overview . . . . . . . . . . . . . . . . . . . . . . . 29
BMC Overview . . . . . . . . . . . . . . . . . . . . . . . 31
System Controller Interaction . . . . . . . . . . . . . . . . . . . . 31
IRU Controllers . . . . . . . . . . . . . . . . . . . . . . . . 32
Chassis Management Controller Functions . . . . . . . . . . . . . . . . 32
1U Console Option . . . . . . . . . . . . . . . . . . . . . . 32
Flat Panel Rackmount Console Option Features . . . . . . . . . . . . . 33

Contents
007-5832-002 vii
3. System Overview . . . . . . . . . . . . . . . . . . . . . . . 35
System Models . . . . . . . . . . . . . . . . . . . . . . . . 37
System Architecture . . . . . . . . . . . . . . . . . . . . . . . 39
System Features . . . . . . . . . . . . . . . . . . . . . . . . 41
Modularity and Scalability . . . . . . . . . . . . . . . . . . . . 41
Distributed Shared Memory (DSM) . . . . . . . . . . . . . . . . . 41
Distributed Shared I/O . . . . . . . . . . . . . . . . . . . . . 43
Chassis Management Controller (CMC) . . . . . . . . . . . . . . . . 43
ccNUMA Architecture . . . . . . . . . . . . . . . . . . . . . 43
Cache Coherency . . . . . . . . . . . . . . . . . . . . . 43
Non-uniform Memory Access (NUMA) . . . . . . . . . . . . . . . 44
Reliability, Availability, and Serviceability (RAS) . . . . . . . . . . . . . 44
System Components. . . . . . . . . . . . . . . . . . . . . . . 46
Optional BaseIO SSDs . . . . . . . . . . . . . . . . . . . . . 48
MIC/GPU Enabled Compute Blade . . . . . . . . . . . . . . . . . 49
Bay (Unit) Numbering . . . . . . . . . . . . . . . . . . . . . 50
Rack Numbering . . . . . . . . . . . . . . . . . . . . . . 50
Optional System Components . . . . . . . . . . . . . . . . . . . 50
4. Rack Information . . . . . . . . . . . . . . . . . . . . . . . 51
Overview . . . . . . . . . . . . . . . . . . . . . . . . . 51
SGI UV 2000 Series Rack (42U) . . . . . . . . . . . . . . . . . . . 52
SGI UV 2000 System Rack Technical Specifications . . . . . . . . . . . . . . 56
5. Optional Octal Router Chassis Information . . . . . . . . . . . . . . . . 57
Overview . . . . . . . . . . . . . . . . . . . . . . . . . 57
SGI UV 2000 Series NUMAlink Octal Router Chassis. . . . . . . . . . . . . . 58
SGI UV 2000 External NUMAlink System Technical Specifications . . . . . . . . . . 60
6. Add or Replace Procedures . . . . . . . . . . . . . . . . . . . . 61
Maintenance Precautions and Procedures . . . . . . . . . . . . . . . . . 61
Preparing the System for Maintenance or Upgrade . . . . . . . . . . . . . 62
Returning the System to Operation. . . . . . . . . . . . . . . . . . 62
Overview of PCI Express (PCIe) Operation . . . . . . . . . . . . . . . . 63

viii 007-5832-002
Contents
Adding or Replacing PCIe Cards in the Expansion Enclosure . . . . . . . . . . . . 64
Installing Cards in the 1U PCIe Expansion Chassis. . . . . . . . . . . . . . 65
Removing and Replacing an IRU Enclosure Power Supply . . . . . . . . . . . 68
7. Troubleshooting and Diagnostics . . . . . . . . . . . . . . . . . . . 71
Troubleshooting Chart . . . . . . . . . . . . . . . . . . . . . . 72
LED Status Indicators . . . . . . . . . . . . . . . . . . . . . . 73
IRU Power Supply LEDs . . . . . . . . . . . . . . . . . . . . 73
Compute/Memory Blade LEDs . . . . . . . . . . . . . . . . . . . 74
SGI Electronic Support . . . . . . . . . . . . . . . . . . . . . . 75
A. Technical Specifications and Pinouts . . . . . . . . . . . . . . . . . . 79
System-level Specifications . . . . . . . . . . . . . . . . . . . . . 79
Physical Specifications . . . . . . . . . . . . . . . . . . . . . . 80
Environmental Specifications . . . . . . . . . . . . . . . . . . . . 81
Power Specifications . . . . . . . . . . . . . . . . . . . . . . . 82
I/O Port Specifications . . . . . . . . . . . . . . . . . . . . . . 83
BaseIO VGA Port Information . . . . . . . . . . . . . . . . . . . 83
Ethernet Port. . . . . . . . . . . . . . . . . . . . . . . . 85
Serial Ports . . . . . . . . . . . . . . . . . . . . . . . . 86
USB Type A Connector . . . . . . . . . . . . . . . . . . . . . 88
B. Safety Information and Regulatory Specifications . . . . . . . . . . . . . . 89
Safety Information . . . . . . . . . . . . . . . . . . . . . . . 89

Contents
007-5832-002 ix
Regulatory Specifications . . . . . . . . . . . . . . . . . . . . . 91
CMN Number . . . . . . . . . . . . . . . . . . . . . . . 91
CE Notice and Manufacturer’s Declaration of Conformity . . . . . . . . . . . 91
Electromagnetic Emissions. . . . . . . . . . . . . . . . . . . . 92
FCC Notice (USA Only) . . . . . . . . . . . . . . . . . . . 92
Industry Canada Notice (Canada Only) . . . . . . . . . . . . . . . 93
VCCI Notice (Japan Only). . . . . . . . . . . . . . . . . . . 93
Korean Class A Regulatory Notice . . . . . . . . . . . . . . . . 93
Shielded Cables. . . . . . . . . . . . . . . . . . . . . . . 94
Electrostatic Discharge . . . . . . . . . . . . . . . . . . . . . 94
Laser Compliance Statements . . . . . . . . . . . . . . . . . . . 95
Lithium Battery Statement . . . . . . . . . . . . . . . . . . . . 96

007-5832-002 xi
List of Figures
Figure 1-1 IRU Power Supply Cable Location Example . . . . . . . . 3
Figure 1-2 Single-Phase 2-Outlet PDU Example . . . . . . . . . . 4
Figure 1-3 Single-Phase 8-Outlet PDU . . . . . . . . . . . . . 5
Figure 1-4 Three-Phase PDU Examples . . . . . . . . . . . . . 6
Figure 1-5 System Management Node Rear Video Connections . . . . . . 8
Figure 1-6 UV CMC Connection Faceplate Example . . . . . . . . . 8
Figure 1-7 PCIe Option Blade Example with Full-Height and Low-Profile Slots . . 24
Figure 1-8 PCIe Option Blade Example with Two Low-Profile Slots . . . . . 24
Figure 1-9 BaseIO Blade and PCIe Disk Controller Example . . . . . . . 25
Figure 2-1 System Management Node Front and Rear Panels . . . . . . . 29
Figure 2-2 SGI UV 2000 LAN-attached System Control Network Example . . . 30
Figure 2-3 Optional 1U Rackmount Console . . . . . . . . . . . 33
Figure 2-4 System Management Node (SMN) Direct Video Connection Ports. . . 34
Figure 3-1 SGI UV 2000 Single-Rack System Example . . . . . . . . 36
Figure 3-2 SGI UV 2000 IRU and Rack. . . . . . . . . . . . . 38
Figure 3-3 Functional Block Diagram of the Individual Rack Unit (IRU) . . . . 40
Figure 3-4 Blade Node Block Diagram Example . . . . . . . . . . 42
Figure 3-5 SGI UV 2000 IRU System Components Example . . . . . . . 47
Figure 3-6 BaseIO Riser Enabled Blade Front Panel Example . . . . . . . 48
Figure 3-7 MIC/GPU Enabled Compute Blade Example Front View . . . . . 49
Figure 4-1 SGI UV 2000 Series Rack Example . . . . . . . . . . . 53
Figure 4-2 Front Lock on Tall (42U) Rack . . . . . . . . . . . . 54
Figure 4-3 Optional Water-Chilled Cooling Units on Rear of SGI 42U Rack . . . 55
Figure 5-1 SGI UV 2000 Optional NUMAlink ORC (Rear View) . . . . . . 58
Figure 5-2 SGI UV 2000 Optional ORC Chassis Example (Front View) . . . . 59
Figure 6-1 Comparison of PCI/PCI-X Connector with PCI Express Connectors . . 63
Figure 6-2 The PCIe Expansion Enclosure . . . . . . . . . . . . 66

xii 007-5832-002
List of Figures
Figure 6-3 Card Slot Locations . . . . . . . . . . . . . . . 67
Figure 6-4 Removing an Enclosure Power Supply . . . . . . . . . . 68
Figure 7-1 UV Compute Blade Status LED Locations Example. . . . . . . 74
Figure 7-2 Full Support Sequence Example . . . . . . . . . . . . 75
Figure A-1 VGA Port Pinouts. . . . . . . . . . . . . . . . 83
Figure A-2 Ethernet Port . . . . . . . . . . . . . . . . . 85
Figure A-3 Serial Port Connector . . . . . . . . . . . . . . . 86
Figure A-4 Pin Number Locations for USB Type A Connector . . . . . . . 88
Figure B-1 VCCI Notice (Japan Only) . . . . . . . . . . . . . 93
Figure B-2 Korean Class A Regulatory Notice . . . . . . . . . . . 93

007-5832-002 xiii
List of Tables
Table 4-1 Tall Rack Technical Specifications . . . . . . . . . . . 56
Table 5-1 External NUMAlink Technical Specifications . . . . . . . . 60
Table 6-1 SGI UV 2000 PCIe Support Levels . . . . . . . . . . . 64
Table 6-2 PCIe Expansion Slot Bandwidth Support Levels . . . . . . . 65
Table 7-1 Troubleshooting Chart . . . . . . . . . . . . . . 72
Table 7-2 Power Supply LED States . . . . . . . . . . . . . 73
Table A-1 SGI UV 2000 System Configuration Ranges . . . . . . . . 79
Table A-2 SGI UV 2000 Physical Specifications . . . . . . . . . . 80
Table A-3 Environmental Specifications . . . . . . . . . . . . 81
Table A-4 Power Specifications . . . . . . . . . . . . . . . 82
Table A-5 VGA Pin Functions . . . . . . . . . . . . . . . 84
Table A-6 Ethernet Pinouts . . . . . . . . . . . . . . . . 85
Table A-7 Serial Port Pinout. . . . . . . . . . . . . . . . 87
Table A-8 Pin Assignments for USB Type A Connector . . . . . . . . 88

007-5832-002 xv
About This Guide
This guide provides an overview of the architecture, general operation and descriptions of the
major components that compose the SGI UV 2000 family of servers.It also providesthe standard
procedures for powering on and powering off the system, basic troubleshooting and maintenance
information, and important safety and regulatory specifications.
Audience
This guide is written for owners, system administrators, and users of SGI UV 2000 computer
systems. It is written with the assumption that the reader has a good working knowledge of
computers and computer systems.
Important Information
Warning: To avoid problems that could void your warranty, your SGI or other approved
system support engineer (SSE) should perform all the set up, addition, or replacement of
parts, cabling, and service of your SGI UV 2000 system, with the exception of the following
items that you can perform yourself:
• Using your system console controller to enter commands and perform system functions such
as powering on and powering off, as described in this guide.
• Adding and replacing PCIe cards, as described in this guide.
• Adding and replacing disk drives in dual-disk enabled riser blades.
• Removing and replacing the IRU power supplies.
• Using the On/Off switch and other switches on the rack PDUs.
• Using the ESI/ops panel (operating panel) on optional mass storage bricks.

xvi 007-5832-002
About This Guide
Chapter Descriptions
The following topics are covered in this guide:
• Chapter 1, “Operation Procedures,” provides instructions for powering on and powering off
your system.
• Chapter 2, “System Control,” describes the function of the overall system control network
interface and provides basic instructions for operating the controllers.
• Chapter 3, “System Overview,” provides technical overview information needed to
understand the basic functional architecture of the SGI UV 2000 systems.
• Chapter 4, “Rack Information,” describes the rack sizes and general features.
• Chapter 5, “Optional Octal Router Chassis Information,” describes the optional NUMAlink
router technology abailable in SGI UV 2000 systems consisting of two or more racks. This
router technology is available in an enclosure “package” known as the Octal Router Chassis.
• Chapter 6, “Add or Replace Procedures,” provides instructions for installing or removing the
customer-replaceable components of your system.
• Chapter 7, “Troubleshooting and Diagnostics,” provides recommended actions if problems
occur on your system.
• Appendix A, “Technical Specifications and Pinouts‚" provides physical, environmental, and
power specifications for your system. Also included are the pinouts for the non-proprietary
connectors.
• Appendix B, “Safety Information and Regulatory Specifications‚" lists regulatory
information related to use of the UV 2000 system in the United States and other countries. It
also provides a list of safety instructions to follow when installing, operating, or servicing
the product.

Related Publications
007-5832-002 xvii
Related Publications
The following SGI documents are relevant to the UV 2000 series system:
•SGI UV CMC Software User Guide
(P/N 007-5636-00x)
This guide describes how to use the system console controller commands to monitor and
manage your SGI UV 2000 system via line commands. Coverage of control includes
descriptions of the interface and usage of the commands. These commands are primarily
used when a system management node is not present in the system. Note that it does not
cover controller command information for the SGI UV 10 or UV 20.
• SGI UV System Management Node Administrator's Guide
(P/N 007-5694-00x)
This guide covers the system management node (SMN) for SGI UV 2000 series systems. It
describes the software and hardware components used with the SMN as well as providing an
overview of the UV system control network. System network addressing is covered and a
chapter on how to use KVM to enable remote console access from the system management
node is included.
•SGI Management Center Quick Start Guide
(P/N 007-5672-00x)
This document may be helpful to users or administrators of SGI UV systems using the SGI
Management Center interface. The guide provides introductory information on
configuration, operation and monitoring of your UV system using the management center
software.
•SGI Management Center System Administrator’s Guide
(P/N 007-5642-00x)
This guide is intended for system administrators who work with the SGI Management
Center software GUI to manage and control SGI UV 2000 systems. Depending on your
system configuration and implementation, this guide may be optional. The manual is written
with the assumption the user has a good working knowledge of Linux.
• SGI UV Software Install Guide
(P/N 007-5675-00x)
In UV systems that come with pre-installed Linux software operating systems; this
document describes how to re-install it when necessary.

xviii 007-5832-002
About This Guide
•SGI UV Systems Linux Configuration and Operations Guide
(P/N 007-5629-00x)
This guide is a reference document for people who manage the operation of SGI UV 2000
systems. It explains how to perform general system configuration and operation under Linux
for SGI UV. For a list of manuals supporting SGI Linux releases and SGI online resources,
see the SGI Performance Suite documentation.
•SGI UV Systems Installation Guide
(P/N 007-5675-00x)
This guide covers software installation on UV 2000 systems and their SMNs.
•Linux Application Tuning Guide for SGI X86-64 Based Systems
(P/N 007-5646-00x)
This guide includes a chapter that covers advanced tuning strategies for applications running
on SGI UV systems as well as other SGI X86 based systems.
•MegaRAID SAS Software User’s Guide, publication number (860-0488-00x)
This document describes the LSI Corporation’s MegaRAID Storage Manager software.
•LSI Integrated SAS for RAID User’s Guide, publication number (860-0476-00x)
This user guide explains how to configure and use the software components of the LSI
Integrated RAID software product used with LSI SAS controllers.
• Man pages (online)
Man pages locate and print the titled entries from the online reference manuals.
You can obtain SGI documentation, release notes, or man pages in the following ways:
• See the SGI Technical Publications Library at http://docs.sgi.com
Various formats are available. This library contains the most recent and most comprehensive
set of online books, release notes, man pages, and other information.
• The release notes, which contain the latest information about software and documentation in
this release, are in a file named README.SGI in the root directory of the SGI ProPack for
Linux Documentation CD.
• You can also view man pages by typing man <title> on a command line.
SGI systems shipped with Linux include a set of Linux man pages, formatted in the standard
UNIX “man page” style. Important system configuration files and commands are documented on
man pages. These are found online on the internal system disk (or DVD) and are displayed using

Conventions
007-5832-002 xix
the man command. References in the documentation to these pages include the name of the
command and the section number in which the command is found. For example, to display a man
page, type the request on a command line:
man commandx
For additional information about displaying man pages using the man command, see man(1). In
addition, the apropos command locates man pages based on keywords. For example, to display
a list of man pages that describe disks, type the following on a command line:
apropos disk
For information about setting up and using apropos, see apropos(1).
Conventions
The following conventions are used throughout this document:
Convention Meaning
Command This fixed-space font denotes literal items such as commands, files,
routines, path names, signals, messages, and programming language
structures.
variable The italic typeface denotes variable entries and words or concepts being
defined. Italic typeface is also used for book titles.
user input This bold fixed-space font denotes literal items that the user enters in
interactive sessions. Output is shown in nonbold, fixed-space font.
[ ] Brackets enclose optional portions of a command or directive line.
... Ellipses indicate that a preceding element can be repeated.
man page(x) Man page section identifiers appear in parentheses after man page names.
GUI element This font denotes the names of graphical user interface (GUI) elements such
as windows, screens, dialog boxes, menus, toolbars, icons, buttons, boxes,
fields, and lists.

xx 007-5832-002
About This Guide
Product Support
SGI provides a comprehensive product support and maintenance program for its products, as
follows:
• If you are in North America, contact the Technical Assistance Center at
[phone]SGI or contact your authorized service provider.
• If you are outside North America, contact the SGI subsidiary or authorized distributor in
your country. International customers can visit http://www.sgi.com/support/
Click on the “Support Centers” link under the “Online Support” heading for information on
how to contact your nearest SGI customer support center.
Reader Comments
If you have comments about the technical accuracy, content, or organization of this document,
contact SGI. Be sure to include the title and document number of the manual with your comments.
(Online, the document number is located in the front matter of the manual. In printed manuals, the
document number is located at the bottom of each page.)
You can contact SGI in any of the following ways:
• Send e-mail to the following address: [e-mail]
• Contact your customer service representative and ask that an incident be filed in the SGI
incident tracking system.
• Send mail to the following address:
Technical Publications
SGI
46600 Landing Parkway
Fremont, California 94538
SGI values your comments and will respond to them promptly.