HP Cluster Platform Introduction v2010 User manual

HP Cluster Platform
Workgroup System Tower Hardware
Installation Guide
HP Part Number: A-CPWST-1B
Published: February 2008

©Copyright 2008 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP
shall not be liable for technical or editorial errors or omissions contained herein.
Voltaire™ is a registered trademark and Grid Backbone™ and Grid Switch™ are trademarks or registered trademarks of Voltaire Ltd. or its
subsidiaries in the United States and other countries.
InfiniBand™ is a trademark of the InfiniBand® Trade Association.
AMD, the AMD Arrow logo and combinations thereof, and AMD Opteron™ are trademarks of Advanced Micro Devices, Inc.
Intel® and Xeon™ are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
Cisco is a registered trademark of Cisco Systems Inc.
Mellanox is a registered trademark of Mellanox Technologies, Inc. and ConnectX, InfiniHost, and InfiniScale are trademarks of Mellanox
Technologies, Inc.
Microsoft® and Windows are U.S. registered trademarks of Microsoft Corporation.
Red Hat is a trademark of Red Hat, Inc.
SUSE is a trademark of Novell, Inc. in the United States and other countries.

Table of Contents
About This Document.........................................................................................................9
Intended Audience.................................................................................................................................9
New and Changed Information in This Edition.....................................................................................9
Document Organization.........................................................................................................................9
Typographic Conventions.....................................................................................................................10
Related Information..............................................................................................................................10
HP Encourages Your Comments..........................................................................................................10
Documentation Updates and Release Notes........................................................................................11
1 Cluster Platform Workgroup System Tower Overview..............................................13
1.1 Workgroup System Tower Views....................................................................................................14
1.2 High Performance Computing – HP Cluster Platform Architecture..............................................15
1.2.1 HP GbE2c Ethernet Blade Switch and the Optional InfiniBand Network..............................16
1.2.1.1 HP GbE2c Ethernet Switch..............................................................................................16
1.2.1.2 HP 4X DDR InfiniBand Interconnect, and InfiniBand Mezzanine HCAs......................18
1.2.2 HP Onboard Administrator and Workgroup System Tower Documentation........................19
2 Workgroup System Tower Pre-installation Procedures..............................................21
2.1 Pre-installation Procedures.............................................................................................................21
2.2 Site Recommendations....................................................................................................................21
2.3 Power Examples..............................................................................................................................22
3 Unpacking and Removing a Workgroup System Tower from a Pallet....................23
4 Installation and Start-up...............................................................................................27
4.1 Postinstallation and Diagnostics.....................................................................................................27
4.2 Setting Up Local Networks.............................................................................................................27
4.2.1 Connecting to an External Network........................................................................................27
4.2.2 Configuring a VLAN for the Administrative Network..........................................................28
4.3 Operating System Information........................................................................................................29
A Workgroup System Tower High-Line Power Examples.............................................31
B Configuration Rules for Workgroup System Tower...................................................39
B.1 Workgroup System Tower Device and Interconnect Bay Rules.....................................................39
B.2 Workgroup System Tower Fan Bays...............................................................................................39
Index.................................................................................................................................41
Table of Contents 3

4

List of Figures
1-1 Workgroup System Tower Example Front View..........................................................................14
1-2 Workgroup System Tower Example Rear View............................................................................15
1-3 HP GbE2c Ethernet Blade Switch..................................................................................................17
1-4 4X DDR InfiniBand Interconnect...................................................................................................18
3-1 Open the Top of the Cardboard Box.............................................................................................23
3-2 Remove the Box.............................................................................................................................24
3-3 Remove the Ramp and Front Cushion..........................................................................................24
3-4 Attach the Ramp............................................................................................................................25
3-5 Roll the Unit Down the Ramp.......................................................................................................26
3-6 Roll the Unit to the Installation Location......................................................................................26
4-1 CP Workgroup System Tower with a Server Blade Control Node...............................................28
A-1 Example Configuration 1: 4kVA, N+N Redundant Enclosure with a 32A International PDU....32
A-2 Example Configuration 2: 2.7kVA, N+N Redundant Enclosure with 16A PDU, C20-C13 Power
Cords (not supplied by HP)..........................................................................................................33
A-3 Example Configuration 3: 3kVA, N+1 Redundant Enclosure with a 16A PDU, C20-C13 cords
(not supplied by HP).....................................................................................................................34
A-4 Example Configuration 4: 5kVA - 7kVA, N+1 Redundant Enclosure with 32A PDU, and C13
Extension Bars...............................................................................................................................35
A-5 Example Configuration 5: 5kVA, N+N Redundant Enclosure with 16A 3-Phase PDU, C20-C13
Power Cords (not supplied by HP)...............................................................................................36
A-6 Example Configuration 6: 5kVA, N+1 Redundant Enclosure with 16A 3-Phase PDU, and
C20-C13 Power Cords (not supplied by HP)................................................................................37
5

6

List of Tables
1-1 HP GbE2c Ethernet Blade Switch for c-Class BladeSystem User and Service Documentation....17
7

8

About This Document
This document provides preparation tasks and hardware installation information for the HP
Cluster Platform Workgroup System Tower.
Workgroup System Tower configurations are easy-to-order, preconfigured, single c-Class
enclosure (c3000) cluster solutions supporting either InfiniBand or gigabit Ethernet system
interconnects. When ordering a Workgroup System Tower, you can specify installation support
from HP Global Services, or complete the onsite installation procedures yourself. Workgroup
System Tower is offered within the Cluster Platform Express portfolio. For more information on
HP Cluster Platform Express systems, go to:
http://h20311.www2.hp.com/HPC/cache/276453-0-0-0-121.html
Intended Audience
This document is intended for customers and system administrators experienced in the use of
high-performance clusters. Note that certain operations described in this document, if performed
incorrectly, might cause system crashes and loss of data. If you are not familiar with installing
and using high-performance clusters, HP recommends that you contact HP Global Services.
This document is also intended for HP service representatives and other persons trained to install
High-Performance Computing Cluster Platform products. Such persons are expected to
understand the hazards of working in this environment and to take suitable precautions to
minimize danger to themselves and others.
Note:
For the HP Cluster Platform Express Workgroup System version which is a c3000 enclosure
rack-mounted in an HP 10K Series 22U rack, go to:
http://www.docs.hp.com/en/A-CPCPE-1D/A-CPCPE-1D-2-27-08.pdf
New and Changed Information in This Edition
The following table provides the revision history of this document.
Changed InformationDocument Revision History
New document released.February 2008
Revisions added from specifications.February 2008
Document Organization
This document is organized as follows:
Chapter 1 Provides an overview of the HP Cluster Platform Workgroup System Tower.
Chapter 2 Provides Workgroup System Tower pre-installation procedures, site
recommendations, and power examples.
Chapter 3 Provides procedures for unpacking and removing Workgroup System Tower
from a pallet.
Chapter 4 Provides HP Cluster Platform Workgroup System Tower hardware installation
procedures, and postinstallation and diagnostics procedures.
Appendix A Provides several power examples for the Workgroup System Tower.
Appendix B Provides the configuration rules for the Workgroup System Tower.
Intended Audience 9

Typographic Conventions
This document uses the following typographic conventions:
Reader Notes
NOTE: Content of the note.
Notes provide additional information, supplementing the adjoining
content or emphasizing points of information.
Reader Warnings
WARNING! Content of the warning.
A warning calls attention to important information that if not understood
or followed will result in personal injury or nonrecoverable system
problems.
Reader Cautions
CAUTION: Content of the caution.
A caution calls attention to important information that if not understood
or followed will result in data loss, data corruption, or damage to
hardware or software.
Reader Cautions
IMPORTANT: Content of the important information.
This alert provides essential information to explain a concept or to
complete a task.
Related Information
The HP Unified Cluster Portfolio offers comprehensive modular solutions utilizing
industry-standard servers, with a choice of operating environments. For more information, go
to:
http://h20311.www2.hp.com/HPC/cache/274276-0-0-0-121.html
You can find the latest versions of the HP Cluster Platform Express documentation at:
http://docs.hp.com/en/highperfcomp.html
You can also find Voltaire InfiniBand documentation at:
http://www.voltaire.com/
See the Products, Grid Backbone and Grid Switches topic. To access proprietary information
(such as drivers and firmware), you will need the appropriate login credentials, which are
available from Voltaire.
The latest Cisco hardware and software documentation is available on the Cisco website at:
http://www.cisco.com/
HP Encourages Your Comments
HP encourages your comments concerning this document. We are committed to providing
documentation that meets your needs. Send any errors found, suggestions for improvement, or
compliments to:
http://docs.hp.com/en/feedback.html
Include the document title, manufacturing part number, and any comment, error found, or
suggestion for improvement you have concerning this document.
10

Documentation Updates and Release Notes
Documentation updates and release notes (if applicable) are provided on the HP High Performance
Computing documentation website:
http://www.docs.hp.com/en/highperfcomp.html
Use the release date of a document to determine that you have the latest version.
Documentation Updates and Release Notes 11

12

1 Cluster Platform Workgroup System Tower Overview
The HP Cluster Platform Workgroup System Tower is delivered to you factory assembled and
ready for deployment. It is available with a choice of pre-installed software for cluster
management. The extensive array of software tools featured in the Unified Cluster Portfolio are
supported on the Workgroup System Tower. For more information, go to the Unified Cluster
Portfolio overview:
http://h20311.www2.hp.com/HPC/cache/275420-0-0-225-121.html?jumpid=reg_R1002_USEN
Workgroup System Tower solutions offer a choice of compute nodes, with HP ProLiant BL460c
server blades with Intel® Xeon™ processors or HP ProLiant BL465c server blades featuring AMD
Opteron™ processors. One server blade in the configuration is designated as the control node,
which you can configure separately from the remaining server blades in the enclosure. The
control node can also be used for preprocessing, postprocessing, and computational workload.
Note:
All of the other server blades in the enclosure are configured identically. Within the Workgroup
System Tower server blade (c3000) enclosure, all of the processor subsystems are identical; that
is, all of the processors are of the same type and frequency and they have the same memory
configuration.
The Workgroup System Tower ships with all of the modules installed and may also have the
optional 4X double-data rate (DDR) InfiniBand interconnect module installed for high-bandwidth,
low-latency message passing. If this network is in your configuration, all of the server blades
must be connected to the InfiniBand network.
Software options include Red Hat and SUSE Linux distributions, as well as the HP
message-passing library, HP-MPI. For Windows environments, clusters are available with the
Microsoft Windows Compute Cluster Server (CCS).
For information on the software available for the Workgroup System Tower, go to:
http://h18000.www1.hp.com/products/quickspecs/12790_div/12790_div.PDF
The Workgroup System Tower is comprised of a single-cabinet cluster containing a variety of
components installed to meet the following specifications:
• A single onboard administrator (OA) module
• Up to eight half-height server blades
• A shared administration/gigabit Ethernet network
• Configurations within one c3000 BladeSystem enclosure
• An optional InfiniBand interconnect
Cluster management in the Workgroup System Tower configuration is supported over an
administration network with both console and operating system-level management branches
sharing the gigabit Ethernet network established by the HP GbE2c switch module.
The following components or features are not supported in Workgroup System Tower:
• External storage
• Multiple control nodes
• Dual onboard administrators
• High-performance interconnects (such as Myrinet and Quadrics)
• Full-height server blades
For configuration information on factory integrated models, features and specifications, go to
the Workgroup System Tower QuickSpecs on the HP website:
http://h18000.www1.hp.com/products/quickspecs/12790_div/12790_div.PDF
For an overview on HP Cluster Platform architecture, see Section 1.2.
13

1.1 Workgroup System Tower Views
The Workgroup System Tower ships with the server blades and all of the factory integrated
modules installed. Figure 1-1 shows an example front view of the Workgroup System Tower.
Note:
HP Cluster Platform Workgroup System is available from HP Sales Representatives and
HPC-Certified Resellers, utilizing Cluster Platform Express configurators. Quotes can be requested
by submitting a form available at the following web site:
http://h20311.www2.hp.com/HPC/cache/276453-0-0-0-121.html?jumpid=go/cp-express
Figure 1-1 Workgroup System Tower Example Front View
5
42
3
1
The following list describes the callouts in Figure 1-1:
1. DVD optical drive
2. HP ProLiant BL460c or BL465c half-height server blades (eight in this figure)
3. Redundant OA module slot (reserved for potential future use)
4. HP Insight Display
5. OA module
Figure 1-2 shows an example rear view of the Workgroup System Tower.
14 Cluster Platform Workgroup System Tower Overview

Figure 1-2 Workgroup System Tower Example Rear View
1
3 3
2
4
6
5
7
89
9
10
The following list describes the callouts in Figure 1-2:
1. Local KVM interface
2. HP GbE2c Ethernet switch installed in interconnect module bay 1 (IMB1)
3. Active cool fans
4. Interconnect module bay 2 (reserved for potential future use)
5. Enclosure uplink and service port
6. Enclosure downlink
7. iLO/Onboard Administrator port 1
8. iLO/Onboard Administrator port 2 (reserved for future use)
9. Power supplies
10. Optional 4X DDR InfiniBand Interconnect installed in interconnect module bays 3 and 4
(IMB3/4)
1.2 High Performance Computing – HP Cluster Platform Architecture
What makes HP Cluster Platform different from standard systems is the assignment of node
types, high-speed networks, and the interoperability efficiency gained by using High Performance
Computing Software manufactured by HP and other HP partners. This section describes the
basics of the HP Cluster Platform hardware architecture.
•Nodes: The two types of nodes included in a typical Cluster Platform configuration are:
—Control node: Each cluster has one control node. If the optional InfiniBand interconnect
is selected, the control node is also connected. The control node can also be used for
preprocessing, postprocessing, and computational workload.
—Compute nodes: Compute nodes are normally used for application computation rather
than administrative workloads.
1.2 High Performance Computing – HP Cluster Platform Architecture 15

Note:
Control and compute nodes must belong to the same processor family. For example, control
nodes that use AMD Opteron processors must be used if you are using the ProLiant BL465c
as the compute node.
•Networks: The networks included in a Workgroup System Tower configuration are:
—Administrative network: An administrative network eases cluster administration and
management functions, such as managing a user's applications.
—Console network: The console network enables the use of node-specific commands such
as power one. Each compute node embedded Integrated Lights-Out (iLO) port is
connected to the Fast Ethernet-based console network.
Note:
The administrative HP GbE2c Gigabit Ethernet switch is integrated in the Workgroup
System Tower. Access to the console network is provided by the iLO port on the Onboard
Administrator (OA).
—Optional InfiniBand interconnect network: An optional InfiniBand interconnect
high-performance network is dedicated to supporting inter nodal application traffic
(such as MPI communication) and you can also select file transfer. In the absence of an
InfiniBand interconnect, this traffic is transmitted over the gigabit Ethernet network
(Administrative network).
•Optical media device: Access to a DVD drive is required for software installation. The
Workgroup System Tower has a DVD drive installed.
For more information about hardware and software used in HP Cluster Platform and throughout
the Unified Cluster Portfolio, go to:
http://h20311.www2.hp.com/HPC/cache/276360-0-0-0-121.html
1.2.1 HP GbE2c Ethernet Blade Switch and the Optional InfiniBand Network
Your Workgroup System Tower includes the HP GbE2c Ethernet switch and might also include
an optional InfiniBand high-speed network. Section 1.2.1.1 describes the HP GbE2c Ethernet
Blade Switch, and Section 1.2.1.2 describes the 4X DDR InfiniBand Interconnect module used for
the optional InfiniBand network.
1.2.1.1 HP GbE2c Ethernet Switch
The HP GbE2c Ethernet switch, which is also documented in the HP Cluster Platform Gigabit
Ethernet Interconnect Guide, is used for administrative and console networking in Workgroup
System Tower solutions. You can also use this network for Message Passing Interface (MPI)
applications.
Cluster management on server blade configurations is supported over an administration network
with both console and operating-system level management branches. These branches share a
physical switch that can be separated using VLANs, if required. In the Workgroup System Tower,
one server blade in the configuration is designated as the control node and can be configured
separately from the others in the cluster. All of the other server blades in the Workgroup System
Tower must be configured identically and the processor subsystems must be identical. Also, all
of the server blade processors must be of the same type and frequency and must have the same
memory configuration.
By default, Workgroup System Tower configurations are available with the gigabit Ethernet
interconnect using the HP GbE2c Ethernet Blade switch.
16 Cluster Platform Workgroup System Tower Overview

The HP GbE2c switch is installed in interconnect module bay 1 (IMB1) of the Workgroup System
Tower. It can be used as the system interconnect and also provides the infrastructure for the
administration and console network in Workgroup System Tower configurations.
Figure 1-3 shows an HP GbE2c Ethernet Blade Switch for c-Class BladeSystem.
Figure 1-3 HP GbE2c Ethernet Blade Switch
132 45
6
The following list describes the callouts in Figure 1-3:
1. Reset button
2. Health LED
3. UID LED
4. DB-9 serial management port
5. RJ-45 1Gb uplink ports
6. Release handle
Note:
If you do not choose the optional InfiniBand interconnect, you can use the In-Band Management
(administrative network) for MPI applications.
For user and service documentation about the HP GbE2c Ethernet Blade Switch for c-Class
BladeSystem, see Table 1-1.
Table 1-1 HP GbE2c Ethernet Blade Switch for c-Class BladeSystem User and Service Documentation
URLDocument Title
http://h18004.www1.hp.com/products/blades/components/ethernet/GbE2c/index.htmlHP GbE2c Ethernet Blade Switch for
c-Class BladeSystem - overview
http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c00700745/c00700745.pdfGbE2c Ethernet Blade Switch for HP
c-Class Quick Setup Instructions
http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c00701973/c00701973.pdfHP GbE2c Ethernet Blade Switch for
c-Class BladeSystem Application Guide
http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c00701566/c00701566.pdfHP GbE2c Ethernet Blade Switch for
c-Class BladeSystem Browser-based
Interface Reference Guide
http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c00701969/c00701969.pdfHP GbE2c Ethernet Blade Switch for
c-Class BladeSystem Command Reference
Guide
http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c00701541/c00701541.pdfHP GbE2c Ethernet Blade Switch for
c-Class BladeSystem User Guide
http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c00714237/c00714237.pdfHP BladeSystem c7000 Enclosure
Maintenance and Service Guide
1.2 High Performance Computing – HP Cluster Platform Architecture 17

1.2.1.2 HP 4X DDR InfiniBand Interconnect, and InfiniBand Mezzanine HCAs
The HP 4X DDR (dual date rate) InfiniBand interconnect for HP BladeSystem c-Class is a
double-wide interconnect module for the c-Class enclosure. It is based on the Mellanox InfiniScale
III 4X DDR InfiniBand switch chip. The 4X DDR InfiniBand interconnect for HP BladeSystem
c-Class provides 24 InfiniBand 4X DDR ports with 20 Gbps port-to-port connectivity. The ports
are arranged as 16 downlinks to connect up to 16 server blades in the enclosure (up to eight in
the Workgroup System Tower), and 8 uplinks to connect to the external InfiniBand switches to
build an InfiniBand fabric, if applicable. All links conform to InfiniBand Trade Association (IBTA)
specifications.
Figure 1-4 shows the optional HP 4X DDR InfiniBand module being installed in a c-Class (c7000
shown) enclosure. The process to install the module is the same for the Workgroup System Tower,
except it is installed in interconnect module bays 3 and 4 (IMB3/4) instead. Workgroup System
Tower configurations are shipped with all of the modules installed, and this figure is provided
for reference purposes only.
Figure 1-4 4X DDR InfiniBand Interconnect
1
2
The following list describes the callouts shown in Figure 1-4:
1. 4X DDR InfiniBand interconnect
2. Latch
Note:
When the 4X DDR InfiniBand interconnect module is used in a Workgroup System Tower, it is
installed in interconnect module bays 3 and 4.
InfiniBand HCAs
The HP 4X DDR InfiniBand dual-port mezzanine HCA (host channel adapter) for HP BladeSystem
c-Class is based on ConnectX InfiniBand technology from Mellanox. The InfiniBand dual-port
mezzanine HCA delivers low-latency and high-bandwidth for High-Performance Computing.
The 4X DDR InfiniBand dual-port mezzanine HCA can be plugged into the PCI-Express connectors
on the server blades. To obtain the best performance, HP recommends that the InfiniBand
18 Cluster Platform Workgroup System Tower Overview

mezzanine HCA be plugged into the x8 PCI-Express connector on the server blade. The InfiniBand
dual-port mezzanine HCA is connected to the 4X DDR InfiniBand interconnect module that is
installed in interconnect module bays 3 and 4 of the Workgroup System Tower through the c3000
enclosure midplane.
HP also offers a single-port 4X DDR InfiniBand mezzanine PCI-Express HCA for c-Class server
blades.
Subnet Management
A subnet manager is required to manage and control an InfiniBand fabric. The subnet manager
functionality can be provided by a host-based subnet manager software on a server blade. HP
supports the host-based subnet managers Voltaire GridVision BladeFM, and the Cisco High
Performance Subnet Manager (HSM) for the subnet management service. HP supports Voltaire
Grid Switch family products with embedded fabric manager, Cisco SFS 7000D 24-port switch
with embedded subnet manager, and host-based subnet managers the Voltaire GridVision
BladeFM and the Cisco High Performance Subnet Manager (HSM) for the subnet management
service. The Voltaire GridVision BladeFM and the Cisco High-Performance Subnet Manager run
on Linux server blades.
RedHat and SuSE Linux distributions are supported using a host-based subnet manager (on a
server blade). Microsoft Windows Compute Cluster Server is supported on Voltaire GridStack
for the single-port mezzanine HCA.
OpenFabrics
You can also use OpenSM from OpenFabric Enterprise Distribution to establish an InfiniBand
fabric, however, support for OpenSM is only available from the OpenFabrics community. For
more information about OpenSM, go to the OpenFabrics website:
http://www.openfabrics.org/
For more information on the 4X DDR InfiniBand interconnect, InfiniBand mezzanine HCAs,
subnet management, and drivers go to the QuickSpecs HP InfiniBand Options for HP BladeSystems
c-Class:
http://h18004.www1.hp.com/products/quickspecs/12586_na/12586_na.pdf
1.2.2 HP Onboard Administrator and Workgroup System Tower Documentation
The HP Onboard Administrator (OA) provides a single point from which to perform basic
management tasks on server blades or switches within the enclosure. The HP Onboard
Administrator performs initial configuration steps for the enclosure, allows for run-time
management and configuration of the enclosure components, and informs you of problems
within the enclosure through email, SNMP, or the Insight Display.
HP recommends that you read the HP BladeSystem c3000 Enclosure User Guide before moving on
to HP BladeSystem Onboard Administrator setup. Reading this information in this order ensures
that you will obtain an overall understanding of the HP BladeSystem Onboard Administrator,
and that you properly complete the initial setup to facilitate proper functioning of the OA.
For various documentation on Workgroup System Tower, including setup and installation,
troubleshooting, service and maintenance, and the specific embedded modules, such as the OA,
go to:
http://h18004.www1.hp.com/products/blades/components/enclosures/c-class/c3000/
Choose Support & Documents in the upper right hand corner and then choose »Manuals.
1.2 High Performance Computing – HP Cluster Platform Architecture 19

20