HP Cluster Platform Express v2010 User manual

HP Cluster Platform
InfiniBand Interconnect Installation and
User's Guide
HP Part Number: A-CPIBI-1E
Published: October 2006

©Copyright 2008 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP
shall not be liable for technical or editorial errors or omissions contained herein. Intel, Pentium, Intel Inside, and the Intel Inside logo are trademarks
or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
Mellanox is a registered trademark of Mellanox Technologies.
Topspin is a registered trademark and the Topspin logo, TopspinOS, Topspin Switched Computing System, Grid-to-Go and VFrame are trademarks
of Topspin Communications, Inc., owned by Cisco Systems Inc.
Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Windows Server 2003 and Compute Cluster Server 2003 are
U.S. trademarks of Microsoft Corporation.
Voltaire™ and the V logo, V™, are registered trademarks of Voltaire Inc.
ISR 9288™ is a registered trademark of Voltaire Inc.
ISR 9024™ and GridVision™ are registered trademarks of Voltaire Inc.
ISR 9096™ is a registered trademark of Voltaire Inc.
VoltaireVision™ is a registered trademark of Voltaire Inc.
Voltaire HCA 400ExV and Voltaire HCA 400V are registered trademarks of Voltaire Inc.
InfiniBand™ is a trademark the InfiniBand® Trade Association
Microsoft®, Windows®, and Windows Server® are U.S. registered trademarks of Microsoft Corporation

Table of Contents
About This Manual..........................................................................................................11
Organization.........................................................................................................................................11
Audience...............................................................................................................................................12
Related Documentation........................................................................................................................12
Voltaire Documentation...................................................................................................................12
HP Cluster Platform Documentation..............................................................................................12
Cluster Component Documentation...............................................................................................13
HP Encourages Your Comments..........................................................................................................13
Text Annotations...................................................................................................................................13
Conventions..........................................................................................................................................14
Safety Considerations.....................................................................................................15
Information on Ergonomics..................................................................................................................16
Recycling...............................................................................................................................................17
1 InfiniBand Technology Overview................................................................................19
1.1 Identifying InfiniBand Fabric Components....................................................................................20
1.2 ISR 9xxxx-series Interconnects.......................................................................................................21
1.3 Identifying the ISR 9024 Interconnect (SDR)...................................................................................22
1.4 Identifying the ISR 9024 S/D Interconnect (RoHS Compliant) ......................................................22
1.5 Identifying the ISR 9096 Interconnect.............................................................................................23
1.6 Identifying the ISR 9288 Interconnect.............................................................................................24
1.7 Understanding the Interconnect Management Capabilities...........................................................26
2 Installing and Maintaining the ISR 9024 Interconnect.............................................29
2.1 Overview of the ISR 9024 Interconnect...........................................................................................29
2.1.1 Internally Managed ISR 9024 Front Panel...............................................................................30
2.1.2 ISR 9024 Management Card....................................................................................................31
2.1.3 Externally Managed ISR 9024 Front Panel..............................................................................32
2.1.4 ISR 9024 Rear Panel.................................................................................................................32
2.1.5 Cooling and Airflow................................................................................................................33
2.2 Unpacking the ISR 9024 Interconnect.............................................................................................33
2.3 Mounting the ISR 9024 in the Rack.................................................................................................34
2.4 Installing the ISR 9024 Power Supply Unit.....................................................................................36
2.5 Operating the ISR 9024 Interconnect...............................................................................................36
2.6 Troubleshooting the ISR 9024..........................................................................................................36
3 Installing and Maintaining the ISR 9024 S/D Interconnect (RoHS Compliant)....39
3.1 Overview of the ISR 9024 S/D Interconnect....................................................................................39
3.1.1 Internally Managed ISR 9024 S/D–M Front Panel...................................................................40
3.1.1.1 ISR 9024S/D-M Management Card.................................................................................41
3.1.2 Externally Managed ISR 9024 S/D Front Panel.......................................................................41
3.1.3 ISR 9024 S/D Rear Panel..........................................................................................................42
3.2 Unpacking the ISR 9024 S/D Interconnect.......................................................................................43
3.3 Mounting the ISR 9024 S/D in the Rack..........................................................................................44
3.4 The ISR 9024 S/D Fan Unit..............................................................................................................45
3.4.1 Replacing the Fan Unit............................................................................................................46
3.5 Installing the ISR 9024 S/D Power Supply Units.............................................................................46
Table of Contents 3

3.6 Operating the ISR 9024 S/D Interconnect........................................................................................47
3.7 Troubleshooting the ISR 9024 S/D...................................................................................................47
4 Installing and Maintaining the ISR 9096 and ISR 9288 Interconnects..................49
4.1 Overview of the ISR 9XXX Interconnects........................................................................................49
4.1.1 ISR ISR 9XXX Interconnect Features.......................................................................................49
4.1.2 ISR ISR 9XXX Interconnect Architecture.................................................................................50
4.1.3 ISR ISR 9XXX Building Blocks.................................................................................................50
4.1.3.1 Field Replaceable Units...................................................................................................51
4.1.4 ISR ISR 9XXX System Firmware..............................................................................................51
4.1.5 ISR ISR 9XXX Control Panels and Reset Buttons....................................................................52
4.2 Unpacking the Chassis....................................................................................................................52
4.2.1 Unpacking the ISR 9288..........................................................................................................53
4.2.2 Unpacking the ISR 9096..........................................................................................................55
4.2.3 Verifying the Shipment...........................................................................................................56
4.3 Repacking the Chassis.....................................................................................................................58
4.4 Mounting the ISR 9XXX in the Rack...............................................................................................58
4.4.1 Resources Required to Perform an Installation.......................................................................59
4.4.2 Mounting The Chassis in the Rack..........................................................................................59
4.4.3 Installing the Cabling Clamps.................................................................................................63
4.5 Standard Configurations.................................................................................................................63
4.6 Post-installation Tasks.....................................................................................................................67
5 HP 4x DDR IB Switch Module for c-Class BladeSystems Overview........................69
5.1 Installing the 4x DDR IB Switch Module........................................................................................69
5.1.1 Subnet Manager......................................................................................................................70
6 Installing Field-Replaceable Units in the ISR 9XXX Chassis.....................................71
6.1 Guidelines for Inserting and Extracting Boards..............................................................................71
6.2 Installing or Replacing the sFB-12 Fabric Board.............................................................................72
6.3 Installing or Replacing sLB-24 Line Boards....................................................................................73
6.4 Installing or Replacing the sMB (Management) Board...................................................................75
6.5 Installing and Replacing Power Supply Units................................................................................76
6.6 Populating the Router Blade Drawer (sRBD)..................................................................................77
6.6.1 Overview of the IPR (TCP/IP) Blade.......................................................................................77
6.6.2 Overview of the FCR (Fibre Channel Router) Blade...............................................................78
6.6.3 Installing a Router Blade.........................................................................................................79
6.7 Installing or Replacing the Router Blade Drawer (sRBD)...............................................................80
6.8 Installing and Replacing the sCTRL Board.....................................................................................81
6.9 Installing the sFU-4 Fan Module.....................................................................................................82
6.10 Installing the sFU-8 Fan Module...................................................................................................83
7 Cabling the Interconnect.............................................................................................85
7.1 InfiniBand Cabling..........................................................................................................................85
7.1.1 General InfiniBand Cabling Guidelines..................................................................................85
7.1.2 Cabling Tables.........................................................................................................................86
7.1.3 Connecting InfiniBand Cables.................................................................................................86
7.1.4 Cable Routing Procedure for the ISR 9024 or ISR 9024 S/D....................................................86
7.1.5 Cable Routing Procedure for the ISR 9096..............................................................................87
7.1.6 Connecting the ISR 9096 Router Blades..................................................................................88
7.1.6.1 Connecting to IPR GbE Ports..........................................................................................88
7.1.6.2 Connecting to FCR FC Ports............................................................................................88
4 Table of Contents

7.1.7 Cable Routing Procedure for the ISR 9288 .............................................................................88
7.2 Connecting the ISR 9288 Router Blades..........................................................................................89
7.2.1 Connecting to IPR GbE Ports..................................................................................................89
7.2.2 Connecting to FCR FC Ports....................................................................................................90
7.3 Chassis Administrative Connections..............................................................................................90
7.3.1 Cabling the ISR 9024 Management Connections....................................................................90
7.3.2 Cabling the ISR 9096 and the ISR 9288 Management Connections........................................90
7.3.3 Connecting the Management Port Serial Interface.................................................................91
8 Installing and Maintaining HCA Cards and Mezzanines.......................................93
8.1 Overview of the Voltaire HCA 400 PCI–X, and HCA 400 PCI-Express Cards...............................93
8.2 Installing the HCA...........................................................................................................................94
8.3 Topspin/Mellanox PCI-X HCA........................................................................................................97
8.4 Topspin/Mellanox PCI-Express HCA.............................................................................................98
8.5 Mellanox PCI-X HCA (SDR)...........................................................................................................99
8.5.1 Mellanox PCI-X SDR HCA LEDs............................................................................................99
8.6 Mellanox PCI-Express HCA (SDR)...............................................................................................100
8.6.1 Mellanox PCI-Express SDR HCA LEDs................................................................................100
8.7 Mellanox Memory Free PCI-Express HCA (SDR).........................................................................101
8.7.1 LEDs......................................................................................................................................102
8.8 Mellanox Memory Free PCI-Express HCA (DDR)........................................................................102
8.8.1 LEDs......................................................................................................................................103
8.9 Mellanox PCI-Express HCA (DDR)..............................................................................................103
8.9.1 Mellanox PCI-Express DDR HCA LEDs...............................................................................104
8.9.2 HP HPC 4x DDR IB Mezzanine HCA...................................................................................105
8.9.3 Operating System and Software Requirements....................................................................105
9 Postinstallation Troubleshooting and Diagnostics...................................................107
9.1 Postinstallation Troubleshooting...................................................................................................107
9.2 Startup Checks for the ISR 9XXX...................................................................................................107
9.3 Debugging a Fabric Failure by Using Performance Management (PM).......................................108
9.4 Identifying a Leaf or Spine Port Malfunction on an ISR 9XXX.....................................................108
9.5 Detecting a Failed Port..................................................................................................................108
A ISR 9024 Interconnect Specifications......................................................................111
B ISR 9024 S/D Interconnect Specifications (RoHS Compliant)..............................113
C ISR 9096 Interconnect Specifications......................................................................117
D ISR 9288 Interconnect Specifications......................................................................119
E 4x DDR IB Switch Module Specifications................................................................123
F HCA PCI Card Specifications...................................................................................125
F.1 Voltaire HCA–400 PCI Card Specifications...................................................................................125
F.2 Topspin/Mellanox PCI-X HCA Specifications...............................................................................126
F.3 Topspin/Mellanox PCI-Express HCA Specifications.....................................................................127
F.4 Mellanox PCI-X HCA Specifications.............................................................................................128
Table of Contents 5

F.5 Mellanox PCI-Express HCA Specifications...................................................................................128
F.6 Mellanox Memory Free PCI-Express HCA (SDR) Specifications..................................................129
F.7 Mellanox Memory Free PCI-Express HCA (DDR) Specifications.................................................129
F.8 Mellanox PCI-Express DDR HCA Specifications..........................................................................130
F.9 HPC 4x DDR IB Mezzanine HCA Specifications..........................................................................130
Index...............................................................................................................................133
6 Table of Contents

List of Figures
1-1 Typical InfiniBand Network..........................................................................................................21
1-2 ISR 9024 Interconnect Front Panel View.......................................................................................22
1-3 ISR 9024 S/D (RoHS Compliant) Interconnect Front Panel View.................................................23
1-4 ISR9096 Interconnect Front Panel View........................................................................................23
1-5 ISR 9096 Interconnect Rear Panel View.........................................................................................24
1-6 ISR 9288 Interconnect Front Panel View.......................................................................................25
2-1 Internally Managed ISR 9024, Front Panel....................................................................................30
2-2 ISR 9024 Management Card and DB-9 Adapter (Mini USB)........................................................31
2-3 Externally Managed ISR 9024, Front Panel...................................................................................32
2-4 ISR 9024 Rear Panel (InfiniBand Ports).........................................................................................32
2-5 The ISR 9024 Rack Kit....................................................................................................................34
3-1 Internally Managed ISR 9024 D–M, Front Panel...........................................................................41
3-2 Externally Managed ISR 9024 S/D, Front Panel............................................................................42
3-3 ISR 9024 S/D Rear Panel (InfiniBand Ports)..................................................................................42
3-4 ISR 9024 S/D Packaging.................................................................................................................43
3-5 The ISR 9024 S/D Rack Kit.............................................................................................................44
3-6 Replacing the Fan Unit..................................................................................................................46
3-7 Installing the ISR 9024 S/D Power Supply Units...........................................................................46
4-1 Top Compartment in the ISR 9288 Crate.......................................................................................53
4-2 Documentation and CD Location..................................................................................................54
4-3 ISR 9288 Crate Door......................................................................................................................54
4-4 Top Compartment in the ISR 9096 Box.........................................................................................55
4-5 ISR 9096 Accessories Box...............................................................................................................56
4-6 ISR 9288 Front View......................................................................................................................57
4-7 ISR 9288 Rear View........................................................................................................................58
4-8 ISR 9288 Interconnect Rail Kit.......................................................................................................59
4-9 ISR 9XXX Rack Mount Procedure.................................................................................................60
4-10 Telescoping Rail Assembly............................................................................................................60
4-11 Assembling 9XXX Rails.................................................................................................................61
4-12 Attaching the Cable Management Hooks.....................................................................................62
4-13 Installing the Cabling Clamps.......................................................................................................63
4-14 ISR 9288 Chassis Front Panel Configuration.................................................................................64
4-15 Maximum InfiniBand Ports, No sRBD Installed...........................................................................65
4-16 Populated I/O Drawer...................................................................................................................66
4-17 ISR 9096 Chassis Front Panel Configuration.................................................................................66
4-18 ISR 9096 InfiniBand Ports..............................................................................................................67
6-1 Board Ejectors................................................................................................................................72
6-2 The sFB-12 Fabric Board................................................................................................................72
6-3 Line Board Front Panel..................................................................................................................74
6-4 The sMB Management Board........................................................................................................75
6-5 Power Supply Unit........................................................................................................................77
6-6 IPR Blade Front Panel....................................................................................................................78
6-7 FCR Blade Front Panel...................................................................................................................79
6-8 Router Blade Lock Lever...............................................................................................................79
6-9 A Populated sRBD.........................................................................................................................80
6-10 sCTRL Board Front Panel..............................................................................................................81
6-11 sFU-4 Vertical Fan Module............................................................................................................82
6-12 sFU-8 Front Panel..........................................................................................................................83
7-1 InfiniBand Cable Connector Head Shells......................................................................................85
7-2 Routing Model for a Fully Configured ISR9288............................................................................89
8-1 HCA-400 PCI Card........................................................................................................................94
8-2 Typical HCA-400 PCI Card Installation........................................................................................95
7

8-3 Topspin/Mellanox SDR PCI–X HCA.............................................................................................98
8-4 Topspin/Mellanox SDR PCI–Express HCA...................................................................................98
8-5 Mellanox PCI–X HCA...................................................................................................................99
8-6 Mellanox PCI–Express HCA.......................................................................................................100
8-7 Mellanox Memory Free PCI–Express HCA (SDR)......................................................................101
8-8 Mellanox Memory Free PCI–Express HCA (DDR).....................................................................102
8-9 Mellanox PCI–Express HCA (DDR)............................................................................................104
8-10 4x DDR IB Mezzanine HCA........................................................................................................105
8 List of Figures

List of Tables
4-1 ISR 9XXX Standard Configurations..............................................................................................64
6-1 sFB-12 Fabric Board LED Status....................................................................................................73
6-2 Line Board Status LEDs.................................................................................................................74
6-3 sMB Board LEDs............................................................................................................................75
6-4 Power Supply Unit LEDs..............................................................................................................77
6-5 IPR Blade LEDs..............................................................................................................................78
6-6 FCR Blade LEDs............................................................................................................................79
6-7 sCTRL Board LEDs........................................................................................................................81
6-8 sFU-8 Fan Module LEDs ..............................................................................................................84
7-1 InfiniBand Cable Characteristics...................................................................................................85
8-1 HCA-400 Status LEDs...................................................................................................................97
9

10

About This Manual
This manual describes how to install and operate an InfiniBand interconnect as part of an HP
Cluster Platform.
This manual limits detailed information and procedures for cluster components that have their
own documentation, such as the cluster nodes, network switches, and system racks. This manual
contains references to the component documentation, which should be readily available during
the installation procedure.
Organization
An HP Cluster Platform model consists of the following three elements:
• A core of common components such as the rack and power components.
• A number of server models from the same server family, such as HP ProLiant DL-series or
HP Integrity rx-series.
• A cluster high-speed interconnect, such as the Voltaire InfiniBand interconnect documented
in this manual
The information in this guide is organized as follows:
DescriptionChapter
Describes the InfiniBand standard, and its implementation in the supported Voltaire
interconnect models.
Chapter 1: InfiniBand
Technology Overview
Describes how to install, maintain, and operate the 24-port Voltaire interconnect model
ISR 9024.
Chapter 2: Installing and
Maintaining the ISR 9024
Interconnect
Describes how to install, maintain, and operate the 24-port Voltaire interconnect model
ISR 9024 S/D (RoHS Compliant).
Chapter 3: Installing and
Maintaining the ISR 9024
S/D Interconnect (RoHS
Compliant)
Describes how to install, maintain, and operate the 96-port Voltaire interconnect model
ISR 9096 and the 288-port Voltaire interconnect model ISR 9288.
Chapter 4: Installing and
Maintaining the ISR 9096
and ISR 9288 Interconnects
Describes how to install field–replaceable units (modules) in the 288-port Voltaire
interconnect model ISR 9288.
Chapter 6: Installing
Field-Replaceable Units in
the ISR 9XXX Chassis
Describes how make cable connections between the interconnect and the cluster nodes
or other HCA destinations.
Chapter 7: Cabling the
Interconnect
Describes how to install, maintain, and operate the various PCI HCA cards used in HP
Cluster Platform.
Chapter 8: Installing and
Maintaining HCA Cards
and Mezzanines
Describes basic troubleshooting and interconnect operational verification after you install
an interconnect chassis in a cluster.
Chapter 9: Postinstallation
Troubleshooting and
Diagnostics
Defines the operational and technical specifications for the 24-port Voltaire interconnect
model ISR 9024.
Appendix A: ISR 9024
Interconnect Specifications
Defines the operational and technical specifications for the RoHS compliant 24-port
Voltaire interconnect model ISR 9024 S/D.
Appendix B: ISR 9024 S/D
Interconnect Specifications
(RoHS Compliant)
Defines the operational and technical specifications for the 96-port Voltaire interconnect
model ISR 9096.
Appendix C: ISR 9096
Interconnect Specifications
Defines the operational and technical specifications for the 288-port Voltaire interconnect
model ISR 9288.
Appendix D: ISR 9288
Interconnect Specifications
Organization 11

DescriptionChapter
Defines the operational and technical specifications for the 4x DDR InfiniBand Switch
Module for the c-Class Bladesystem enclosure.
Appendix E: 4x DDR IB
Switch Module
Specifications
Defines the operational and technical specifications for the HCA-400-series and other
HCA cards installed in cluster nodes.
Appendix F: HCA PCI
Card Specifications
Audience
This manual is intended for experienced hardware administrators of large-scale computer systems,
and for HP Global Service representatives. This manual references skilled tasks and describes
important safety considerations and is not intended as a training aide. The information in this
manual assumes that the reader has the following skills and abilities:
• Familiar with the theory and implementation of the high-speed system interconnect
technology used to create clusters.
• Familiar with the fundamentals of router-based networking, networking concepts, network
switch technology, and network cables.
• Trained to configure hardware equipment. It is assumed that readers are familiar with
InfiniBand technology and terminology.
• Familiar with the operating environment software.
• Familiar with HP rack-mounted servers and associated rack hardware.
• Has read the HP Cluster Platform Overview and the HP Cluster Platform Site Preparation Guide,
and is familiar with HP Cluster Platform architecture and concepts.
Related Documentation
The following documents may be useful references when you are installing and administering
the HP Cluster Platform cluster.
Voltaire Documentation
The Voltaire documentation consists of the following information:
•Voltaire ISR 9024 S/D Installation Manual — Explains how to install the ISR 9024 Switch
•Voltaire Switch 9024, 9096, and 9288 User Manual — Explains how to use Voltaire switches
used in HP Cluster Platform solutions
• Release notes for InfiniBand interconnects
You can also find Voltaire documentation at the following URL:
http://www.voltaire.com/partner_login.htm
You will require the appropriate login credentials which are available from Voltaire.
HP Cluster Platform Documentation
The HP Cluster Platform documentation consists of the following information:
• The HP Cluster Platform Overview — Provides a technical overview of the architecture,
hardware, software, and features of an HP Cluster Platform solution.
• The HP Cluster Platform Site Preparation Guide – Explains how to prepare a site to receive an
HP Cluster Platform solution.
• The InfiniBand Interconnect Installation and User's Guide – Describes the system interconnect
and installation procedures for the InfiniBand interconnect.
•HP Cluster Platform InfiniBand Cabling Tables – Describes the cabling port ID naming
conventions for InfiniBand interconnects used in HP Cluster Platform solutions.
•Ethernet Network Cabling Tables – Describes procedures for cabling the cluster's console and
administrative networks.
12

•Blades InfiniBand Interconnect Cabling Tables – Provides the point-to-point InfiniBand cabling
tables for c-Class BladeSystems.
•c-Class Blade Cable Management Bracket Installation Guide – Provides the installation procedure
for the c-Class Blade Cable Management Bracket.
•HP Cluster Platform InfiniBand Fabric Management and Diagnostic Guide – Provides test and
diagnostic procedures that are specific to the application of Voltaire InfiniBand interconnects.
Important:
Go to http://www.docs.hp.com under High Performance Computing to download the latest HP
Cluster Platform information.
Cluster Component Documentation
A cluster is assembled primarily from existing HP components that each have their own
documentation set. Two full sets of documentation for each type of component are included with
the cluster when it is delivered. The following components might be included, depending on
the configuration of the cluster and the operating system that is installed:
• One or more server models used as a control, compute, or utility node.
• ProCurve network switches, such as the ProCurve 2824 Gigabit Ethernet switch.
• HP 10000-series rack, including the following rack components:
— Rackmount workstation or KVM (keyboard, video, and mouse).
— Power distribution units (PDU).
• Optional storage, in the form of disk shelves (just a bunch of disks, JBOD), a storage area
network (SAN) unit or a separate modular storage cluster.
Depending on the component, documentation might be in electronic format on CD-ROMs or in
the printed documentation kit delivered with the cluster. If you need to know more about the
features of particular cluster components, you can obtain additional component documentation
from the following URL:
http://docs.hp.com/
HP Encourages Your Comments
HP encourages your comments concerning this document. We are committed to providing
documentation that meets your needs. Send any errors found, suggestions for improvement, or
compliments to http://www.docs.hp.com/en/feedback.html
Include the document title, manufacturing part number, and any comment, error found, or
suggestion for improvement you have concerning this document.
Text Annotations
The following annotations might appear in this guide:
HP Encourages Your Comments 13

Warning !
Text set off in this manner indicates that failure to follow directions could result in bodily harm
or loss of life.
Caution:
Text set off in this manner indicates that failure to follow directions could result in equipment
damage or data loss.
Important:
Text set off in this manner presents clarifying information or specific instructions.
Note:
Text set off in this manner presents commentary, sidelights, or interesting points of information.
Conventions
This manual uses the following typographic conventions:
Monospace type This type denotes literal items, such as command names, file names,
routines, directory names, path names, signals, messages, and
programming language structures.
Boldface type In command and interactive examples, this type denotes literal items
entered by the user (typed user input).
Boldface type in text indicates a word that is defined in the glossary.
Italic type Italic (slanted) type indicates variable values, placeholders, and function
argument names. Also denotes word or concept definitions, for example,
file and document titles.
Ctrl/xThis symbol indicates that you hold down the first named key while
pressing the key or mouse button that follows the slash. In examples, this
key combination is enclosed in a box (for example, Ctrl/C ).
14

Safety Considerations
To avoid bodily harm and damage to electronic components, read the following safety and
comfort guidelines before unpacking and configuring the cluster components. Heed the following
additional warnings and refer to the HP Cluster Platform Overview and the HP Cluster Platform
Site Preparation Guide to obtain specific information on safety issues. Before working on any
cluster components, be sure to read the component-specific safety information that is in the
documentation provided for each component.
Avoiding Electrical Shocks To avoid electrical shock, do not open the power supplies
or any other parts and components that bear a warning
label. There are no user-serviceable parts inside.
Only certified technicians should perform the following
tasks:
• Adding electrical circuits for the cluster.
• Checking ground paths.
• Measuring voltage levels of completed circuits.
Laser Radiation Do not open a CD-ROM reader or DVD reader laser
module. There are no user-serviceable parts inside. The
laser module should be serviced by service personnel only.
Do not attempt to make any adjustment to the laser unit.
See the label on the unit for power requirements and
wavelength. HP recommends that you read the ergonomics
information provided on the documentation CD-ROM
before using your system.
Moving and Stabilizing Racks To reduce the risk of personal injury or damage to
equipment in the cluster racks, do not attempt to move the
racks without adequate assistance, due to their height and
weight. Also, do not attempt to move a rack on an incline
that is greater than 10 degrees from the horizontal. No rack
stabilization kits are provided with the cluster. For a
multiple-rack installation, ensure that the racks are coupled
together using the specified baying kit.
Removing and Replacing Cluster
Components
Many cluster components are not equipped with
extendable rail kits and cable management devices. Refer
to the cluster documentation for instructions on how to
move components in the rack. There are no safety interlocks
which prevent the extension of multiple components at
the same time. To reduce the risk of personal injury or
damage to equipment in the cluster racks, do not attempt
to extend more than one component from a rack at a time.
The rack might become unstable and tip over if more than
one component is extended.
Be careful when pressing component rail release latches
and sliding a component into the rack. The slide rails can
pinch your fingertips.
If you need to remove more than one component from a
rack for any reason, always start from the top of the rack
and remove the lightest items first. When reloading the
rack, start from the bottom of the rack and replace the
heaviest items first.
15

Refer to the specifications section of the component
documentation to find the weight of a component.
Removing and Replacing
Component Covers
For your safety, never remove the cover from a cluster
component without first disconnecting the power cord
from the power outlet and removing any connection to a
telecommunications network. If a Power Protection Device
is fitted to your system, you must shut down the computer
using its on/off switch, then remove the power cord before
removing the component's cover. Remove the Power
Protection Device cables before any servicing operation.
Always replace the cover before switching the component
on again.
Using Cellular Phones and Other
Wireless Technology
Cellular telephones and other wireless technology can
interfere with the operation of components in the HP
Cluster Platform when the racks are opened and
components are removed for service. A cellular telephone
might emit a particularly strong signal when it is hunting
for a signal. Shut off all wireless technology when servicing
the cluster.
Battery Safety Information There is a danger of explosion if a battery is incorrectly
installed. For your safety, never attempt to recharge,
disassemble, or burn an old battery. Replace the battery
with the same or equivalent type, as recommended by the
manufacturer. The battery in this system is a lithium
battery that does not contain any heavy metals. However,
to protect the environment, do not dispose of batteries in
household waste. Return used batteries either to the shop
from which you bought them, to the dealer from whom
you purchased your system, or to HP so that they can either
be recycled or disposed of correctly. Returned batteries are
accepted free of charge.
Avoiding Hearing Damage Your HP Cluster Platform might exceed local safety
requirements for noise. For your safety and comfort, always
use hearing protection if you are going to be within 10 feet
of the cluster. The cluster documentation contains
information that enables you to calculate the sound energy
produced by the cluster.
Avoiding Burn Injuries Some parts of the cluster or cluster components become
hot during normal use. Wait approximately 3–5 minutes
after shutting off the components for them to cool down
before touching them.
Avoiding Static Electricity Static electricity can damage electronic components and
cause minor electrical shocks.
Information on Ergonomics
Ergonomics and safety information is provided with all components. HP recommends that you
read and ensure that you understand all such information before using a component. Safety and
comfort information is provided at the following URL:
http://www.hp.com/ergo/checklist.html.
16

Recycling
Shipping an integrated cluster generates far less packaging than the individual components that
it contains. However, large clusters use a substantial amount of packaging material that is not
reusable. The bulk of the packaging material is recyclable, and is labeled as such. You should
plan on providing a number of dumpsters into which this packaging can be sorted and recycled.
HP has a strong commitment to protecting the environment. Your HP system has been designed
to be as environmentally safe as possible. HP can also take back your old system for recycling
when it reaches the end of its useful life. HP has a product take-back program in several countries.
The collected equipment is sent to HP recycling facilities in Europe or the U.S.A. As many parts
as possible are reused. The remainder is recycled. Special care is taken for batteries and other
potentially toxic substances; these are reduced into non-harmful components through special
chemical processes. If you require more information about the HP product take-back program,
contact your local dealer or your nearest HP Sales Office. Further information is also available
on the “Product Return and Recycling link at
http://www.hp.com/hpinfo/globalcitizenship/environment/recycle/index.html.
Recycling 17

18

1 InfiniBand Technology Overview
InfiniBand™ is a specification of the InfiniBand® Trade Association of which Hewlett-Packard
is a member. The trade association has generated specifications for a 100 Gb/s communications
protocol for high-bandwidth, low latency server clusters. The same communications protocol
can operate across all system components for computing, communications, and storage as a
distributed fabric. InfiniBand supplements other interconnect technologies such as SCSI I/O,
providing an interconnect for storage and communications networks that is efficient, reliable,
and scalable. The InfiniBand specification defines the physical layers, application layers,
application programming interfaces, and fabric management (the complete I/O stack).
An InfiniBand switched network provides a fabric that has the following features:
• A high-performance, channel-based interconnect architecture that is modular and highly
scalable, enabling networks to grow as needed.
•It provides hardware management features such as device discovery, device failover, remote
boot, and I/O sharing.
• It enables devices such as servers, storage, and I/O to communicate at high speed, avoiding
the serialization of data transfers required by shared I/O buses.
• It provides inter-processor communication and memory sharing at speeds from 2.5 Gb/s to
30 Gb/s
• It has advanced fault isolation controls that provide fault tolerance.
InfiniBand Network Features
The InfiniBand architecture consists of single data rate (SDR) and double data rate (DDR) channels
created by linking host channel adapters (HCAs) and target channel adapters through InfiniBand
interconnects (switches). The host channel adapters are PCI bus devices installed in a server, or
are integrated into devices such as storage arrays. An HP Cluster platform consists of many
servers connected in a fat tree (clos) topology through one or more InfiniBand interconnects.
Where necessary, a target channel adapter connects the cluster to remote storage and networks
such as Ethernet, creating an InfiniBand fabric. Features of an InfiniBand fabric are:
• Performance:
— The following bandwidth options are supported by the architecture:
◦ 1X (2.5 Gb/s)
◦ 4X (10 Gb/s or 20 Gb/s — depending on configuration)
◦ 12X (30 Gb/s)
◦ DDR–up to 60 Gb/s
Note:
1X bandwidth occurs only if a 4X or 12X link auto-negotiates to 1X because of link
problems.
— Low latency
— Reduced CPU utilization
— Fault-tolerance through automatic path migration
— Physical multiplexing through virtual lanes
— Physical link aggregation
• Flexibility:
— Linear scalability
— Industry standard components and open standards
19

— Security and partitioning
— Integrated management tools
• A strong technical future, with support for:
— PCI cards:
◦ Voltaire SDR PCI-X
◦ Topspin/Mellanox SDR PCI-X Rev B; Mellanox SDR PCI-X Rev C
◦ Topspin/Mellanox SDR PCI-Express Rev B; Mellanox SDR PCI-Express Rev C
◦ Mellanox Mem Free SDR PCI-Express
◦ Mellanox Mem Free DDR PCI-Express
◦ Mellanox DDR PCI-Express
— Remote direct memory access (RDMA)
— Faster link speeds
InfiniBand Protocol
InfiniBand fabrics provide a single point of management for all HCAs (host channel adapters)
in the fabric. Management utilities enable you discover and configure devices, handle device or
path failures and allocate resources. InfiniBand enables applications to access the HCA directly
without involving the operating system by mapping the application memory to the physical
memory in tables that are maintained in the HCA.
An applications programming interface (API) facilitates control of HCAs and enables reliable
message passing between the channel adapters in different devices. The entire transport layer
is implemented in hardware (on integrated circuits, rather than is a software layer), Each channel
adapter provides queue pairs, for read and write that are accessible directly by programs. Each
queue pair provides a reliable connection between connected applications, nodes, or an unreliable
datagram message queue. The communications protocol consists of four operations:
• Transmit a message.
• Using remote data memory access (RDMA), writes data directly into a device's memory.
• Using remote data memory access (RDMA), reads data directly from a device's memory.
• Atomic operations, updating remote memory words.
The HCA device provides the following functions:
• Receives and posts the following operations to a queue as work queue elements:
— Messages.
— RDMA operations.
• Handles notifications and completion events through completion queues. To reduce the
volume of events in the system, completion queues might be shared for several queue pairs.
The remaining sections in this chapter include the following topics:
• Identifying InfiniBand fabric components (Section 1.1).
• ISR 9xxxx-series Interconnects (Section 1.2).
• Identifying the ISR 9024 interconnect (Section 1.3).
• Identifying the ISR 9024 S/D RoHS compliant interconnect (Section 1.4).
• Identifying the ISR 9096 interconnect (Section 1.5).
• Identifying the ISR 9288 interconnect (Section 1.6).
• Understanding the interconnect management capabilities (Section 1.7).
1.1 Identifying InfiniBand Fabric Components
Figure 1-1 shows a simplified model of an InfiniBand fabric and identifies the following typical
fabric components:
• HCA (Host Channel Adapter), which can have two hardware formats:
20 InfiniBand Technology Overview