HP Cluster Platform Express v2010 User manual

HP Cluster Platform
Myrinet System Interconnect Guide
Part Number: A-CPMYI-1A
May 2005
Revision/Update Information: Version 1.0
Product Version: HP Clusters
This manual describes the Myricom Myrinet system interconnects used in HP Cluster Platform
solutions.
Hewlett-Packard Company
Palo Alto, California

© Copyright 2005 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice. The only warranties for HP products and
services are set forth in the express warranty statements accompanying such products and services. Nothing herein
should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors
or omissions contained herein.
Printed in the United States.

Contents
About This Manual
1 Myrinet interconnect overview
1.1 Interconnect characteristics ................................................. 1-1
1.2 Backplane and chassis types ................................................ 1-2
1.3 Wiring rules ................................................................... 1-4
2 PCI cards
2.1 General characteristics ...................................................... 2-1
2.2 Single-port XP PCI card ..................................................... 2-2
2.3 Dual-port 2XP PCI ............................................................ 2-3
2.4 Single-port 2XP PCI .......................................................... 2-5
3 Switch, spine and monitoring line cards
3.1 Switch card .................................................................... 3-1
3.2 Spine card ...................................................................... 3-3
3.3 Monitoring line card .......................................................... 3-4
4LEDindication
s and diagnostics
4.1 Line card and PCI LEDs ..................................................... 4-1
4.2 Using LEDs for diagnostics ................................................. 4-1
4.3 Using the monitoring line card for diagnostics ........................... 4-2
5 Replacing components
5.1 Installing a monitoring line card ........................................... 5-1
5.1.1 Rebooting the monitoring line card ................................... 5-2
5.2 Removing and installing an 8-port line card ............................. 5-2
5.3 Replacing a fan tray .......................................................... 5-4
5.4 Replacing an interconnect chassis ......................................... 5-5
5.4.1 Removing the interconnect from the rack ............................ 5-5
5.4.2 Swapping the line cards ................................................. 5-9
5.4.3 Replacing the interconnect in the rack ............................... 5-10
6 Powering up procedure
Index
Figures
1-1 Fat Tree Topology ............................................................. 1-1
1-2 Myrinet System Interconnect Backplane ................................. 1-2
1-3 Federated Configuration Switches ......................................... 1-2
1-4 Nonfederated Configuration Switch ....................................... 1-3
Contents iii

1-5 128-Port Myrinet System Interconnect .................................... 1-4
1-6 Fully Populated 128-Port Myrinet System Interconnect ............... 1-5
2-1 PCI Identification Label ..................................................... 2-1
2-2 Single-port XP PCI Rev. D card ............................................ 2-2
2-3 Myricom 2XP PCI Rev. E card .............................................. 2-4
2-4 Myricom 2XP PCI Rev. F card .............................................. 2-6
3-1 Myrinet system Interconnect 8-port switch card ......................... 3-1
3-2 External features of the Myrinet system interconnect fiberoptic
8-port switch card ............................................................. 3-1
3-3 Myrinet system interconnect 8-Port GigE switch card .................. 3-2
3-4 Block Diagram of the gigabit ethernet switch card ...................... 3-3
3-5 Spine Card Block Diagram .................................................. 3-4
3-6 Monitoring Line Card ........................................................ 3-5
3-7 External Features of the Monitoring Line Card ......................... 3-5
5-1 Unlocking Switch Card Handles ........................................... 5-2
5-2 Ejecting a Switch Card ....................................................... 5-3
5-3 Sliding a Switch Card ........................................................ 5-3
5-4 Backplane Power Connector ................................................ 5-3
5-5 Removing a Fan Tray ........................................................ 5-4
5-6 Interconnect Fan Tray ....................................................... 5-5
5-7 Removing the Retainer Brackets ........................................... 5-6
5-8 Sliding Interconnect Chassis from the Rack .............................. 5-7
5-9 Blanking Plate ................................................................ 5-7
5-10 Hold-Down Bracket ........................................................... 5-8
5-11 Left Bracket Retainer ........................................................ 5-8
5-12 Right Bracket Retainer ...................................................... 5-8
5-13 Removing the Existing Brackets ........................................... 5-9
Tables
1-1 Chassis types .................................................................. 1-4
2-1 XP PCI Host Interface Card Specifications ............................... 2-3
2-2 XP PCI Card Physical Characteristics .................................... 2-3
2-3 2XP PCI Card Specifications ................................................ 2-5
2-4 2XP PCI Card Physical Characteristics ................................... 2-5
iv Contents

About This Manual
This manual describes the Myricom Myrinet system interconnects used in some
HP Cluster Platform solutions.
This manual does not describe the procedures and tools that are required to install
and configure the system hardware or software. It does contain references for
cluster components that have their own documentation.
Audience
This manual is intended for experienced hardware system administrators. The
information in this manual assumes that the end user:
• Is familiar with HP rack-mounted servers and associated rack hardware
• Is familiar with basic networking concepts, network switch technology, and
network cables
• Is familiar with HP cluster architecture and concepts
Organization
This manual is organized as follows:
Chapter 1
Presents an overview of the Myrinet-2000
system interconnects.
Chapter 2
Presents an explanation of the Myricom PCI
cards.
Chapter 3
Describes the 8-port infiniband and gigabit
ethernet switch cards, spine cards and
monitoring line cards.
Chapter 4
Presents an explanation of the LEDs and
some diagnostic procedures.
Chapter 5
Provides instructions for replacing some
components in the system interconnect.
Chapter 6
Describes how to power up the system
interconnect, its operating temperatures
and its specificiations.
Additional documentation
More information about cluster components can be found in the documentation set
that shipped with your cluster. More information about the Myricom interconnects
can be found on the Myricom Web site, http://www.myri.com. You can search
for additional HP documents online at http://docs.hp.com.
Providing feedback
HP welcomes your comments on this manual. Please send your comments and
suggestions by email to [e-mail].
About This Manual v

Symbols in text
!
zk-2071
WARNING: Text set off in this manner indicates that failure to follow
directions in the warning could result in bodily harm or loss of life.
!
zk-2071 CAUTION: Textsetoffinthismannerindicatesthatfailuretofollow
directions in the caution could result in damage to equipment or loss of
information.
_______________________ Important _______________________
Text set off in this manner presents clarifying information or specfic
instructions.
_________________________ Note _________________________
Text set off in this manner presents commentary, sidelights, or
interesting points of information.
vi About This Manual

1
Myrinet interconnect overview
The high-speed Myrinet system interconnect is used in some HP Cluster Platform
solutions. The chassis for the interconnect is 5U or 9U, depending on the cluster
configuration. A PCI adapter card connects each application node in the cluster
to the interconnect. The PCI card can havesingle(XP)ordual(2XP)ports. This
chapter presents the following information about the Myrinet system interconnect:
• A description of the interconnect characteristics (Section 1.1)
• An explanation of the backplane and the chassis types (Section 1.2)
• An overview of the interconnect wiring rules (Section 1.3)
1.1 Interconnect characteristics
The Myrinet system interconnect (Figure 1-5) provides transport for user
application communication between the control node and application nodes.
Applications communicate across the interconnect with message protocols, such as
the Message Passing Interface (MPI). User I/O requests for files also communicate
across the interconnect. The interconnect is also used for process management,
such as launching, signaling, and exiting applications.
The Myrinet system interconnect implements a fat tree topology (Figure 1-1),
which maximizes cross-sectional bandwidth and minimizes the possibility of one
transfer being blocked by another.
Figure 1-1: Fat Tree Topology
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
Ports to up to 128 hosts (line cards)
Spine of the fat tree network (backplane)
The characteristics of the interconnect switch include the following:
• 5U (64-port) , or 9U (128-port) chassis
• Eight-port switch cards (one card for every eight processors)
• One monitoring line card
• Flow control, error control, and continuity monitoring on every link
• HTTP monitoring card
Myrinet interconnect overview 1-1

1.2 Backplane and chassis types
The Myrinet interconnect used in an HP Cluster Platform can be a 5U (64-port)
or 9U (128-port) chassis. The backplane (Figure 1-2) provides connectivity for
the 8-port switch cards.
Figure 1-2: Myrinet System Interconnect Backplane
For configurations greater than 128 total nodes, you must build the network out
to increase the number of available ports and to maintain the cross-sectional
bandwidth. You accomplish this by connecting additional switches together in a
hierarchical manner (Figure 1-3). This type of configuration is called a federated
configuration. A federated configuration contains two types of Myrinet system
interconnects:
• Node-level switch
• Top-level switch
Figure 1-3: Federated Configuration Switches
Top-Level Chassis Node-Level Chassis
Monitor Line Card
Switch Cards
10/100 LAN
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Monitor Line Card
Switch Cards
10/100 LAN
Switch Cards
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1-2 Myrinet interconnect overview

The 128-port, node-level switch in a federated configuration provides 64
downlinks to nodes and dedicates the other 64 ports to uplinks that provide
connectivity to deeper parts of the network. A top-level switch, another Myrinet
system interconnect, consolidates the uplinks from two node-level switches. Each
switch supports up to 128 links, 64 links from each node-level switch; therefore,
two top-level switches are necessary to support configurations up to 256 nodes. The
node-level switch is populated with switch cards and spine cards, and the top-level
switch is populated with spine cards.
In a non-federated configuration, a 128-port interconnect provides 128 downlinks
and is populated with switch cards (Figure 1-4).
Figure 1-4: Nonfederated Configuration Switch
Monitor Line Card
Switch Cards
10/100 LAN Not Connected
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
Figure 1-5 shows a fully populated 128-node chassis. If the switch is not fully
populated, filler panels must be used in all the blank slots in the chassis.
Myrinet interconnect overview 1-3

Figure 1-5: 128-Port Myrinet System Interconnect
xc-013B-IG
Table 1-1 explains the chassis types, and the types and location of line cards within
the interconnects.
Table 1-1: Chassis types
Chassis Type Component Quantity Location
Bounded chassis M3-E128 17 slot chassis 1
Monitoring Line Card 1SlotM
8 Port Switch Line Card 1 -- 16 Slot 0 - 15
Node level chassis M3-E128 17 slot chassis 1
Monitoring Line Card 1SlotM
8 Port Spine Card 1--8 Slot0-7
8 Port Switch Line Card 1 -- 8 Slot 8 - 15
Top level chassis M3-E128 17 slot chassis 1
Monitoring Line Card 1SlotM
8 Port Spine Card 1--16 Slot0-15
1.3 Wiring rules
When you connect cables to a Myrinet system interconnect in a bounded
configuration with 128 or fewer nodes, you connect downlinks to the application
nodes and utility nodes starting from opposite sides of the switch (Figure 1-6).
When populating node-level Myrinet system interconnects in a modular
configuration, you connect downlinks to the application nodes and utility nodes
from one side of the switch and uplinks to the top-level switches from the opposite
side of the switch. You maintain a ratio of one-to-one between the 8-port spine
cards and the 8-port switch cards in each node-level chassis. Node level chassis
slots 0-7 are reserved for spine cards (uplinks). Node level chassis slots 8-15 are
reserved for switch cards (node connections).
1-4 Myrinet interconnect overview