HP Cluster Platform Express v2010 User manual

HP Cluster Platform
Myrinet System Interconnect Guide
Part Number: A-CPMYI-1A
May 2005
Revision/Update Information: Version 1.0
Product Version: HP Clusters
This manual describes the Myricom Myrinet system interconnects used in HP Cluster Platform
solutions.
Hewlett-Packard Company
Palo Alto, California

© Copyright 2005 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice. The only warranties for HP products and
services are set forth in the express warranty statements accompanying such products and services. Nothing herein
should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors
or omissions contained herein.
Printed in the United States.

Contents
About This Manual
1 Myrinet interconnect overview
1.1 Interconnect characteristics ................................................. 1-1
1.2 Backplane and chassis types ................................................ 1-2
1.3 Wiring rules ................................................................... 1-4
2 PCI cards
2.1 General characteristics ...................................................... 2-1
2.2 Single-port XP PCI card ..................................................... 2-2
2.3 Dual-port 2XP PCI ............................................................ 2-3
2.4 Single-port 2XP PCI .......................................................... 2-5
3 Switch, spine and monitoring line cards
3.1 Switch card .................................................................... 3-1
3.2 Spine card ...................................................................... 3-3
3.3 Monitoring line card .......................................................... 3-4
4LEDindication
s and diagnostics
4.1 Line card and PCI LEDs ..................................................... 4-1
4.2 Using LEDs for diagnostics ................................................. 4-1
4.3 Using the monitoring line card for diagnostics ........................... 4-2
5 Replacing components
5.1 Installing a monitoring line card ........................................... 5-1
5.1.1 Rebooting the monitoring line card ................................... 5-2
5.2 Removing and installing an 8-port line card ............................. 5-2
5.3 Replacing a fan tray .......................................................... 5-4
5.4 Replacing an interconnect chassis ......................................... 5-5
5.4.1 Removing the interconnect from the rack ............................ 5-5
5.4.2 Swapping the line cards ................................................. 5-9
5.4.3 Replacing the interconnect in the rack ............................... 5-10
6 Powering up procedure
Index
Figures
1-1 Fat Tree Topology ............................................................. 1-1
1-2 Myrinet System Interconnect Backplane ................................. 1-2
1-3 Federated Configuration Switches ......................................... 1-2
1-4 Nonfederated Configuration Switch ....................................... 1-3
Contents iii

1-5 128-Port Myrinet System Interconnect .................................... 1-4
1-6 Fully Populated 128-Port Myrinet System Interconnect ............... 1-5
2-1 PCI Identification Label ..................................................... 2-1
2-2 Single-port XP PCI Rev. D card ............................................ 2-2
2-3 Myricom 2XP PCI Rev. E card .............................................. 2-4
2-4 Myricom 2XP PCI Rev. F card .............................................. 2-6
3-1 Myrinet system Interconnect 8-port switch card ......................... 3-1
3-2 External features of the Myrinet system interconnect fiberoptic
8-port switch card ............................................................. 3-1
3-3 Myrinet system interconnect 8-Port GigE switch card .................. 3-2
3-4 Block Diagram of the gigabit ethernet switch card ...................... 3-3
3-5 Spine Card Block Diagram .................................................. 3-4
3-6 Monitoring Line Card ........................................................ 3-5
3-7 External Features of the Monitoring Line Card ......................... 3-5
5-1 Unlocking Switch Card Handles ........................................... 5-2
5-2 Ejecting a Switch Card ....................................................... 5-3
5-3 Sliding a Switch Card ........................................................ 5-3
5-4 Backplane Power Connector ................................................ 5-3
5-5 Removing a Fan Tray ........................................................ 5-4
5-6 Interconnect Fan Tray ....................................................... 5-5
5-7 Removing the Retainer Brackets ........................................... 5-6
5-8 Sliding Interconnect Chassis from the Rack .............................. 5-7
5-9 Blanking Plate ................................................................ 5-7
5-10 Hold-Down Bracket ........................................................... 5-8
5-11 Left Bracket Retainer ........................................................ 5-8
5-12 Right Bracket Retainer ...................................................... 5-8
5-13 Removing the Existing Brackets ........................................... 5-9
Tables
1-1 Chassis types .................................................................. 1-4
2-1 XP PCI Host Interface Card Specifications ............................... 2-3
2-2 XP PCI Card Physical Characteristics .................................... 2-3
2-3 2XP PCI Card Specifications ................................................ 2-5
2-4 2XP PCI Card Physical Characteristics ................................... 2-5
iv Contents

About This Manual
This manual describes the Myricom Myrinet system interconnects used in some
HP Cluster Platform solutions.
This manual does not describe the procedures and tools that are required to install
and configure the system hardware or software. It does contain references for
cluster components that have their own documentation.
Audience
This manual is intended for experienced hardware system administrators. The
information in this manual assumes that the end user:
• Is familiar with HP rack-mounted servers and associated rack hardware
• Is familiar with basic networking concepts, network switch technology, and
network cables
• Is familiar with HP cluster architecture and concepts
Organization
This manual is organized as follows:
Chapter 1
Presents an overview of the Myrinet-2000
system interconnects.
Chapter 2
Presents an explanation of the Myricom PCI
cards.
Chapter 3
Describes the 8-port infiniband and gigabit
ethernet switch cards, spine cards and
monitoring line cards.
Chapter 4
Presents an explanation of the LEDs and
some diagnostic procedures.
Chapter 5
Provides instructions for replacing some
components in the system interconnect.
Chapter 6
Describes how to power up the system
interconnect, its operating temperatures
and its specificiations.
Additional documentation
More information about cluster components can be found in the documentation set
that shipped with your cluster. More information about the Myricom interconnects
can be found on the Myricom Web site, http://www.myri.com. You can search
for additional HP documents online at http://docs.hp.com.
Providing feedback
HP welcomes your comments on this manual. Please send your comments and
suggestions by email to [e-mail].
About This Manual v

Symbols in text
!
zk-2071
WARNING: Text set off in this manner indicates that failure to follow
directions in the warning could result in bodily harm or loss of life.
!
zk-2071 CAUTION: Textsetoffinthismannerindicatesthatfailuretofollow
directions in the caution could result in damage to equipment or loss of
information.
_______________________ Important _______________________
Text set off in this manner presents clarifying information or specfic
instructions.
_________________________ Note _________________________
Text set off in this manner presents commentary, sidelights, or
interesting points of information.
vi About This Manual

1
Myrinet interconnect overview
The high-speed Myrinet system interconnect is used in some HP Cluster Platform
solutions. The chassis for the interconnect is 5U or 9U, depending on the cluster
configuration. A PCI adapter card connects each application node in the cluster
to the interconnect. The PCI card can havesingle(XP)ordual(2XP)ports. This
chapter presents the following information about the Myrinet system interconnect:
• A description of the interconnect characteristics (Section 1.1)
• An explanation of the backplane and the chassis types (Section 1.2)
• An overview of the interconnect wiring rules (Section 1.3)
1.1 Interconnect characteristics
The Myrinet system interconnect (Figure 1-5) provides transport for user
application communication between the control node and application nodes.
Applications communicate across the interconnect with message protocols, such as
the Message Passing Interface (MPI). User I/O requests for files also communicate
across the interconnect. The interconnect is also used for process management,
such as launching, signaling, and exiting applications.
The Myrinet system interconnect implements a fat tree topology (Figure 1-1),
which maximizes cross-sectional bandwidth and minimizes the possibility of one
transfer being blocked by another.
Figure 1-1: Fat Tree Topology
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
8
hosts
Ports to up to 128 hosts (line cards)
Spine of the fat tree network (backplane)
The characteristics of the interconnect switch include the following:
• 5U (64-port) , or 9U (128-port) chassis
• Eight-port switch cards (one card for every eight processors)
• One monitoring line card
• Flow control, error control, and continuity monitoring on every link
• HTTP monitoring card
Myrinet interconnect overview 1-1

1.2 Backplane and chassis types
The Myrinet interconnect used in an HP Cluster Platform can be a 5U (64-port)
or 9U (128-port) chassis. The backplane (Figure 1-2) provides connectivity for
the 8-port switch cards.
Figure 1-2: Myrinet System Interconnect Backplane
For configurations greater than 128 total nodes, you must build the network out
to increase the number of available ports and to maintain the cross-sectional
bandwidth. You accomplish this by connecting additional switches together in a
hierarchical manner (Figure 1-3). This type of configuration is called a federated
configuration. A federated configuration contains two types of Myrinet system
interconnects:
• Node-level switch
• Top-level switch
Figure 1-3: Federated Configuration Switches
Top-Level Chassis Node-Level Chassis
Monitor Line Card
Switch Cards
10/100 LAN
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Monitor Line Card
Switch Cards
10/100 LAN
Switch Cards
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
76543210
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1-2 Myrinet interconnect overview

The 128-port, node-level switch in a federated configuration provides 64
downlinks to nodes and dedicates the other 64 ports to uplinks that provide
connectivity to deeper parts of the network. A top-level switch, another Myrinet
system interconnect, consolidates the uplinks from two node-level switches. Each
switch supports up to 128 links, 64 links from each node-level switch; therefore,
two top-level switches are necessary to support configurations up to 256 nodes. The
node-level switch is populated with switch cards and spine cards, and the top-level
switch is populated with spine cards.
In a non-federated configuration, a 128-port interconnect provides 128 downlinks
and is populated with switch cards (Figure 1-4).
Figure 1-4: Nonfederated Configuration Switch
Monitor Line Card
Switch Cards
10/100 LAN Not Connected
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
15 14 13 12 11 10 9 8
Figure 1-5 shows a fully populated 128-node chassis. If the switch is not fully
populated, filler panels must be used in all the blank slots in the chassis.
Myrinet interconnect overview 1-3

Figure 1-5: 128-Port Myrinet System Interconnect
xc-013B-IG
Table 1-1 explains the chassis types, and the types and location of line cards within
the interconnects.
Table 1-1: Chassis types
Chassis Type Component Quantity Location
Bounded chassis M3-E128 17 slot chassis 1
Monitoring Line Card 1SlotM
8 Port Switch Line Card 1 -- 16 Slot 0 - 15
Node level chassis M3-E128 17 slot chassis 1
Monitoring Line Card 1SlotM
8 Port Spine Card 1--8 Slot0-7
8 Port Switch Line Card 1 -- 8 Slot 8 - 15
Top level chassis M3-E128 17 slot chassis 1
Monitoring Line Card 1SlotM
8 Port Spine Card 1--16 Slot0-15
1.3 Wiring rules
When you connect cables to a Myrinet system interconnect in a bounded
configuration with 128 or fewer nodes, you connect downlinks to the application
nodes and utility nodes starting from opposite sides of the switch (Figure 1-6).
When populating node-level Myrinet system interconnects in a modular
configuration, you connect downlinks to the application nodes and utility nodes
from one side of the switch and uplinks to the top-level switches from the opposite
side of the switch. You maintain a ratio of one-to-one between the 8-port spine
cards and the 8-port switch cards in each node-level chassis. Node level chassis
slots 0-7 are reserved for spine cards (uplinks). Node level chassis slots 8-15 are
reserved for switch cards (node connections).
1-4 Myrinet interconnect overview

Figure 1-6: Fully Populated 128-Port Myrinet System Interconnect
4
Application nodes
start here, move left,
and then down to
the next card.
If the control node is
connected to the
system interconnect,
it populates the
highest available
port. The utility nodes
follow the control node,
move right,and then
up to the next card.
Switch cards are
numbered 0-15,
beginning with the
card below the
monitoring line card (1).
Item Description
1M3-M monitoring line card
2M3-M MAC address
3Status light (common to all cards)
48-port switch card
5Card ejection/insertion handles
6Cooling fan module
Detailed wiring tables are presented in the HP Cluster Platform Myrinet System
Interconnect Configuration and Cabling Tables Guide .
Myrinet interconnect overview 1-5

2
PCI cards
Every node in a cluster that uses a Myricom system interconnect has a PCI host
bus adapter (HBA), and each host interface has a connection to the interconnect.
This chapter presents the following information relating to the three versions of
the PCI card that might be used in an HP Cluster Platform solution:
• General characteristics of the PCI card (Section 2.1)
• A description of the single-port XP PCI Rev. D card (Section 2.2)
• A description of the dual-port 2XP PCI Rev. E card (Section 2.3)
• A description of the single-port 2XP PCI Rev. F card (Section 2.4)
2.1 General characteristics
The Myricom PCI card can have a single port (LANai XP), as shown in Figure 2-2,
or dual (LANai 2XP) port, as shown in Figure 2-3. Each has at least one
multiprotocol communication port, referred to as the X-port. (LANai is the Myrinet
terminology for its PCI-X interface processor.)
The faceplate of each PCI card has two LEDs. A green “link” LED illuminates
when the host is powered, the link is active, and the firmware is loaded on the host.
It blinks to show when packet traffic is flowing through the port. A yellow LED is
controlled by the RISC processor. It pulses when there is packet-sending activity.
The LED on each switch port indicates the state of that port. Green indicates the
port is powered, and a link is connected. Yellow indicates a fault. If the switch port
is unlit, the switch is not powered, or there is a problem with the line card. The
LEDs for the ports only light when there is a cable attached and the GM message
passing system from Myrinet is running. For more information about GM, see your
system software documentation and the following Myricom Web site:
http://www.myri.com/scs/index.html#documentation
A label similar to that shown in Figure 2-1 appears on every PCI interface. This
label contains the product code, the serial number, and the Ethernet MAC address.
Figure 2-1: PCI Identification Label
PCI cards 2-1

2.2 Single-port XP PCI card
The single-port XP PCI card (Figure 2-2) is the Myricom Rev. D card. It supports
one X-port interface and one PCI-X interface.
Figure 2-2: Single-port XP PCI Rev. D card
2-2 PCI cards

Table 2-1 shows the single-port XP PCI card’s technical specifications.
Table 2-1: XP PCI Host Interface Card Specifications
Feature Specifications Notes
64-bit PCI-XXP PCI card
PCIlow-profilePCIshort
card
75-133 MHz PCI-X slotsBus speed
66 MHz (3.3V) PCI slots
Memory 2 MB
Memory bandwidth 1,800 MB/s
Processor 225 MHz RISC LANai XP
Data rate 490 MB/s User-level bidirectional data
rate.
Fiber port physical LC-type optical connector,
50/125 multimode fiber pair
Class I Laser Product (no
hazard)
Fiber port data rate 2.0+2.0 Gb/s data rate,
2.5+2.5 GBaud signaling
EEPROM 512 KB Serial number, MAC
address, product code, and
other information
Table 2-2describes the XP PCI card’s physical and environmental characteristics.
Table 2-2: XP PCI Card Physical Characteristics
Characteristic Specifications
Dimensions (excluding
faceplate)
6.4 cm high, 6.4 cm long, 2.2 cm total thickness
Mass 0.082 kg
Power requirement 3.3V from the PCI bus, 1.8A (6W) maximum
0C to 50C up to 10,000 foot altitudeEnvironmental, operating
Relative humidity 15% to 80% @ 50C
-40C to 70CEnvironmental, nonoperating
Relative humidity 90% @ 65C
2.3 Dual-port 2XP PCI
The dual-port 2XP PCI NIC (Figure 2-3) is the Myricom Rev. E card. It supports
two X-port interfaces and one PCI-X interface.
PCI cards 2-3

Figure 2-3: Myricom 2XP PCI Rev. E card
2-4 PCI cards

Table 2-3 describes the technical specifications for the 2XP PCI card.
Table 2-3: 2XP PCI Card Specifications
Feature Specifications Notes
2XP PCI card 64-bit PCI-X and PCI
protocols
Can be used in any 3.3V PCI
slot.
Bus speed 66-133 MHz Capable of peak PCI
date rates at the limits
of the PCI-X or PCI buses
(1067MB for 64-bit, 133MHz
PCI-X and 533MB for 64-bit
at 66 MHz).
Local memory 2MB
Memory bandwidth 2664 MB/s
Processor 333 MHz RISC LANai 2XP.
Fiber port physical LC-type optical connector,
50/125 multimode fiber pair
Class I Laser Product (no
hazard).
Fiber port data rate 2.0+2.0 Gb/s data rate,
2.5+2.5 GBaud signaling,
8b-10/b encoded
The standard firmware
distributes packets
adaptivelyacrossthe
two ports, such that the two
ports act as a single 4.0+4.0
Gb/s data-rate port.
EEPROM 512 KB Includes the PCI-
configuration-space data,
such as serial number,
MAC address, and product
code. The EEPROM can be
reprogrammed by the 2XP.
Table 2-4 describes the physical and environmental characteristics of the 2XP
PCI card.
Table 2-4: 2XP PCI Card Physical Characteristics
Characteristic Specifications
Dimensions (excluding faceplate) 6.4 cm high, 16.4 cm long, 2.2 cm thick
(low-profile PCI short card)
Mass 82g (including the faceplate)
Power requirement 3.3V from the PCI bus, 2.0A (6.6W) maximum
0C to 50C up to 10,000 foot altitudeEnvironmental, operating
Relative humidity 15% to 80% @ 50C
-40C to 70C Environmental, nonoperating
Relative humidity 90% @ 65C
2.4 Single-port 2XP PCI
The single-port 2XP PCI NIC (Figure 2-4) is the Myricom Rev. F card. It supports
one X-port interface and one PCI-X interface. These NICS must use the Myricom
GM-2 or Myrinet Express (MX) software.
Other than the difference in the number of ports, the Rev. F card has the same
specifications as the Rev. E card.
PCI cards 2-5

Figure 2-4: Myricom 2XP PCI Rev. F card
HPTC-0105
2-6 PCI cards

3
Switch, spine and monitoring line cards
The Myrinet system interconnect uses three types of cards. This chapter presents
the following information:
• A description of the switch card (Section 3.1).
• A description of the spine card (Section 3.2).
• A description of the monitoring line card (Section 3.3).
3.1 Switch card
The Myrinet system interconnect 8-port switch card (Figure 3-1) connects up to
eight nodes together. It has connectors to the backplane on the back side, and
connectors to external links on the front panel. It has a Status LED located on the
left front side of the card. This LED turns green after passing its self diagnostics.
If it fails, the LED turns yellow. These tests assess voltage levels, temperatures,
and internal status bits.
Figure 3-1: Myrinet system Interconnect 8-port switch card
xc-014-ig
Figure 3-2 identifies the external features of the Myrinet system interconnect
8-port fiberoptic switch card.
Figure 3-2: External features of the Myrinet system interconnect fiberoptic 8-port
switch card
Item Description
1Switch card status light
2Port status light
Switch, spine and monitoring line cards 3-1

Item Description
3Port identification number
4Type LC fiber port
Figure 3-3 identifies the external features of the Myrinet system interconnect
8-port GigE switch card. Each GigE port has two green LEDs in the RJ45
connector. The right LED indicates link activity, and the left LED indicates traffic.
Figure 3-3: Myrinet system interconnect 8-Port GigE switch card
8
9
10
11
12
13
14
15
YXZ[\
Item Description
1Card latch in unlocked position.
2Switch card power status light.
3RJ45 Ethernet port.
4Port LEDs. The right LED indicates link activiy, and the left LED
indicates traffic.
5Port identification number, which is used as part of the port ID string
for point-to-point cabling.
Every 8-port switch card in the system interconnect is numbered 8–15. This is
normal. The port number on the panel of the line card is only one element used to
identify a port. In port numbering discussions, a port is uniquely identified by the
rack number, chassis number, slot number, and the line card port number.
The backplane interface includes 8 SAN-2000 links numbered 0-7, corresponding
to the port numbering of the XBar16. Figure 3-4 shows the backplane of the GigE
switch card, and Figure 3-5 shows the backplane of the fiberoptic switch card. The
backplane interface also includes dual +12V power, and a serial-link interface to
the monitoring line card.
3-2 Switch, spine and monitoring line cards