HP 489183-B21 - InfiniBand DDR Switch User manual

HP BLc 4X DDR InfiniBand Gen 2 Switch
Installation Guide
for HP c-Class BladeSystems
HP Part Number: 5992-5296
Published: October 2008
Edition: 1.02

©Copyright 2008 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP
shall not be liable for technical or editorial errors or omissions contained herein.
Printed in the U.K.
InfiniBand™ is a trademark of the InfiniBand® Trade Association.
Gore™ is a trademark of W. L. Gore & Associates, Inc.

Table of Contents
1 Preparing for Installation................................................................................................7
1.1 Audience...........................................................................................................................................7
1.2 Documentation Resources.................................................................................................................7
1.3 Kit Contents.......................................................................................................................................7
1.4 Additional Information.....................................................................................................................7
1.5 HP BLc 4X DDR InfiniBand Gen 2 Switch Connector Bays..............................................................8
1.6 HP BLc 4X DDR InfiniBand Gen 2 Switch LEDs..............................................................................9
2 Hardware Installation..................................................................................................11
2.1 Installing the HP BLc 4X DDR InfiniBand Gen 2 Switch................................................................11
2.1.1 Prepare the Interconnect Module Bay in the HP BladeSystem c7000 Enclosure....................11
2.1.2 Install the HP BLc 4X DDR InfiniBand Gen 2 Switch into the HP BladeSystem Enclosure....12
2.2 Installing InfiniBand Cables in the HP BLc 4X DDR InfiniBand Gen 2 Switch..............................12
2.2.1 Removing an InfiniBand Cable from the HP BLc 4X DDR InfiniBand Gen 2 Switch............14
2.3 InfiniBand Cable Management Bracket..........................................................................................14
2.4 InfiniBand Subnet Management.....................................................................................................15
3 Removing the HP BLc 4X DDR InfiniBand Gen 2 Switch for Service......................17
4 Regulatory Compliance...............................................................................................19
4.1 Regulatory Compliance Identification Numbers............................................................................19
4.2 Class A Equipment..........................................................................................................................19
4.3 Modifications...................................................................................................................................19
4.4 Cables..............................................................................................................................................19
4.5 Canadian Notice..............................................................................................................................19
4.6 European Union Regulatory Notice................................................................................................19
4.7 Japanese Notice...............................................................................................................................20
4.8 Korean Notice..................................................................................................................................20
Table of Contents 3

List of Figures
1-1 HP BLc 4X DDR InfiniBand Gen 2 Switch......................................................................................7
1-2 HP Onboard Administrator Screen Example.................................................................................8
1-3 HP BLc 4X DDR InfiniBand Gen 2 Switch Connector Bays............................................................8
1-4 HP BLc 4X DDR InfiniBand Gen 2 Switch LEDs............................................................................9
2-1 Removing a Divider from the HP BladeSystem c7000 Enclosure Interconnect Module Bay.......11
2-2 Installing the HP BLc 4X DDR InfiniBand Switch into a c7000 Enclosure...................................12
2-3 QSFP InfiniBand DDR/QDR Cable...............................................................................................13
2-4 Align the InfiniBand Cable with the HP BLc 4X DDR InfiniBand Gen 2 Switch Port..................13
2-5 InfiniBand Cable Installed in Port 1 of the HP BLc 4X DDR InfiniBand Gen 2 Switch................14
4-1 EU Declaration of Conformity Markings......................................................................................20
4-2 Japanese Notice.............................................................................................................................20
4-3 Korean Notice................................................................................................................................20
4 List of Figures

List of Tables
1-1 HP BLc 4X DDR InfiniBand Gen 2 Switch Descriptions and Status...............................................9
5

6

1 Preparing for Installation
1.1 Audience
This document is intended for HP service representatives and other persons trained to install
hardware options in HP BladeSystem configurations. Such persons are expected to understand
the hazards of working in this environment and to take suitable precautions to minimize danger
to themselves and others.
1.2 Documentation Resources
You can download the documentation for components referenced in this installation guide from
the HP BladeSystem Technical Resources website:
http://h71028.www7.hp.com/enterprise/cache/499697-0-0-0-121.html
Check the HP BladeSystem Technical Resources website frequently for updates.
1.3 Kit Contents
The kit contains the following items:
• An HP BLc 4X DDR InfiniBand Gen 2 Switch (489183–B21) for HP c-Class BladeSystems
(see Figure 1-1)
• This hardware installation document
• Warranty documentation
Note:
This kit does not include a subnet manager, the InfiniBand cables, or the cable management
bracket. You must install a subnet manager on a host node or install an internally managed
InfiniBand interconnect to manage the InfiniBand fabric. You must also order the InfiniBand
cables and the cable management bracket separately.
Figure 1-1 HP BLc 4X DDR InfiniBand Gen 2 Switch
1.4 Additional Information
Mezzanine HCA installation determines interconnect bay assignments for switch installation.
For more information on the association between the mezzanine connector and the interconnect
bays, see the following documents under Mezzanine-to-Interconnect Mappings on the HP
BladeSystem Technical Resources website:
•HP BladeSystem c7000 Enclosure Setup and Installation Guide:
http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00698286/c00698286.pdf
•HP BladeSystem c3000 Enclosure Setup and Installation Guide:
http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01167165/c01167165.pdf
1.1 Audience 7

For specific port connections for each server blade, see the HP BladeSystem Onboard Administrator
User Guide. Connections differ by server blade type. The information contained in this document
explains the cabling between the HP 4X DDR InfiniBand Gen 2 Switch and the network switches.
The HP Onboard Administrator screen example (see Figure 1-2) is a reference for port mapping.
Figure 1-2 HP Onboard Administrator Screen Example
You can install up to 16 InfiniBand cables in the uplink ports on HP 4X DDR InfiniBand Gen 2
Switch. An external subnet manager (not included in the HP 4X DDR InfiniBand Gen 2 Switch)
manages the topology for the InfiniBand fabric.
1.5 HP BLc 4X DDR InfiniBand Gen 2 Switch Connector Bays
Figure 1-3 shows the Quad Small Form-factor Pluggable (QSFP) connector bays on the front of
the HP BLc 4X DDR InfiniBand Gen 2 Switch.
Figure 1-3 HP BLc 4X DDR InfiniBand Gen 2 Switch Connector Bays
1 2 4 53
The following list describes the callouts in Figure 1-3:
1. Uplink connectors, bays 1-4
2. Uplink connectors, bays 5-8
3. I²C port (inter-integrated circuit port)
4. Uplink connectors, bays 9-12
5. Uplink connectors, bays 13-16
8 Preparing for Installation

1.6 HP BLc 4X DDR InfiniBand Gen 2 Switch LEDs
Figure 1-4 shows the LEDs and Table 1-1 describes the LEDS in the HP BLc 4X DDR InfiniBand
Gen 2 Switch.
Figure 1-4 HP BLc 4X DDR InfiniBand Gen 2 Switch LEDs
2
1 3
4
Table 1-1 describes the callouts shown in Figure 1-4:
Table 1-1 HP BLc 4X DDR InfiniBand Gen 2 Switch Descriptions and Status
StatusDescriptionItem
Blue = ActivatedUnit ID LED1
Off = Deactivated
Green = NormalInternal health LED2
Amber = Component degraded
Off = Power
Green = Physical link establishedPhysical link LED3
Green flashing = Physical link error
Off = No physical link established
Amber = Logical link establishedLogical link / Activity LED4
Amber flashing = Logical link with activity 1
Off = No logical link established
1 The logical link activity LED flashes with greater frequency as network activity increases.
1.6 HP BLc 4X DDR InfiniBand Gen 2 Switch LEDs 9

10

2 Hardware Installation
This chapter contains the following sections:
• Installing the HP BLc 4X DDR InfiniBand Gen 2 Switch (Section 2.1)
• Installing InfiniBand cables in the HP BLc 4X DDR InfiniBand Gen 2 Switch (Section 2.2)
• Cable Management (Section 2.3)
• InfiniBand subnet managerment (Section 2.4)
2.1 Installing the HP BLc 4X DDR InfiniBand Gen 2 Switch
To install the HP BLc 4X DDR InfiniBand Gen 2 Switch, follow the steps in Section 2.1.1 and
Section 2.1.2.
Caution:
All panels and doors should be in place and securely fastened while the unit is in operation for
proper cooling and for EMI emissions control.
2.1.1 Prepare the Interconnect Module Bay in the HP BladeSystem c7000 Enclosure
To prepare the interconnect module bay in the HP BladeSystem c7000 enclosure, follow these
steps:
1. Remove any devices or blanks.
2. To remove the divider, press the divider release button (see callout 1 in Figure 2-1) and slide
the divider out of the interconnect module bay (see callout 2 in Figure 2-1).
Figure 2-1 Removing a Divider from the HP BladeSystem c7000 Enclosure Interconnect Module Bay
1
2
The following list describes the callouts shown in Figure 2-1:
1. Press the divider release button.
2. Slide divider out from the interconnect module bay.
2.1 Installing the HP BLc 4X DDR InfiniBand Gen 2 Switch 11

2.1.2 Install the HP BLc 4X DDR InfiniBand Gen 2 Switch into the HP BladeSystem
Enclosure
To install the HP BLc 4X DDR InfiniBand Gen 2 Switch into the HP BladeSystem c7000 enclosure,
follow these steps:
1. Slide the HP BLc 4X DDR InfiniBand Gen 2 Switch all the way into the HP BladeSystem
c7000 enclosure (see callout 1 in Figure 2-2).
2. Push the blade handle in until it is in the locked position (see callout 2 in Figure 2-2). The
blade handle is locked when you can not pull it back out without pushing the blade handle
release.
Figure 2-2 Installing the HP BLc 4X DDR InfiniBand Switch into a c7000 Enclosure
1
2
The following list describes the callouts shown in Figure 2-2:
1. Slide the HP BLc 4X DDR InfiniBand Gen 2 Switch into the HP BladeSystem c7000 Enclosure.
2. Push the blade handle in until it is in the locked position.
2.2 Installing InfiniBand Cables in the HP BLc 4X DDR InfiniBand Gen 2
Switch
Figure 2-3 shows an InfiniBand cable with a QSFP connector used with the HP BLc 4X DDR
InfiniBand Gen 2 Switch. QSFP connector to QSFP connector InfiniBand DDR/QDR cables are
used to connect multiple HP BLc 4X DDR InfiniBand Gen 2 Switches together. For example, you
might connect two or more HP BLc 4X DDR InfiniBand Gen 2 Switches when another HP BLc
4X DDR InfiniBand Generation Switch is installed in another HP BladeSystem c7000 enclosure
that is mounted in the same HP 10K series rack. The HP BLc 4X DDR InfiniBand Gen 2 Switches
are connected together by the QSFP to QSFP InfiniBand cables to form an InfiniBand
high-performance network (also referred to as InfiniBand fabric).
For more information about InfiniBand networks, see InfiniBand support for HP ProLiant and HP
Integrity servers on the HP website:
http://h20311.www2.hp.com/HPC/cache/595863-0-0-0-121.html
12 Hardware Installation

Note:
InfiniBand cables with QSFP to CX4 connectors are required when the HP BLc 4X DDR InfiniBand
Gen 2 Switch uplinks to an InfiniBand spine interconnect with CX4 connectors.
Figure 2-3 QSFP InfiniBand DDR/QDR Cable
1
23
The following list describes the callouts shown in Figure 2-3:
1. InfiniBand cable
2. QSFP connector release pull tab
3. QSFP connector
Note:
The InfiniBand cables are not included in this kit and must be purchased separately.
Installing an InfiniBand Cable in the HP BLc 4X DDR InfiniBand Gen 2 Switch
To install an InfiniBand cable in the HP BLc 4X DDR InfiniBand Gen 2 Switch, follow these steps:
1. Align the InfiniBand cable QSFP connector (see callout 1 in Figure 2-4) with the HP BLc 4X
DDR InfiniBand Gen 2 Switch port (see callout 2 in Figure 2-4).
2. Insert the QSFP connector into the HP BLc 4X DDR InfiniBand Gen 2 Switch port (see callout
1 in Figure 2-5) until it is fully engaged and locked.
Figure 2-4 Align the InfiniBand Cable with the HP BLc 4X DDR InfiniBand Gen 2 Switch Port
3
1
2
The following list describes the callouts shown in Figure 2-4:
2.2 Installing InfiniBand Cables in the HP BLc 4X DDR InfiniBand Gen 2 Switch 13

1. QSFP connector
2. HP BLc 4X DDR InfiniBand Gen 2 Switch - port 1
3. InfiniBand cable
Figure 2-5 InfiniBand Cable Installed in Port 1 of the HP BLc 4X DDR InfiniBand Gen 2 Switch
12
The following list describes the callouts shown in Figure 2-5:
1. InfiniBand cable installed in port 1 of the HP BLc 4X DDR InfiniBand Gen 2 Switch
2. HP BLc 4X DDR InfiniBand Gen 2 Switch
You can install up to 16 InfiniBand cables in the HP BLc 4X DDR InfiniBand Gen 2 Switch. It is
not necessary to install the InfiniBand cables in any particular order. However, for the sake of
consistency, begin installing the InfiniBand cables in port 1 and continue installing the cables up
to port 16.
2.2.1 Removing an InfiniBand Cable from the HP BLc 4X DDR InfiniBand Gen 2
Switch
To remove an InfiniBand cable from the HP BLc 4X DDR InfiniBand Gen 2 Switch, pull the QSFP
connector release tab (see callout 2 in Figure 2-3) which releases the connector while using your
other hand to remove the connector from the switch port.
To completely remove the cable, it might be necessary to remove a releasable cable tie on the HP
c-Class blade cable management bracket.
If it is necessary to remove a releasable cable tie from the cable management bracket to fully
release a QSFP InfiniBand connector from the switch, reverse the installation procedure
documented in the HP Cluster Platform c-Class Blade Cable Management Bracket Installation Guide
(see Section 2.3).
2.3 InfiniBand Cable Management Bracket
Once the InfiniBand cables are installed in the HP BLc 4X DDR InfiniBand Gen 2 Switch they
will require additional support and strain relief. To provide support and strain relief for the
InfiniBand cables, you must install a cable management bracket. For more information on the
cable management bracket for the HP BLc 4X DDR InfiniBand Gen 2 Switch, see the HP Cluster
Platform c–Class Blade Cable Management Bracket Installation Guide on the High Performance
Computing - Cluster Platform website:
http://docs.hp.com/en/highperfcomp.html
14 Hardware Installation

Note:
The c-Class cable management bracket is not included in this kit and must be ordered separately.
2.4 InfiniBand Subnet Management
The HP BLc 4X DDR InfiniBand Gen 2 Switch does not have an embedded subnet manager. All
InfiniBand networks require a subnet manager to manage, monitor and troubleshoot the
InfiniBand fabric. A subnet manager is required for the HP BLc 4X DDR InfiniBand Gen 2 Switch.
A subnet manager can be host-based which must be installed and run on a host node connected
to the InfiniBand fabric. You can also use a switch-based subnet manager when certain managed
switches are incorporated into the InfiniBand fabric.
For more information about InfiniBand subnet management, refer to the InfiniBand interconnect
documentation that came with your system or cluster. Also refer to the HP InfiniBand Options for
HP ProLiant and Integrity Servers Quickspec on the HP website:
http://h18000.www1.hp.com/products/quickspecs/13078_div/13078_div.HTML
2.4 InfiniBand Subnet Management 15

16

3 Removing the HP BLc 4X DDR InfiniBand Gen 2 Switch
for Service
Put the HP 10K series rack and the HP BladeSystem c7000 enclosure into a safe and stable state
for component removal and bring the component to an appropriate state for removal. You must
ensure that you can reach the component easily and handle it safely.
For more information about removing c-Class components from the c-Class enclosure, see the
HP BladeSystem c7000 Enclosure Service and Maintenance Guide:
http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00714237/c00714237.pdf
To remove the switch, follow these steps:
1. Disconnect and remove each InfiniBand cable until all of the networking ports are unplugged.
When disconnecting the cables, ensure that you do not bend them more than the
recommended bend radius. It might be necessary to loosen the releasable cable ties enough
to remove the cable from the switch. HP recommends that you do not remove the cables
from the releasable cable ties on the c-Class cable management bracket.
2. Remove the cable management bracket from the 10K rack's rear rails, but do not remove the
cables from the bracket.
3. Press the switch blade handle release button and slide the switch all the way out of the HP
BladeSystem c7000 enclosure.
Note:
HP provides various cable management solutions. Installation and removal of the HP BLc 4X
DDR InfiniBand Gen 2 Switch might depend on which type of cable management bracket is used
to support the InfiniBand cables used in your configuration. If you are unsure of the process to
remove a switch for service, contact an HP service representative. Also check the HP BladeSystem
Technical Resources web page frequently for updates to this guide:
http://h71028.www7.hp.com/enterprise/cache/316735-0-0-0-121.html
17

18

4 Regulatory Compliance
The chapter lists all of the regulatory notices of compliance for the HP BLc 4X DDR InfiniBand
Gen 2 Switch.
4.1 Regulatory Compliance Identification Numbers
For the purpose of regulatory compliance certifications and identification, this product has been
assigned a unique regulatory model number. The regulatory model number can be found on the
product nameplate label, along with all required approval markings and information. When
requesting compliance information for this product, always refer to this regulatory model number.
The regulatory model number is not the marketing name or model number of the product.
4.2 Class A Equipment
This equipment has been tested and found to comply with the limits for a Class A digital device,
pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection
against harmful interference when the equipment is operated in a commercial environment. This
equipment generates, uses, and can radiate radio frequency energy and, if not installed and used
in accordance with the instructions, may cause harmful interference to radio communications.
Operation of this equipment in a residential area is likely to cause harmful interference, in which
case the user will be required to correct the interference at personal expense.
4.3 Modifications
The FCC requires the user to be notified that any changes or modifications made to this device
that are not expressly approved by Hewlett-Packard Company may void the user’s authority to
operate the equipment.
4.4 Cables
Connections to this device must be made with shielded cables with metallic RFI/EMI connector
hoods in order to maintain compliance with FCC Rules and Regulations.
4.5 Canadian Notice
This Class A digital apparatus meets all requirements of the Canadian Interference-Causing
Equipment Regulations. Cet appareil numérique de la classe A respecte toutes les exigences du
Règlement sur le matériel brouilleur du Canada.
4.6 European Union Regulatory Notice
This product complies with the following EU Directives:
• Low Voltage Directive 2006/95/EC
• EMC Directive 2004/108/EC
Compliance with these directives implies conformity to applicable harmonized European
standards (European Norms) which are listed on the EU Declaration of Conformity issued by
Hewlett-Packard for this product or product family.
This compliance is indicated by the following conformity marking placed on the product:
4.1 Regulatory Compliance Identification Numbers 19

Figure 4-1 EU Declaration of Conformity Markings
Hewlett-Packard GmbH, HQ-TRE, Herrenberger Strasse 140, 71034 Boeblingen, Germany
4.7 Japanese Notice
Figure 4-2 Japanese Notice
4.8 Korean Notice
Figure 4-3 Korean Notice
20 Regulatory Compliance