Sun Oracle Sun Network QDR InfiniBand Gateway Switch User manual

Sun Network QDR InfiniBand Gateway Switch
Installation Guide for Firmware Version 2.1
Part No.: E36259-01
March 2013, Revision A

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by
intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate,
broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering,
disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us
in writing.
If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the
following notice is applicable:
U.S. GOVERNMENT END USERS. Oracle programs, including any operating system, integrated software, any programs installed on the hardware,
and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition
Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including
any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license
restrictions applicable to the programs. No other rights are granted to the U.S. Government.
This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any
inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software or hardware in dangerous
applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle
Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or
registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of
Advanced Micro Devices. UNIX is a registered trademark of The Open Group.
This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle
Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and
services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party
content, products, or services.
Copyright © 2013, Oracle et/ou ses affiliés. Tous droits réservés.
Ce logiciel et la documentation qui l’accompagne sont protégés par les lois sur la propriété intellectuelle. Ils sont concédés sous licence et soumis à des
restrictions d’utilisation et de divulgation. Sauf disposition de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire, traduire,
diffuser, modifier, breveter, transmettre, distribuer, exposer, exécuter, publier ou afficher le logiciel, même partiellement, sous quelque forme et par
quelque procédé que ce soit. Par ailleurs, il est interdit de procéder à toute ingénierie inverse du logiciel, de le désassembler ou de le décompiler, excepté à
des fins d’interopérabilité avec des logiciels tiers ou tel que prescrit par la loi.
Les informations fournies dans ce document sont susceptibles de modification sans préavis. Par ailleurs, Oracle Corporation ne garantit pas qu’elles
soient exemptes d’erreurs et vous invite, le cas échéant, à lui en faire part par écrit.
Si ce logiciel, ou la documentation qui l’accompagne, est concédé sous licence au Gouvernement des Etats-Unis, ou à toute entité qui délivre la licencede
ce logiciel ou l’utilise pour le compte du Gouvernement des Etats-Unis, la notice suivante s’applique :
U.S. GOVERNMENT END USERS. Oracle programs, including any operating system, integrated software, any programs installed on the hardware,
and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition
Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including
any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license
restrictions applicable to the programs. No other rights are granted to the U.S. Government.
Ce logiciel ou matériel a été développé pour un usage général dans le cadre d’applications de gestion des informations. Ce logiciel ou matériel n’est pas
conçu ni n’est destiné à être utilisé dans des applications à risque, notamment dans des applications pouvant causer des dommages corporels. Si vous
utilisez ce logiciel ou matériel dans le cadre d’applications dangereuses, il est de votre responsabilité de prendre toutes les mesures de secours, de
sauvegarde, de redondance et autres mesures nécessaires à son utilisation dans des conditions optimales de sécurité. Oracle Corporation et ses affiliés
déclinent toute responsabilité quant aux dommages causés par l’utilisation de ce logiciel ou matériel pour ce type d’applications.
Oracle et Java sont des marques déposées d’Oracle Corporation et/ou de ses affiliés.Tout autre nom mentionné peut correspondre à des marques
appartenant à d’autres propriétaires qu’Oracle.
Intel et Intel Xeon sont des marques ou des marques déposées d’Intel Corporation. Toutes les marques SPARC sont utilisées sous licence et sont des
marques ou des marques déposées de SPARC International, Inc. AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marques
déposées d’Advanced Micro Devices. UNIX est une marque déposée d’The Open Group.
Ce logiciel ou matériel et la documentation qui l’accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits et
des services émanant de tiers. Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ou
services émanant de tiers. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûts
occasionnés ou des dommages causés par l’accès à des contenus, produits ou services tiers, ou à leur utilisation.

iii
Contents
Using This Documentation vii
Understanding the Gateway 1
Gateway Overview 1
Physical Specifications 2
Environmental Requirements 2
Acoustic Noise Emissions 3
Electrical Specifications 3
NET MGT Connector and Pins 4
USB Management Connector and Pins 5
Data QSFP Connector and Pins 5
Understanding Cabling 9
Routing Service Cables 9
Power Cord Requirements 9
Management Cable Requirements 10
Understanding Data Cabling 11
Data Cable Cautions 12
Data Cable Guidelines 13
Data Cable Types 14
Data Cable Path Lengths 14
Data Cable Bundling 15
Floor and Underfloor Delivery of Data Cables 16

iv Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013
Overhead Delivery of Data Cables 16
Preparing to Install the Gateway 17
Installation Preparation 17
Suggested Tools for Installation 18
Antistatic Precautions for Installation 19
Installation Responsibilities 19
Installation Sequence 20
▼Verify Shipping Carton Contents 21
▼Assemble the Optical Fiber Data Cables 22
▼Route the Data Cables 25
Installing the Gateway 27
▼Install the Gateway in the Rack 27
Powering On the Gateway 33
▼Attach the Management Cables 33
▼Attach the Power Cords 36
Accessing the Management Controller 38
▼Access the Management Controller From the NET MGT Port 38
▼Access the Management Controller From the USB Management
Port 40
▼Verify the Gateway Status 41
▼Start the Subnet Manager 45
Connecting Data cables 47
▼Attach the Data Cables 47
▼Check Link Status 53
Verifying the InfiniBand Fabric 53
InfiniBand Node Description 54
▼Discover the InfiniBand Fabric Topology 54
▼Perform Diagnostics on the InfiniBand Fabric 55

Contents v
Index 57

vi Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013

vii
Using This Documentation
This installation guide provides detailed procedures that describe the preparation
and installation of the Sun Network QDR InfiniBand Gateway Switch from Oracle.
This document is written for technicians, system administrators, and users who have
advanced experience installing and administering InfiniBand fabric hardware.
■“Product Notes” on page vii
■“Related Documentation” on page vii
■“Feedback” on page viii
■“Access to Oracle Support” on page viii
Product Notes
For late-breaking information an known issues about this product, refer to the
product notes at:
http://docs.oracle.com/cd/E36256_01
Related Documentation
Documentation Links
Sun Network QDR InfiniBand Gateway Switch
Firmware Version 2.1
http://docs.oracle.com/cd/E36256_01

viii Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013
Feedback
Provide feedback on this documentation at:
http://www.oracle.com/goto/docfeedback
Access to Oracle Support
Oracle customers have access to electronic support through My Oracle Support. For
information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=
info or http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs visit
if you are hearing impaired.
Oracle Solaris 11 OS http://www.oracle.com/goto/Solaris11/docs
Oracle Integrated Lights Out Manager (ILOM) 3.0 http://docs.oracle.com/cd/E19860-01
All Oracle products http://docs.oracle.com
Documentation Links

1
Understanding the Gateway
These topics describe the purpose and specifications of the gateway, and the
connectors found on the gateway chassis.
■“Gateway Overview” on page 1
■“Physical Specifications” on page 2
■“Environmental Requirements” on page 2
■“Acoustic Noise Emissions” on page 3
■“Electrical Specifications” on page 3
■“NET MGT Connector and Pins” on page 4
■“USB Management Connector and Pins” on page 5
■“Data QSFP Connector and Pins” on page 5
Related Information
■“Understanding Cabling” on page 9
■“Preparing to Install the Gateway” on page 17
■“Installing the Gateway” on page 27
Gateway Overview
The Sun Network QDR InfiniBand Gateway Switch provides both InfiniBand fabric
switching and management capabilities and in-band access to 10GbE networks. The
gateway utilizes the OFED InfiniBand software stack and the BXOFED BridgeX
software stack through an internally configured management controller. The gateway
has 32 physical InfiniBand QDR ports and eight 10GbE ports available through
splitter cables.
Monitoring the many aspects of the gateway is simplified by a web-based Fabric
Monitor, allowing point and click polling and verification of settings, parameters,
and status.

2Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013
Related Information
■“Understanding Cabling” on page 9
■“Preparing to Install the Gateway” on page 17
■“Installing the Gateway” on page 27
Physical Specifications
This table lists the physical dimensions of the gateway.
Related Information
■“Verify Shipping Carton Contents” on page 21
■“Install the Gateway in the Rack” on page 27
Environmental Requirements
This table lists the gateway operating environment parameters.
Dimension Measurements
Width 17.52 in. (445.0 mm)
Depth 24 in. (609.6 mm)
Height 1.75 in. (44.5 mm)
Weight 23.0 lbs (11.4 kg)
Parameter Operating
Ambient temperature 41˚F to 89.6˚F (5˚C to 32˚C)
Relative humidity 5% to 85% noncondensing, 80˚F (27˚C) maximum wet bulb
Elevation (Sun requirement) Maximum 9840 feet (3000 meters) at 104˚F (40˚C)

Understanding the Gateway 3
Acoustic Noise Emissions
This table lists the permitted noise emissions of the gateway.
Electrical Specifications
This table lists the electrical requirements of the gateway.
Related Information
■“Power Cord Requirements” on page 9
■“Attach the Power Cords” on page 36
Parameter Operating Idling
Acoustic power LWAd (1B=10dB) 7.1 B 7.2 B
Acoustic pressure LpAm 58.9 dBA 59.0 dBA
Parameter AC Version Requirement
Voltage 100 VAC to 240 VAC single phase, 47 to 63 Hz
Current (per input) 5.4 A maximum per input at 100 VAC
Current (total) 5.6 A maximum total for all inputs at 100 VAC
Power 550 Watts (Total input power is approximately equally
divided among the operating power supplies)

4Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013
NET MGT Connector and Pins
This table lists the pinout of the NET MGT connector.
Related Information
■“Management Cable Requirements” on page 10
■“Attach the Management Cables” on page 33
■“Access the Management Controller From the NET MGT Port” on page 38
Pin. Signal
1 TXD+
2 TXD-
3 RXD+
4 Not used
5 Not used
6 RXD-
7 Not used
8 Not used

Understanding the Gateway 5
USB Management Connector and Pins
This table lists the pinout of the USB management connector.
Related Information
■“Management Cable Requirements” on page 10
■“Attach the Management Cables” on page 33
■“Access the Management Controller From the USB Management Port” on page 40
Data QSFP Connector and Pins
The QSFP connector is a single InfiniBand port connection and can carry InfiniBand
or Ethernet data.
Pin Signal
1 +5 VDC
2 - Data
3 + Data
4 GND

6Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013
This table lists the pinout for the connector.
This table provides descriptions of the QSFP signals.
Pin Signal Pin Signal Pin Signal Pin Signal
1 GND 11 SCL 21 RX2n 31 Reserved
2 TX2n 12 SDA 22 RX2p 32 GND
3 TX2p 13 GND 23 GND 33 TX3p
4 GND 14 RX3p 24 RX4n 34 TX3n
5 TX4n 15 RX3n 25 RX4p 35 GND
6 TX4p 16 GND 26 GND 36 TX1p
7 GND 17 RX1p 27 ModPrsL 37 TX1n
8 ModSelL 18 RX1n 28 IntL 38 GND
9 LPMode_Reset 19 GND 29 VccTx
10 VccRx 20 GND 30 Vcc1
Signal Description
GND Ground for both signal and power return
SDA I2C interface data
SCL I2C interface clock
ModSelL Module select on low - enables reception of I2C commands
ResetL Reset on low
LPMode Low power mode
ModPrsL Module presence on low - identifies existence of QSFP connector

Understanding the Gateway 7
Related Information
■“Data Cable Types” on page 14
■“Connecting Data cables” on page 47
IntL Interrupt on low - enables fault indication
Signal Description

8Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013

9
Understanding Cabling
These topics provide information that helps you understand the cabling
requirements of the gateway.
■“Routing Service Cables” on page 9
■“Understanding Data Cabling” on page 11
Related Information
■“Understanding the Gateway” on page 1
■“Preparing to Install the Gateway” on page 17
■“Installing the Gateway” on page 27
Routing Service Cables
These topics describe cable routing requirements:
■“Power Cord Requirements” on page 9
■“Management Cable Requirements” on page 10
Related Information
■“Understanding Data Cabling” on page 11
Power Cord Requirements
The power supplies are in a N+N redundancy. Line power is provided from two
sources, A and B.
Your gateway country kit should contain two power cords that are specific to your
country or application. This table describes the power cords available.

10 Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013
Caution – Install and route power cabling only in a manner that complies to federal,
state, and local electrical codes.
Related Information
■“Electrical Specifications” on page 3
■“Attach the Power Cords” on page 36
Management Cable Requirements
Management of the gateway is done at the management console, which is either a
10/100 Ethernet connection at the NET ports or a USB-to-serial device attached to the
USB port.
Typically, the NET connection (network management) is the default means of
communicating with the management controller. The controller has a DHCP client in
operation and requires the Ethernet network to have a DHCP server. The DHCP
server must be configured with the MAC address of the management controller, so
the server can provide an IP address to the management controller upon boot. If a
DHCP server is not available, then the USB connection is used.
Cable Part Number Description
X311L (180-1097) North America/Asia, IEC 320 C13 to NEMA 5-15P - 15A/125V 2.5M Black, RoHS:Y
X312E (180-1982) China, IEC 320 C13 to GB 2099/GB 1002 - 10A/250V 2.0M, RoHS:Y
X312F (180-1999) Argentina, IEC 320 C13 to IRAM 2073 - 10A/250V 2.0M Black, RoHS:Y
X312G (180-1662) Korea, IEC 320 C13 to KSC 8305 - 15A/250V 2.0M Black, RoHS:Y
X312L (180-1993) Continental Europe, IEC 320 C13 to CEE 7/7 10A/250V 2.0M Black, RoHS:Y
X314L (180-1994) Swiss, IEC 320 C13 to SEV 1011 - 10A/250V 2.0M Black, RoHS:Y
X317L (180-1997) U.K., IEC 320 C13 to BS 1363 - 10A/250V 2.0M Black, RoHS:Y
X332A (180-2121) Taiwan, IEC 320 C13 to NEMA 5-15P - 10A/125V 2.5M Black, RoHS:Y
X383L (180-1995) Danish, IEC 320 C13 to Asfnit 107 - 10A/250V 2.0M Black, RoHS:Y
X384L (180-1996) Italian, IEC 320 C13 to CEI 23-16/VII - 10A/250V 2.0M Black, RoHS:Y
X386L (180-1998) Australian, IEC 320 C13 to AS 3112 - 10A/250V 2.0M Black, RoHS:Y

Understanding Cabling 11
The advantage of the NET connection over the USB connection is that administration
of the gateway can happen from anywhere on the network. There is no cable length
constraint for the network management route because of the re-amplification,
filtering, and processing that happens at each hub or switch within the Ethernet
network. No network management cable should be any longer than 100 meters.
The USB connection requires a USB-to-serial adapter. The adapter must be
configured to communicate with your serial device management console. The serial
device can be a serial terminal, a terminal server, or a serial connection running on a
system or laptop. Because of the nature of the serial signal, a serial management
cable cannot be used reliably if it is more than 10 meters long.
The USB-to-serial adapter is not included with your gateway. You can purchase such
an adapter from computer and electronics stores.
Related Information
■“NET MGT Connector and Pins” on page 4
■“USB Management Connector and Pins” on page 5
■“Attach the Management Cables” on page 33
Understanding Data Cabling
These topics describe data (InfiniBand, Ethernet) cabling:
■“Data Cable Cautions” on page 12
■“Data Cable Guidelines” on page 13
■“Data Cable Types” on page 14
■“Data Cable Path Lengths” on page 14
■“Data Cable Bundling” on page 15
■“Floor and Underfloor Delivery of Data Cables” on page 16
■“Overhead Delivery of Data Cables” on page 16
Related Information
■“Routing Service Cables” on page 9
■“Connecting Data cables” on page 47

12 Sun Network QDR InfiniBand Gateway Switch Installation Guide for Firmware Version 2.1 • March 2013
Data Cable Cautions
To prevent data cable damage, you must follow these cautions:
Related Information
■“Data Cable Guidelines” on page 13
■“Data Cable Types” on page 14
■“Data Cable Path Lengths” on page 14
■“Data Cable Bundling” on page 15
Do not uncoil the cable, as a kink
might occur. Hold the coil closed
as you unroll the cable, pausing
to allow the cable to relax as it is
unrolled.
Do not step on the cable or
connectors. Plan cable paths away
from foot traffic or rolling loads.
Do not pull the cable out of the
shipping box, through any
opening, or around any corners.
Unroll the cable as you lay it
down and move it through turns.
Do not bend the cables to a radius
tighter than 85 mm (3.4 inches).
Ensure that cable turns are as
wide as possible.
Do not twist the cable to open a
kink. If it is not severe, open the
kink by unlooping the cable.
Do not pack the cable to fit a tight
space. Use an alternative cable
route.
Do not straighten the cable to
correct a bend that is too tight.
Leave the cable bend as is.
Do not hang the cable for a length
more than 2 meters (7 feet).
Minimize the hanging weight
with intermediate retention
points.
Do not drop the cable or
connectors from any height.
Gently set the cable down, resting
the cable connectors on a stable
surface.
Do not cinch the cable with hard
fasteners or cable ties. Use soft
hook-and-loop fastener for
bundling and securing cables.
Do not drag the cable or its
connectors over any surface.
Carry the entire cable to and from
the points of connection.
Do not force the cable connector
into the receptacle by pushing on
the cable. Apply connection or
disconnection forces at the
connector only.