Voltaire 4036 (QDR) User manual

Grid Director
4036 (QDR) / 2036 (DDR)
August 2010
Doc P/N: DOC-00467
Doc Rev: A09

VOLTAIRE, INC. AND ITS AFFILIATES (“VOLTAIRE”) FURNISH THIS DOCUMENT “AS IS,” WITHOUT WARRANTY OF
ANY KIND. VOLTAIRE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION,
THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-
INFRINGEMENT AND THOSE ARISING FROM A COURSE OF PERFORMANCE, A COURSE OF DEALING, OR TRADE
USAGE. VOLTAIRE SHALL NOT BE LIABLE FOR ANY ERROR, OMISSION, DEFECT, DEFICIENCY OR
NONCONFORMITY IN THIS DOCUMENT AND DISCLAIMS ALL LIABILITY, INCLUDING LIABILITY FOR INFRINGEMENT
OF ANY INTELLECTUAL PROPERTY RIGHTS RELATED TO THE INFORMATION CONTAINED IN THIS DOCUMENT.
No license, expressed or implied, to any intellectual property rights is granted under this document. This document, as well
as the software described in it, are furnished under a separate license and shall only be used or copied in accordance with
the terms of the applicable license. The information in this document is furnished for informational use only, is subject to
change without notice, and should not be construed as any commitment by Voltaire. Except as permitted by the applicable
license, no part of this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any
means without the express written consent of Voltaire.
Names and logos identifying products of Voltaire in this document are registered trademarks or trademarks of Voltaire. All
other trademarks mentioned in this document are the property of their respective owners.
Copyright © 2010 Voltaire, Inc. All rights reserved.
Voltaire Doc P/N: DOC-00467
Business Headquarters
Voltaire Inc.
100 Apollo Drive
Chelmsford, MA 01824
USA 01821
Tel: [phone]
Fax: [phone]
R&D Center
Voltaire Ltd.
13 Zarchin St
Raanana 43662
Israel
Tel: [phone]
Fax: [phone]

Contents
Contents ............................................................................................................................................... iii
Figures ................................................................................................................................................ v
Tables ............................................................................................................................................... vi
About this Manual..................................................................................................................................... vii
Chapter 1. Introduction ..........................................................................................................................1
4036/2036 Switch Overview...............................................................................................................2
4036/2036 Switch Main Features.......................................................................................................3
Management ......................................................................................................................................4
Chapter 2. Unpacking.............................................................................................................................5
Overview ............................................................................................................................................6
Package Contents..............................................................................................................................6
Unpacking the 4036/2036 Switch.......................................................................................................7
Rail Kit (KIT-00008)............................................................................................................................9
Multipurpose (Mng) Console Cable Kit ............................................................................................10
Packing the 4036/2036 Switch.........................................................................................................10
Chapter 3. Getting Started ...................................................................................................................12
Overview ..........................................................................................................................................13
Front Panel Description....................................................................................................................13
Rear Panel Description ....................................................................................................................15
Chapter 4. Product Options.................................................................................................................17
4036/2036 Switch Available Configurations.....................................................................................18
InfiniBand Clusters Using the 4036/2036 Switch.............................................................................18
Ordering Information ........................................................................................................................20
Chapter 5. Technical Specifications and Certifications....................................................................22
4036/2036 Switch Technical Specifications.....................................................................................23
Power Specifications - Power Supply Module .................................................................................24
Certifications.....................................................................................................................................25
Gost-R Certification..........................................................................................................................26
Declarations .....................................................................................................................................26
Declaration of Conformity......................................................................................................26
Hazardous Substances (RoHS 5) Compliance Declaration..................................................27
Label(s) ............................................................................................................................................27
Shock and Vibration.........................................................................................................................28
Chassis Clearance Requirements....................................................................................................28
Weights and Dimensions .................................................................................................................29
Acoustic Data...................................................................................................................................29

Contents
Doc P/N DOC-00467 A09
4036/2036 Installation Manual Page iv
Export Information............................................................................................................................30
Shipping Restrictions........................................................................................................................30
Chapter 6. Hardware Installation.........................................................................................................31
Overview ..........................................................................................................................................32
Installation Options...........................................................................................................................32
Cooling...................................................................................................................................32
Required Tools.................................................................................................................................33
Site Planning....................................................................................................................................33
Rack and Clearance Requirements ......................................................................................33
Site Environment Specification..............................................................................................34
Power Requirements.............................................................................................................34
Site Preparation Checklist.....................................................................................................34
Measuring the Distance between Mounting Rails............................................................................35
Assembling and Mounting the Rail...................................................................................................35
Rail Kit35
Option 1.................................................................................................................................37
Option 2.................................................................................................................................43
4036/2036 Switch Power Up............................................................................................................48
Field Replaceable Units ...................................................................................................................48
Fan Unit.................................................................................................................................48
Power Supplies......................................................................................................................49
Chapter 7. Cabling ................................................................................................................................51
Overview ..........................................................................................................................................52
Cabling Guide Bracket Installation...................................................................................................52
InfiniBand Cabling - QDR.................................................................................................................55
Multipurpose Management (Mng) Console Cable ...........................................................................56
Chapter 8. Operation ............................................................................................................................58
Overview ..........................................................................................................................................59
Powering on the 4036/2036 Switch..................................................................................................59
4036/2036 Switch LED Indicators....................................................................................................60
Where to Go Next.............................................................................................................................61
Chapter 9. Troubleshooting.................................................................................................................62
Overview ..........................................................................................................................................63
Solving Startup Problems.................................................................................................................63
Preparation before Contacting Customer Service ...........................................................................64
Appendix A Option 3 – Chassis Installation in an iDataPlex Rack ....................................................65
Unpacking the 4036/2036 Switch.....................................................................................................65
Mounting the iDataplex Kit...............................................................................................................67
Appendix B Cabling Information ...........................................................................................................72
IB Port Cable Specifications.............................................................................................................72
1 GbE Ports (Management).............................................................................................................73
Appendix C QSFP Cable.........................................................................................................................74

Figures
Figure 1-1. 4036/2036 Front View ................................................................................................................2
Figure 1-2. 4036/2036 Rear View.................................................................................................................3
Figure 2-1. 4036/2036 Cardboard Package (Open) - Standard ...................................................................7
Figure 2-2. Removing the 4036/2036 Switch from the Cardboard Package................................................8
Figure 2-3. Fixed Rail (KIT-00008) ...............................................................................................................9
Figure 2-4. Console Cable Kit (Mng) ..........................................................................................................10
Figure 2-5. Packing the 4036/2036 Switch.................................................................................................11
Figure 3-1. 4036/2036 Front Panel.............................................................................................................13
Figure 3-2. 4036/2036 Rear Panel..............................................................................................................15
Figure 4-1. InfiniBand Cluster Using 4036 Switch Single-hop Configuration .............................................18
Figure 4-2. InfiniBand Cluster Using 2036 Switch Single-hop Configuration .............................................18
Figure 4-3. InfiniBand Cluster Using 4036/2036 Switch Triple-hop Configuration .....................................19
Figure 4-4. InfiniBand Cluster Using 4036/2036 Switch Three-hop Configuration.....................................19
Figure 5-1. 4036 Label (Example) ..............................................................................................................27
Figure 5-2. 2036 Label (Example) ..............................................................................................................28
Figure 6-1. 4036/2036 – Option 1 Assembly ..............................................................................................38
Figure 6-2. Mounting the long bracket........................................................................................................39
Figure 6-3. Positioning and aligning the mounting bar and long bracket....................................................39
Figure 6-4. Securing the mounting bar .......................................................................................................39
Figure 6-5. Installing the clip nuts and placing the power cord...................................................................40
Figure 6-6. Inserting the switch into the rack..............................................................................................40
Figure 6-7. Power cord over the long bracket.............................................................................................40
Figure 6-8. Securing power cords...............................................................................................................41
Figure 6-9. Connecting the power cables...................................................................................................41
Figure 6-10. Secure the long brackets to the rack rail................................................................................41
Figure 6-11. Mounting the short bracket.....................................................................................................42
Figure 6-12. Adjusting the short bracket.....................................................................................................42
Figure 6-13. Securing Screws and Brackets...............................................................................................42
Figure 6-14. Switch mounted in a rack – Rack rear view ...........................................................................43
Figure 6-15. Switch mounted in a rack – Rack front view...........................................................................43
Figure 6-16. 4036/2036 – Option 2 Assembly ............................................................................................44
Figure 6-17. Mounting the short bracket.....................................................................................................45
Figure 6-18. Mounting the long bracket......................................................................................................45
Figure 6-19. Positioning and aligning the mounting bar and long bracket..................................................45
Figure 6-20. Securing the rail......................................................................................................................46
Figure 6-21. Placing the clip nuts on the vertical rack rails.........................................................................46
Figure 6-22. Insert the switch into the rack.................................................................................................46
Figure 6-23. Secure the rack screws..........................................................................................................46
Figure 6-24. Installing the long bracket.......................................................................................................47
Figure 6-25. Securing the rack screws .......................................................................................................47

Figures
Doc P/N DOC-00467 A09
4036/2036 Installation Manual Page vi
Figure 6-26. Securing the long bracket screws to the mounting bar..........................................................47
Figure 6-27. Switch mounted in a rack – Rack front view...........................................................................47
Figure 6-28. Replacing the Fan ..................................................................................................................48
Figure 6-29. Replacing the Power Supply ..................................................................................................49
Figure 7-1. Cabling Guide Bracket (CG-24)................................................................................................52
Figure 7-2. Installing a Cabling Guide Bracket ...........................................................................................53
Figure 7-3. Installing a Cabling Guide Bracket (sliding rail)........................................................................53
Figure 7-4. Cable Management..................................................................................................................53
Figure 7-5. Cabling......................................................................................................................................54
Figure 7-6. Console Cable Kit (Mng) ..........................................................................................................56
Figure A-1. 4036/2036 Cardboard Package (Open) - Standard.................................................................66
Figure A-2. 4036/2036 – Assembly in an iDataplex Rack ..........................................................................68
Figure A-3. Attaching the mount bracket ....................................................................................................69
Figure A-4. Securing the mount holders.....................................................................................................69
Figure A-5. Securing clip nuts and the mount plates onto the rack............................................................70
Figure A-6. Inserting the mount holder into the mount plate.......................................................................70
Figure A-7. Securing the front of the chassis..............................................................................................71
Tables
Table 3-1. 4036/2036 Front Panel Components........................................................................................14
Table 3-2. Rear Panel Components ..........................................................................................................16
Table 4-1. 4036/2036 Switch Ordering Information...................................................................................20
Table 4-2. 4036/2036 Module Ordering Information..................................................................................20
Table 4-3. 4036/2036 Switch Packing List (According to Configuration)...................................................21
Table 5-1. 4036/2036 Switch Technical Data............................................................................................23
Table 5-2. Power Supply Module Data......................................................................................................24
Table 5-3. Certifications.............................................................................................................................25
Table 5-4. Clearance Requirements..........................................................................................................28
Table 5-5. Unpacked Weights & Dimensions ............................................................................................29
Table 5-6. Packed Weights & Dimensions.................................................................................................29
Table 5-7. Export Information ....................................................................................................................30
Table 6-1. Material and Tools Required.....................................................................................................33
Table 6-2. Site Preparation Checklist ........................................................................................................34
Table 6-3. Rail Kit (P/N KIT-00008) Detailed Part List...............................................................................36
Table 8-1. 4036/2036 Switch LED Indications...........................................................................................60
Table 9-1. iDataplex Mounting Kit (P/N KIT-00004) Part List....................................................................67
Table 10-2. 4X InfiniBand Port Cabling Specifications..............................................................................72

About this Manual
This manual provides installation instructions for the Voltaire high-bandwidth, low-latency
scalable 36-port InfiniBand 4036 QDR and 2036 DDR chassis. It includes the product
specifications, unpacking and installation information, unit power up, and initiation and
troubleshooting procedures.
NOTE:
Refer to the Voltaire release notes for last minute updates and restrictions
Contact Us:
Please send your documentation-related comments and feedback or report mistakes to:
[email protected].
Audience
The manual is intended primarily for system administrators who are authorized to install a
4036 or a 2036 system.
It is assumed that the readers are familiar with the InfiniBand technology and terminology.
Document Conventions
The following lists conventions used in this document.
NOTE: Identifies important information that contains helpful suggestions.
CAUTION: Alerts you to the risk of personal injury, system damage, or loss of data.
WARNING: Warns you that failure to take or avoid a specific action might result in personal
injury or a malfunction of the hardware or software. Be aware of the hazards involved with
electrical circuitry and be familiar with standard practices for preventing accidents before you
work on any equipment.
Text set off in this manner indicates terminal/CLI screen/command
format.

About this Manual
Doc P/N DOC-00467 A09
4036/2036 Installation Manual Page viii
Document Organization
Chapter 1 – Introduction: Brief overview and graphical representation of the switch.
Chapter 2 – Unpacking: Provides a full detail of the package contents and the rai. kit, as
well as step-by-step instructions on how to unpack the 4036/2036.
Chapter 3 – Getting Started: Provides a description of the front and rear panels and of
the product architecture, including a description of the various components.
Chapter 4 – Product Options : Lists the Voltaire 4036/2036 configurations, provides
examples of 4036/2036 in various clusters, and provides the necessary data for ordering
a product configuration, as well as comprehensive technical specifications.
Chapter 5 – Technical Specifications and Certifications: Provides specifications,
certifications, declarations, requirements, weights and dimensions, clearance
information, export information, and shows the label applied on the product’s box,
Chapter 6 – Hardware Installation: Provides detail on how to prepare your site for
installation, preparation and installation of the 4036/2036. It also provides instructions on
how assemble the mounting rails (two options), and replace the Field Replaceable Units.
Chapter 7 – Cabling: Provides information on cabling guide installation, InfiniBand
cabling, including do’s and don’ts, InfiniBand Cable specifications as well as on the
management cable.
Chapter 8 – Operation: Describes how to start up the unit and use the unit indicators.
Chapter 9 – Troubleshooting: Helps you solve startup problems and provides detail on
customer service contacts.
Appendix A – Option 3: Chassis Installation in an iDataPlex Rack.
Appendix B – Cable and Port Pinout: Provides cabling and port pinout information for the
4036/2036.
Appendix C – QSFP Cable: Provides a description of the QSFP cable and connector.
Related Documentation
In addition to this guide, the following product documentation is available:
4036/2036 Getting Started Guide, Voltaire [LIT-00037]
4036/2035 User Manual, Voltaire [DOC-00619]
NOTE:
Please remember to review all product release notes for known issues and limitations.

1
1
Chapter 1. Introduction
This chapter includes the following sections:
4036/2036 Switch Overview......................................................................................................2
4036/2036 Switch Main Features..............................................................................................3

Introduction
Doc P/N DOC-00467 A09
4036/2036 Installation Manual Page 2
4036/2036 Switch Overview
The Voltaire Grid Director 4036/2036 is a high performance, low latency and fully non-
blocking InfiniBand switch for high performance clusters. Delivering 2.88 Tbps of non-
blocking bandwidth with less than 100 nanoseconds of latency (port-to-port), I/O bottlenecks
are removed making applications operate at maximum efficiency.
The 4036/2036 was designed to fit into today’s densely configured racks. It has thirty-six
40Gbps ports that use the new smaller and intelligent QSFP connector in a 1U chassis that is
only 15” deep. The efficient 4036/2036’s smart design makes it is easy to build clusters that
can scale-out into the thousands of nodes.
To meet varying application needs, the 4036 offers Quad Data Rate 4X ports (QDR provides
a date rate of 40, 20 or 10 Gb/s auto-negotiable) while the 2036 offers Dual Data Rate 4X
ports (DDR provides a data rate of 20 or 10 Gb/sec auto-negotiable).
The 4036/2036 switch incorporates redundant, hot-swappable power supplies for High
Availability, as well as a hot-swappable fan unit. The 4036/2036 offers a plug-and-play
environment, allowing servers to be added without taking down the fabric.
The 4036/2036 includes a high-performance CPU for management purposes.
Grid Directors 2036 and 4036 are identical in appearance, distinguishable only by the product
ID label on the chassis.
The following figure shows the front view of a 4036 Unit. The front view of the 2036 is
identical.
Figure 1-1. 4036/2036 Front View