Mellanox Technologies MSX6536-10R Manual

www.mellanox.com
648 Port InfiniBand FDR Switch Platform
Hardware Installation Guide
PN: MSX6536-10R, MSX6536-NR
Rev 1.4

Document Number: 3504
Mellanox Technologies2
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
Beit Mellanox
PO Box 586 Yokneam 20692
Israel
www.mellanox.com
Tel: [phone]
Fax: [phone]
© Copyright 2013. Mellanox Technologies. All Rights Reserved.
Mellanox®, Mellanox logo, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®, PhyX®,
SwitchX®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of Mellanox Technologies, Ltd.
Connect-IB™, FabricIT™, MLNX-OS™, MetroX™, ScalableHPC™, Unbreakable-Link™, UFM™ and Unified Fabric
Manager™ are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE
THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT
HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE
PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND
DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST
QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES
FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND
(INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED
DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4
Mellanox Technologies 3
Table of Contents
Chapter 1 Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.1 Environmental and Safety Recommendations . . . . . . . . . . . . . . . . . . . . . 8
1.2 Chassis Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 Leaf Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Spine Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.5 Management Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 Physical Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6.1 Starting with the Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.6.2 ESD Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.6.3 Installation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6.4 Installing the Cable Holder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
1.6.5 Ground Connections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
1.7 Power Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
1.8 Powering Up the Switch Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Chapter 2 InfiniBand QSFP Cable Installation. . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.1 Port Connector Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.2 Supported Approved Cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.3 Cable Power Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Chapter 3 Chassis Power Up. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.1 Power Supply, Management Board, and Spine Board Indicator Status at Power
ON 38
Chapter 4 InfiniBand Fabric Initialization and Management . . . . . . . . . . . . . . 39
4.1 Configuring the Switch Management Modules for the First Time . . . . . . 39
4.2 Rerunning the Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Chapter 5 MLNX-OS Management Software . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1 Starting an SSH Connection to the Switch . . . . . . . . . . . . . . . . . . . . . . . 45
5.2 Starting a WebUI Connection to the Switch . . . . . . . . . . . . . . . . . . . . . . 45
Chapter 6 Resetting the Switch – RST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Chapter 7 TroubleShooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7.1 Power Supply Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7.2 Leaf Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7.3 Management Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.3.1 Yellow Status LED (for the Chassis) on the Management Module is Lit . 49
7.3.2 Yellow LED for the Leaf Fan on the Management Module is Lit . . . . . . . 49
7.3.3 Yellow LED for the Spine Fan on the Management Module is Lit . . . . . . 50
7.4 Spine Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.5 MLNX-OS Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Mellanox Technologies
4
List of Figures
Figure 1: Distance Between the Vertical Supports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Figure 2: Installation Kit Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Figure 3: Shock and Tilt Stickers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Figure 4: Container Screws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Figure 5: Vertical Rack Supports Dimension. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Figure 6: Adjustable Shelf Brackets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Figure 7: Placing the Shelf Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Figure 8: Caged Nut Locations for the Shelf Brackets . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Figure 9: Shelf Brackets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Figure 10: U Numbering Above Shelf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Figure 11: Putting the Chassis in the Rack Using a Fork Lift . . . . . . . . . . . . . . . . . . . . . . . 21
Figure 12: Putting the Chassis in the Rack Using a Mechanical Lift. . . . . . . . . . . . . . . . . . 22
Figure 13: Place the Upper Bracket Offset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Figure 14: Attach the Shelf Extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Figure 15: Place Caged Nuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Figure 16: Location for the Lock-down Bar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Figure 17: Placement of Chassis in Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Figure 18: Fork Lift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Figure 19: Mechanical Lift. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Figure 20: Face Plate Mounting Bolt Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Figure 21: Upper Bracket Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Figure 22: Upper Brackets and Offset Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Figure 23: Offset to Upper Bracket Bolts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Figure 24: Location for the Lock-down Bar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Figure 25: Lock-Down Bar. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Figure 26: Lock Down Bars Bolt and Washer Order. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Figure 27: Cable Holder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Figure 28: Placement of Chassis in Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Figure 29: Ground Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Figure 30: Multiple Power Inlets - Electric Caution Notification. . . . . . . . . . . . . . . . . . . . . . 34
Figure 31: Spine Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Figure 32: Port Numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Figure 33: Top and Bottom Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Figure 34: Spine Side Panel Display Status Indications . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Figure 35: Management Module Status Indications for Normal Operation . . . . . . . . . . . . . 38
Figure 36: Connecting the Harness to the Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Figure 37: Web UI Login Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Figure 38: Reset Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4
Mellanox Technologies 5
List of Tables
Table 1: Revision History of this Installation Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Table 2: Reference Documents and Web Sites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Table 3: Serial Terminal Program Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Table 4: Configuration Wizard Session - IP Configuration by DHCP. . . . . . . . . . . . . . . . 40
Table 5: Configuration Wizard Session - IP Zeroconf Configuration . . . . . . . . . . . . . . . . 42
Table 6: Configuration Wizard Session - Static IP Configuration. . . . . . . . . . . . . . . . . . . 43

Mellanox Technologies6
Revision History
About this Manual
This manual provides installation and set-up instructions for the SX6536 QSFP Chassis InfiniBand
Switch Platform.
Intended Audience
This manual is intended for users and system administrators responsible for installing and setting
up the switch platform.
The manual assumes familiarity with the InfiniBand®architecture specification.
Related Documentation
The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the
following:
Table 1 - Revision History of this Installation Guide
Revision Date Details
1.4 January 2013 Added new figures
Added nut for ground post.
1.3 January 2012 Added N+N information
Added bookmarks
1.2 January 2012 Added Troubleshooting section
Updated Initial Configuration to include IPv6
New Address for Israel
New copyright
1.1 Nov 2011 Added shock stickers
Aligned the UM with the IG procedures.
1.0 July 2011 Initial Release

648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4
Mellanox Technologies 7
Table 2 - Reference Documents and Web Sites
Document Conventions
Throughout this manual, the name SX6536 and the terms chassis and switch are used to
describe the switch, unless explicitly indicated otherwise.
The following pictures are used throughout this document to indicate information that is
important to the user.
InfiniBand Architecture Specification, Vol. 1,
Release 1.2.1 The InfiniBand Architecture Specification that is provided by IBTA
Switch Hardware Errata For any possible errata due to hardware issues see the switch
support product page. This requires a customer support login.
Mellanox MLNX-OS™ SwitchX® Software WebUI
User’s Manual WebUI Overview for MLNX-OS software.
Mellanox MLNX-OS™ SwitchX® Software User
Manual This document contains information regarding configuring and managing Mellanox Technologies' SwitchX® Switch Platforms.
MLNX-OS™ Software Command Reference Guide Command Reference Guide for MLNX-OS listing all of the
commands available through MLNX-OS with explanations and
examples.
MLNX-OS™ Software Configuration Guide Configuration Guide for MLNX-OS displaying different config-
uration scenarios.
This symbol makes recommendations to the user.
This symbol indicates information that is helpful to the user.
This symbol indicates a situation that can potentially cause damage to hardware or
software.
WARNING! This symbol indicates a situation that can potentially cause personal
injury and / or damage to hardware or software.

Installation
Mellanox Technologies8
1Installation
Installation and initialization of the chassis is a simple process requiring attention to the normal
mechanical, power, and thermal precautions for rack-mounted equipment. Your chassis comes
only with the power supplies and fans pre-installed. The rest of the openings are populated with
blanks. All of the leafs, spines, and management modules come shipped in a separate package.
The chassis requires initial configuration to get the chassis and Fabric management up and run-
ning through remote management. See the Installation Guide that is packed in the box for the
instructions to make the initial configuration.
1.1 Environmental and Safety Recommendations
The following are Mellanox recommendations.
This chassis can be installed in standard 19” racks that have depths between 65cm and
80cm between the vertical supports of the rack.
This unit is intended for installation in a Restricted Access Location. A restricted
access area can be accessed only through the use of a special tool, lock and key, or
other means of security.
Recommended ambient temperature in the System room is 20o± 5oC.
Recommended humidity range is 40% ± 15% without condensing.
It is highly recommended that the installation sites be as isolated as possible from all
sources of radio transmissions and electrical interference.
It is highly recommended that the installation site building be equipped with a light-
ning rod.
It is highly recommended that the installation site be equipped with smoke detectors
and a fire alarm warning system.

648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4
Mellanox Technologies 9
1.2 Chassis Package Contents
The package includes:
•1 chassis with the following modules installed:
•8fans
• 10 PSUs
•1 Installation Guide
•1 chassis
•1 box containing various other parts
•1 installation kit box
•1 box containing 10 power cords 250v 15a 2.0M, C14 to C13, USA UL Standard
•1 cable management kit
Before you install your new SX6536 series chassis, unpack the system and check to
make sure that all the parts have been sent, check this against the parts list. Check the
parts for visible damage that may have occurred during shipping.
1.3 Leaf Package Contents
The leafs are ordered by the customer and are shipped 4 to a box. The customer will
receive as many boxes as needed to fill the order.
The system requires a KVA rated UPS system. It is recommended that a UPS system
be installed to protect the equipment in the event of unexpected power failure.
Make sure that the outlets and circuits will not be overloaded. Spread out the load over
at least two or three circuits or use a 3 phase circuit.
If anything is damaged or missing, contact your customer representative immediately.
The rack mounting holes conform to the EIA-310 standard for 19-inch racks. Guaran-
tee proper ventilation, by leaving 8cm (3”) of space to the front and rear of the switch.
This will ensure proper air flow through the chassis. This is crucial for maintaining
good airflow at ambient temperature. In particular, route cables such that they do not
impede the air into or out of the chassis.

Installation
Mellanox Technologies10
1.4 Spine Package Contents
The spines are shipped 3 to a box. The customer will receive as many boxes as needed to fill the
chassis with a full complement of spines.
1.5 Management Package Contents
The package includes:
•all of the management modules ordered by the customer
•1 RJ45 to DB9 harness for each management module received
1.6 Physical Installation
The switch platform uses 31U of rack pace in a standard 19” rack, 29U for the chassis and 2U for
the shelf. The switch ships from the factory with mounting holes on the spine side. There are
upper brackets to connect the leaf side to the rack near the top of the chassis, and there are two
lock-down bars to secure the chassis to the shelf. The weight of the switch is supported from
underneath the unit by the shelf.
This chassis can be installed in standard 19” racks that have between 65cm and 80cm between
the vertical supports of the rack. Make sure that a fully populated rack including cables will have
sufficient air flow for cooling.
Insert the bottom spine board first.
Warning: This equipment is very heavy. Safety is the first concern. Make sure that
adequate manpower and proper equipment is used for transporting and moving the
chassis.
The fully loaded chassis weighs:
327.15kg (721.24 LBS.) full configuration
128.97kg (284.33 LBS.) empty configuration
157.57kg (347 LBS.) shipped configuration
Choose a rack which is able to support the mechanical and environmental characteris-
tics of a fully populated switch chassis.

648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4
Mellanox Technologies 11
1.6.1 Starting with the Rack
1.6.1.1 Rack Recommendation
1.6.1.2 Preparing the Rack
The rack may need to be modified to accommodate the SX6536 chassis. The distance
between the vertical supports must be between 65 and 80 cm.
Figure 1: Distance Between the Vertical Supports
The vertical supports may have to be moved so that the vertical support closest to the
door is a minimum of 6 cm from the inside of the door. This is required so that the shelf
will not hit the door.
1.6.2 ESD Connection
Before starting any procedure on the SX6536 switch system:
Due to the space required by up to 648 connector cables Mellanox recommends a rack
that is 120 cm long and 80cm wide. This will allow for proper cable management and
enough ventilation to properly cool the chassis.
If you move the vertical supports make sure that you reinstall them vertically
plumb.
This edge of the shelf must be
as close as possible to the
inside of the door when
closed.
Vertical Supports
Between 65 and 80 cm.

Installation
Mellanox Technologies12
1. Put an ESD prevention wrist strap on your wrist, and make sure there is good contact between
your body and the strap.
2. Plug the other end of the wrist strap to a valid ground. Make sure that this is a tight fit.
1.6.3 Installation Procedure
1.6.3.1 Requirements
You will need:
1.6.3.2 Installation and Cable Management Kit Parts
• Parts included in the installation kit:
Installing the chassis will be a lot easier if you can remove the sides of the rack.
You will need a fork lift, to move and insert the chassis into the rack.
• #2 phillips screwdriver • #3 phillips screwdriver
• a fork lift • a grounding lug
• measuring tape • ground wire of sufficient length and gauge to properly ground the
chassis
The installation will be much easier with a power screwdriver.
It is recommended to use AWG6 or 4mm diameter wire for grounding purposes.
It is recommended to have at least two people for the duration of the installation proce-
dure. Use a mechanical lift to raise this chassis. If not, use enough manpower to ensure
the safety and wellbeing of all of the people involved in the installation.
- 1 shelf with brackets pre-installed - 1 shelf extension
- 1 left side upper bracket offset - 1 right side upper bracket offset
- 2 lock down bars - 2 upper bracket washers
- 8 lock washers for the upper brackets to
vertical support - 12 lock washers for the upper brackets to upper bracket offsets
- 12 hex head bolts – 8 for the upper 
brackets and 4 for the shelf extension - 34 caged nuts 20 for the shelf and 14 for the chassis faceplate

648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4
Mellanox Technologies 13
• Parts included in the cable management installation kit:
Figure 2: Installation Kit Parts
1.6.3.3 Container Mishandling
The container has shock and tilt stickers applied. These will turn red if the container has
been mishandled or roughly handled. Upon receipt of the container look for and inspect
the shock and tilt stickers to confirm that they have not tripped. If one or more are red
- 34 pan head screws for the caged nuts 20
for the shelf and 14 for the chassis face-
plate
- 20 flat washers– 8 for the lock down bars
and 12 for the upper brackets
- 32 lock washers – 20 for the lock down
bars and 12 for the upper brackets to
upper bracket offsets
- 32 hex head bolts – 20 for the lock down
bars and 12 for the upper brackets to upper
bracket offsets
- 2 upper brackets - 12 flat head screws with locktite for the
upper bracket offsets
- 1 open end ratchet wrench 10mm - 1 socket nut driver 8mm
- 1 socket wrench - 1 socket nut driver 10mm
- 2 extensions for the socket driver - 1 socket nut driver 14mm
- 4 flat washers for the shelf extension - 1 box wrench
- 1 nut Size M-8 for ground post - 6 two hole lock washers for lock down bars
- 1 cable management rack RH - 1 cable management rack LH
- 16 caged nuts M6 - 16 M6 bolts
Shelf
Caged nuts
Upper bracket
Lock washers
#6 hex head bolt
Phillips pan head 
screw
Upper bracket offset
Upper bracket
Shelf extension
Lock-down bar
Flat washers
#8 hex head bolt
Flat washers
Flat head screws with
locktite
Lock washers

Installation
Mellanox Technologies14
notify the shipper and Mellanox. This on its own does not indicate damage to the contents. But,
be sure to carefully inspect the contents if any of the shock and tilt stickers have tripped.
Figure 3: Shock and Tilt Stickers
1.6.3.4 Opening the Container
Step 1. Before starting the procedure, put the ESD strap on and connect it to a valid ground.
HANDLEWITHCAREHANDLEWITHCARE
GGIINNNNRRWWAAGG
II
NNNNRRWWAA
HANDLEWITHCAREHANDLEWITHCARE
GG
II
NNNNRRWWAAGG
II
NNNNRRWWAA
TI L TWA
T
CH
TIL TWA
T
CH
Stickers unaffected by 
shipping
Stickers showing excessive
shock or tilt.

648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4
Mellanox Technologies 15
Figure 4: Container Screws
Step 1. Remove the four sides of the container by unscrewing the phillips head screws
holding the sides on.
Step 2. Remove and put aside the box.
Step 3. Visually inspect the chassis, make sure that:
• there is no visible damage
• 10 PSUs are installed for the chassis
• all 8 fans are installed
Step 4. Remove all protective plastic film from all sides and top of the chassis.
It is highly recommended to have a screw gun or electric screwdriver for this step.
The leafs, spines, and management modules are shipped separately.
Container screws

Installation
Mellanox Technologies16
1.6.3.5 Installing the Shelf
Step 1. Select the location in the rack. Try to keep the chassis as low as possible to keep the
center of gravity low.
The maximum height that the chassis can be mounted, will be determined by the maximum
height that the fork lift can raise the pallet.
Step 2. Open the Installation kit and make sure that all of the parts are in the box. See
Figure 2,“Installation Kit Parts”.
The chassis will sit in the rack with the spine side as close as possible to the door. This leaves
maximum room for the cable management system.
Do not remove any of the blocks or either of the lock-down bars at this time.
The chassis is on ball bearings and can roll easily. Be aware that the chassis can roll off
of the pallet. Should the chassis start to fall, move out of the way. The chassis can
cause grave bodily harm should it fall on or near you.
It is recommended to set the top of the shelf at U5. Try and keep the chassis as low as
possible. Set the shelf from U3 to U5.
Placing the shelf too low may make it impossible to tighten the shelf brackets, in step 9
The distance between the vertical supports must be between 65 cm(25.6”) and 80
cm(31.5”). If the distance in your rack is different, move the vertical supports so that
one pair is ~10 cm(4in.) away from the door.
The side of the chassis with the spine units will sit flush with the vertical supports of
the rack. The side of the chassis with the QSFP connectors will sit at the edge of the
shelf closer to the center of the rack.

648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4
Mellanox Technologies 17
Figure 5: Vertical Rack Supports Dimension
Step 3. Measure the distance between the vertical supports and adjust the shelf brack-
ets according to this distance. This chassis can be installed in a rack with a
distance between the vertical supports of from 65cm(25.6in.) to
80cm(31.5in.).
Step 4. The shelf comes with the four brackets pre-installed. Two of the brackets
should be loose. They may need to be removed and repositioned to meet the
distance between the vertical supports of the rack.
Figure 6: Adjustable Shelf Brackets
65-80cm
Vertical Supports
These two brackets will
need to be adjusted. They
may even need to be
removed and repositioned.
The holes in the bracket are off-
set to one side.
The holes must be closer to the
top of the shelf.

Installation
Mellanox Technologies18
Step 5. Move the shelf brackets on the shelf. Place the shelf bracket so that it measures
slightly more than the measured distance between the vertical rack supports. Make
sure that there is enough play in the bracket so that it can move to the vertical support.
Do not tighten the hex head bolts.
Figure 7: Placing the Shelf Bracket
Figure 8: Caged Nut Locations for the Shelf Brackets
You may need to remove the brackets and reinstall them.
Adjusting slots

648 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.4
Mellanox Technologies 19
Step 6. Insert five nuts into the rack for each of the four shelf brackets (the 2 brackets
that are adjustable and the 2 that are fixed). Make sure that the caged nuts are
all at the same height on the vertical supports. Do not use the top hole in the
bracket.
Step 7. Put the Shelf into the rack and twist the shelf into place. Make sure the spine
side of the shelf is closest to the rack doors. See Figure 9 to identify the spine
side of the shelf.
Step 8. Install and tighten the 20 M6 bolts into the caged nuts.
Figure 9: Shelf Brackets
Step 9. Tighten the 16 hex head bolts on the adjustable brackets.
The spine side of
the shelf has two
slots for the
extension
Adjustable shelf
brackets

Installation
Mellanox Technologies20
Figure 10: U Numbering Above Shelf
Step 10. Place 14 caged nuts into the vertical rack support at the locations corresponding to the
holes in the chassis faceplate. The holes in the faceplate start at 2U above the shelf and
are spaced every 4U centered as you go up the vertical rack support. There are 7 holes
on each side.
1.6.3.6 Inserting the Chassis
Step 1. Put the container on the fork lift.
The chassis is on ball bearings and can roll easily. Be aware that the chassis can roll off
of the pallet. Should the chassis start to fall, move out of the way. The chassis can
cause grave bodily harm should it fall on or near you.
1
Top of
Shelf
1
2
3
2
7.1 cm
The first cage nut on the spine side goes
here. The rest of the caged nuts go every
4 U units up from here.