Mellanox Technologies MSX6518-6R Manual

www.mellanox.com
324 Port InfiniBand FDR Switch Platform
Hardware Installation Guide
PN: MSX6518-6R, MSX6518-NR
Rev 1.0

Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
Beit Mellanox
PO Box 586 Yokneam 20692
Israel
Tel: [phone] ; 
[phone]
Fax: [phone]
© Copyright 2012.
Mellanox, Mellanox logo, BridgeX, ConnectX, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, PhyX, SwitchX, Virtual Protocol
Interconnect and Voltaire are registered trademarks of Mellanox Technologies, Ltd. FabricIT, MLNX-OS, Unbreakable-Link, UFM and
Unified Fabric Manager are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
324 Port InfiniBand FDR Switch Platform Hardware Installation Guide
Document Number: 3805
Rev 1.0
Mellanox Technologies2
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE
PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE
OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE
CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX
TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX
TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE
DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,
PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND
RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0
Mellanox Technologies 3
Table of Contents
Chapter 1 Chassis Installation 10
1.1 Environmental and Safety Recommendations 10
1.2 ChassisPackageContents 11
1.3 LeafPackageContents 12
1.4 SpinePackageContents 12
1.5 ManagementPackageContents 12
1.6 PhysicalInstallation 12
1.6.1 Starting with the Rack 12
1.6.2 ESD (ElectroStatic Discharge) Connection 13
1.6.3 Installation Procedure 13
1.7 Installing the Leafs 30
1.8 Inserting a Management Module 31
1.9 Inserting a Spine Board 31
Chapter 2 Installing the Cable Holder 33
Chapter 3 Ground Connections 34
Chapter 4 Power Connections 35
4.1 Powering Up the Switch Platform 35
Chapter 5 InfiniBand QSFP Cable Installation 37
5.1 PortConnectorInterfaces 37
5.2 SupportedApprovedCables 38
5.3 CablePower Classes 38
Chapter 6 Chassis Power Up 39
6.1 Power Supply, Management Board, and Spine Board Indicator Status at Power ON 40
Chapter 7 InfiniBand Fabric Initialization and Management 41
7.1 Configuring the Switch Management Modules for the First Time 41
7.2 RerunningtheWizard 46
Chapter 8 MLNX-OS Management Software 47
8.1 Starting an SSH Connection to the Switch 47
8.2 Starting a WebUI Connection to the Switch 47
Chapter 9 Resetting the Switch – RST 49
Chapter 10 TroubleShooting 50
10.1 PowerSupplyUnit 50
10.2 LeafBoard 50
10.3 ManagementModule 51
10.3.1 Yellow Status LED (for the Chassis) on the Management Module is Lit 51
10.3.2 Yellow LED for the Leaf Fan on the Management Module is Lit 51
10.3.3 Yellow LED for the Spine Fan on the Management Module is Lit 52
10.4 SpineBoard 52
10.5 MLNX-OSSoftware 53

Mellanox Technologies4
Rev 1.0
List of Figures
Figure 1: Distance Between the Vertical Supports 13
Figure 2: Installation Kit Parts 15
Figure 3: Shock and Tilt Stickers 16
Figure 4: Container Screws 17
Figure 5: Vertical Rack Supports Dimension 18
Figure 6: Adjustable Shelf Brackets 19
Figure 7: Top of Shelf Bracket 19
Figure 8: Placing the Shelf Bracket 20
Figure 9: Caged Nut Locations for the Shelf Brackets 20
Figure 10: Shelf Brackets 21
Figure 11: U Numbering Above Shelf 21
Figure 12: Putting the Chassis in the Rack 22
Figure 13: Place the Upper Bracket Offset 23
Figure 14: Attach the Shelf Extension 23
Figure 15: Location for the Lock-down Bar 24
Figure 16: Caged Nut Placement by U Numbers 25
Figure 17: Fork Lift 26
Figure 18: Face Plate Mounting Bolt Locations 26
Figure 19: Upper Bracket Installation 27
Figure 20: Upper Brackets and Offset Location 28
Figure 21: Offset to Upper bracket Bolts 28
Figure 22: Location for the Lock-down Bar 29
Figure 23: Lock-Down Bar 29
Figure 24: Bolt and Washer Order 30
Figure 25: Ejector Latch 30
Figure 26: Ejector Latch 31
Figure 27: Spine Board Insertion 31
Figure 28: Ejector Latch 32
Figure 29: Cable Holder 33
Figure 30: Ground Connection 34
Figure 31: Multiple Power Inlets - Electric Caution Notification 36
Figure 32: Spine Module 36
Figure 33: Port Numbering 38
Figure 34: Top and Bottom Ports 38
Figure 35: Spine Side Panel Display Status Indications 40
Figure 36: Management Module Status Indications for Normal Operation 40
Figure 37: Connecting the Harness to the Console 41

324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0
Mellanox Technologies 5
Figure 38: Web UI Login Page 48
Figure 39: Reset Button 49

Mellanox Technologies6
Rev 1.0
List of Tables
Table 1: Revision History of this Installation Guide 7
Table 2: Reference Documents and Web Sites 8
Table 3: Serial Terminal Program Configuration 42
Table 4: Configuration Wizard Session - IP Configuration by DHCP 42
Table 5: Configuration Wizard Session - IP Zeroconf Configuration 44
Table 6: Configuration Wizard Session - Static IP Configuration 45

324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0
Mellanox Technologies 7
Revision History
Table 1 - Revision History of this Installation Guide
Revision Date Details
1.0 March 2012 Initial release

Mellanox Technologies8
Rev 1.0
About this Manual
This manual provides installation and set-up instructions for the SX6518 QSFP Chassis InfiniBand
Switch Platform.
Intended Audience
This manual is intended for users and system administrators responsible for installing and setting
up the switch platform.
The manual assumes familiarity with the InfiniBand®architecture specification.
Related Documentation
The documentation set accompanying the QSFP Chassis InfiniBand Switch platform includes the
following:
Table 2 - Reference Documents and Web Sites
Document Conventions
Throughout this manual, the name SX6518 and the terms chassis and switch are used to describe
the switch, unless explicitly indicated otherwise.
The following pictures are used throughout this document to indicate information that is important
to the user.
InfiniBand Architecture Specification, Vol. 1,
Release 1.2.1 The InfiniBand Architecture Specification that is provided by IBTA
Switch Hardware Errata For any possible errata due to hardware issues see the switch
support product page. This requires a customer support login.
Mellanox MLNX-OS™ SwitchX® Software WebUI
User’s Manual WebUI Overview for MLNX-OS software.
Mellanox MLNX-OS™ SwitchX® Software User
Manual This document contains information regarding configuring and managing Mellanox Technologies' SwitchX Switch Platforms.
MLNX-OS™ Software Command Reference Guide Command Reference Guide for MLNX-OS listing all of the
commands available through MLNX-OS with explanations and
examples.
MLNX-OS™ Software Configuration Guide Configuration Guide for MLNX-OS displaying different config-
uration scenarios.
This symbol makes recommendations to the user.

324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0
Mellanox Technologies 9
This symbol indicates information that is helpful to the user.
This symbol indicates a situation that can potentially cause damage to hardware or
software.
WARNING! This symbol indicates a situation that can potentially cause personal
injury and / or damage to hardware or software.

Chassis Installation
Mellanox Technologies10
Rev 1.0
1 Chassis Installation
Installation and initialization of the chassis is a simple process requiring attention to the normal
mechanical, power, and thermal precautions for rack-mounted equipment. Your chassis comes
only with the power supplies and fans pre-installed. The rest of the openings are populated with
blanks. All of the leafs, spines, and management modules come shipped in separate packages.
The chassis requires initial configuration to get the chassis and Fabric management up and running
through remote management. see “Configuring the Switch Management Modules for the First
Time” on page 41.
1.1 Environmental and Safety Recommendations
The following are Mellanox recommendations.
This chassis can be installed in standard 19” racks that have depths between 65cm
(25.6in.) and 80cm (31.5in.) between the vertical supports of the rack.
This unit is intended for installation in a Restricted Access Location. A restricted
access area can be accessed only through the use of a special tool, lock and key, or
other means of security.
Recommended ambient temperature in the System room is 20o± 5oC.
Recommended humidity ranges is 40% ± 15% without condensing.
It is highly recommended that the installation sites be as isolated as possible from all
sources of radio transmissions and electrical interference.
It is highly recommended that the installation site building be equipped with a light-
ning rod.
It is highly recommended that the installation site be equipped with smoke detectors
and a fire alarm warning system.

324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0
Mellanox Technologies 11
1.2 Chassis Package Contents
The chassis package includes:
• 1 chassis with the following modules installed:
4 fans
6 PSUs
• 1 installation kit
• 1 box containing 6 power cords 250v 15a 2.0M, C14 to C13, USA UL Standard
• 1 cable management kit
• 1 Installation Guide
Before you install your new SX6518 chassis, unpack the system and check to make sure
that all the parts have been sent, check this against the parts list see “Installation and Cable
Management Kit Parts” on page 14. Check the container for visible damages that may
have occurred during shipping.
The system requires a KVA rated UPS system. It is recommended that a UPS system
be installed to protect the equipment in the event of unexpected power failure.
Make sure that the outlets and circuits will not be overloaded. Spread out the load over
at least two or three circuits or use a 3 phase circuit. N+N grid redundancy requires 2
independent power supply sources.
The leafs, spines, and management modules are shipped separately.
If anything is damaged or missing, contact your customer representative immediately.
The rack mounting holes conform to the EIA-310 standard for 19-inch racks. Guaran-
tee proper ventilation, by leaving 8cm (3”) of space to the front and rear of the switch.
This will ensure proper air flow through the chassis. This is crucial for maintaining
good airflow at ambient temperature. In particular, route cables such that they do not
impede the air into or out of the chassis.

Chassis Installation
Mellanox Technologies12
Rev 1.0
1.3 Leaf Package Contents
The leafs are ordered by the customer and are shipped 4 to a box. The customer will receive as
many boxes as needed to fill the order.
1.4 Spine Package Contents
The spines are shipped 3 to a box. The customer will receive as many boxes as needed to complete
a full complement of spines.
1.5 Management Package Contents
The package includes:
• all of the management modules ordered by the customer
• 1 RJ45 to DB9 harness for each management module received
1.6 Physical Installation
The switch platform uses 17U of rack space in a standard 19” rack, 16U for the chassis and 1U for
the shelf. The switch ships from the factory with mounting holes on the spine side only. There are
upper brackets to connect the leaf side to the rack near the top of the chassis, and there are two
lock-down bars to secure the chassis to the shelf. The weight of the switch is supported from
underneath the unit by the shelf.
This chassis can be installed in standard 19” racks that have between 65cm (25.6in.) and 80cm
(31.5in.) between the vertical supports of the rack. Make sure that a fully populated rack including
cables will have sufficient air flow for cooling.
1.6.1 Starting with the Rack
The rack may need to be modified to accommodate the SX6518 chassis. The distance between the
vertical supports must be between 65 and 80 cm.
Warning: This equipment is very heavy. Safety is the first concern. Make sure that
proper manpower and equipment are used for transporting and moving the chassis.
The fully loaded chassis weighs:
212.36 kg (468 LBS) full configuration
90 kg (199 LBS) empty configuration
112 kg (247LBS) As shipped configuration
Choose a rack which is able to support the mechanical and environmental characteris-
tics of a fully populated switch chassis.

324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0
Mellanox Technologies 13
Figure 1: Distance Between the Vertical Supports
The vertical supports may have to be moved so that the vertical support closest to the door
is 6 cm from the inside of the door. This may be required so that the shelf does not pro-
trude from the rack.
1.6.2 ESD (ElectroStatic Discharge) Connection
Before starting any procedure on the SX6518 switch system do the following.
Step 1. Put an ESD prevention wrist strap on your wrist, and make sure there is good
contact between your body and the strap.
Step 2. Plug the other end of the wrist strap to a valid ground. Make sure that this is a
tight fit.
1.6.3 Installation Procedure
If you move the vertical supports make sure that you reinstall them vertically
plumb.
Installing the chassis will be a lot easier if you can remove the sides of the rack.
65-80cm
This edge of the shelf
must be as close as
possible to the inside of
the door when closed.
Vertical Supports

Chassis Installation
Mellanox Technologies14
Rev 1.0
1.6.3.1 Requirements
You will need to have the following tools and equipment:
1.6.3.2 Installation and Cable Management Kit Parts
• Parts included in the installation kit:
You will need a fork lift, to move and insert the chassis into the rack.
Use enough people to safely lift this product.
• #2 phillips screwdriver • #3 phillips screwdriver
• a fork lift • a grounding lug
• measuring tape • ground wire of sufficient length and gauge to properly ground the
chassis
The installation will be much easier with a power screwdriver.
It is recommended to use AWG6 or 4mm diameter wire for grounding purposes.
It is recommended to have at least two people for the duration of the installation proce-
dure. Use a mechanical lift to raise this chassis. If not, use enough manpower to ensure
the safety and wellbeing of all of the people involved in the installation.
- 1 shelf with brackets pre-installed - 1 shelf extension
- 1 left side upper bracket offset - 1 right side upper bracket offset
- 2 lock down bars - 2 upper bracket washers
- 8 lock washers for the upper brackets to
vertical support - 12 lock washers for the upper brackets to upper bracket offsets
- hex head bolts – 8 for the upper 
brackets and for the shelf extension - 20 caged nuts 12 for the shelf and 8 for the chassis faceplate
<40 lbs
<18 kgs 40 - 70 lbs
18 - 32 kgs 70 - 121 lbs
32 - 55 kgs >121 lbs >55 kgs

324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0
Mellanox Technologies 15
• Parts included in the cable management installation kit:
Figure 2: Installation Kit Parts
- 20 pan head screws for the caged nuts 12
for the shelf and 8 for the chassis faceplate - 20 flat washers– 8 for the lock down bars and 12 for the upper brackets
- 32 lock washers – 20 for the lock down
bars and 12 for the upper brackets to
upper bracket offsets
- 32 hex head bolts – 20 for the lock down
bars and 12 for the upper brackets to upper
bracket offsets
- 2 upper brackets - 12 flat head screws with locktite for the
upper bracket offsets
- 1 open end ratchet wrench 10mm - 1 socket nut driver 8mm
- 1 socket wrench - 1 socket nut driver 10mm
- 2 extensions for the socket driver - 1 socket nut driver 13mm
- flat washers for the shelf extension - 1 box wrench
- 1 cable management rack RH - 1 cable management rack LH
- 10 caged nuts M6 - 10 M6 bolts
Shelf
Caged nuts
Upper bracket
Lock washersPhillips pan head 
screw
Upper bracket offset
Shelf extension
Lock-down bar
Flat washers hex head bolt
Upper bracket
washer
Flat head screw

Chassis Installation
Mellanox Technologies16
Rev 1.0
1.6.3.3 Container Mishandling
The container has shock and tilt stickers applied. These will turn red if the container has been mis-
handled or roughly handled. Upon receipt of the container look for and inspect the shock and tilt
stickers to confirm that they have not tripped. If one or more are red notify the shipper and Mella-
nox. This on its own does not indicate damage to the contents. But, be sure to carefully inspect the
contents if any of the shock and tilt stickers have tripped.
Figure 3: Shock and Tilt Stickers
1.6.3.4 Opening the Container
Step 1. Check the shock and tilt stickers on the container to see if it was subjected to excessive
shock or tilting. If any of these stickers are tripped contact your Mellanox representa-
tive.
Step 2. Before starting the procedure, put the ESD strap on and connect it to a valid ground.
HANDLEWITHCAREHANDLEWITHCARE
GGIINNNNRRWWAAGG
II
NNNNRRWWAA
HANDLEWITHCAREHANDLEWITHCARE
GG
II
NNNNRRWWAAGG
II
NNNNRRWWAA
TI L TWA
T
CH
TIL TWA
T
CH
Stickers unaffected by shipping
Stickers showing excessive
shock or tilt.

324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0
Mellanox Technologies 17
Figure 4: Container Screws
Step 3. Remove the four sides of the container by unscrewing the phillips head screws
holding the sides on.
Step 4. Remove and put aside the box containing the cables.
Step 5. Remove and put aside the box containing the cable management kit.
Step 6. Remove and put aside the box containing the Installation kit.
Step 7. Visually inspect the chassis, make sure that:
there is no visible damage
6 PSUs are installed
all 4 fans are installed
Step 8. Remove all protective plastic film from all sides and top of the chassis.
1.6.3.5 Installing the Shelf
Step 1. Select the location in the rack. Try to keep the chassis as low as possible to
keep the center of gravity low.
The leafs, spines, and management modules are shipped separately.
Do not remove any of the blocks or either of the lock-down bars at this time.
Container screws

Chassis Installation
Mellanox Technologies18
Rev 1.0
The maximum height that the chassis can be mounted, will be determined by the maximum height
that the fork lift can raise the pallet.
Step 2. Open the Installation kit and make sure that all of the parts are in the box. See
Figure 2,“Installation Kit Parts”.
Installing the chassis in the rack with the spine side as close as possible to the door leaves maxi-
mum room for the cable management system.
Figure 5: Vertical Rack Supports Dimension
It is recommended to set the top of the shelf at U5. Try and keep the chassis as low as
possible.
The distance between the vertical supports must be between 65 cm (25.6”) and 80 cm
(31.5”). If the distance in your rack is different, move the vertical supports so that one
pair is 10 cm(4in.) away from the door and that the vertical supports are between 65
cm (25.6”) and 80 cm (31.5”) apart.
The side of the chassis with the spine units will sit flush with the vertical supports of
the rack. The side of the chassis with the QSFP connectors will sit at the edge of the
shelf closer to the center of the rack.
65-80cm
Vertical Supports

324 Port InfiniBand FDR Switch Platform Hardware Installation Guide Rev 1.0
Mellanox Technologies 19
Step 3. Measure the distance between the vertical supports and adjust the shelf brack-
ets according to this distance. This chassis can be installed in a rack with a
distance between the vertical supports of from 65cm (25.6in.) to 80cm
(31.5in.).
Step 4. The shelf comes with the four brackets pre-installed. Two of the brackets
should be loose. They may need to be removed and repositioned to meet the
distance between the vertical supports of the rack.
Figure 6: Adjustable Shelf Brackets
Figure 7: Top of Shelf Bracket
These two brackets will
need to be adjusted. They
may even need to be
removed and repositioned.
The holes in the bracket are
offset to one side.
The holes must be closer to
the top of the shelf.
The holes in the bracket are
offset to one side.
The holes must be closer to
the top of the shelf.
Top of shelf
Bottom of shelf is even with this side of the
bracket, the greater distance between the holes
and the edge of the bracket show that this is the
bottom side.

Chassis Installation
Mellanox Technologies20
Rev 1.0
Figure 8: Placing the Shelf Bracket
Figure 9: Caged Nut Locations for the Shelf Brackets
Step 5. Insert three caged nuts into the rack for each of the four shelf brackets (the 2 brackets
that are adjustable and the 2 that are fixed). Make sure that the caged nuts are all at the
same height on the vertical supports.
Step 6. Put the Shelf into the rack and twist the shelf into place. Make sure the spine side of
the shelf is closest to the rack doors. See Figure 10 to identify the spine side of the
shelf.
Step 7. Install and tighten the 12 M6 bolts into the caged nuts.
Thisdistanceshouldbetheslightly
largerthanthemeasureddistance
betweentherackverticalsupports.
Adjusting slots