Mellanox Technologies InfiniScale MTS3600Q-1BNC User manual

www.mellanox.com
InfiniScale®IV 36-Port QSFP
InfiniBand Switch User Manual
P/N: MTS3600Q-1BNC, MTS3600Q-2BNC, MTS3600Q-1UNC, MTS3600Q-2UNC, MTS3600R-1BNC,
MTS3600R-2BNC, MTS3600R-1UNC, MTS3600R-2UNC
Rev 2.4

Mellanox Technologies
350 Oakmead Parkway
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
PO Box 586 Hermon Building
Yokneam 20692
Israel
Tel: [phone]
Fax: [phone]
© Copyright 2009. Mellanox Technologies, Inc. All Rights Reserved.
Mellanox®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniRISC®, InfiniScale®, and InfiniPCI® are registered
trademarks of Mellanox Technologies, Ltd.
BridgeX and Virtual Protocol Interconnect are trademarks of Mellanox Technologies, Ltd.
All other marks and names mentioned herein may be trademarks of their respective companies.
ConnectX®EN PortEthernet Adapter Cards User’s Manual
Document Number: 2983
Rev 2.4
Mellanox Technologies
2
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE
PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE
OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE
CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX
TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX
TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE
DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,
PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND
RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Rev 2.4
Mellanox Technologies 3
Table of Contents
Revision History 7
About this Manual 8
Intended Audience 8
Related Documentation 8
Conventions 9
Switch Products Covered in this User’s Manual 9
Mellanox Part Numbering Legend 9
Chapter 1 Overview 10
1.1 Serial Number and Product Version Information 10
Chapter 2 Installation and Basic Operation 12
2.1 Switch Platform Hardware Overview 12
2.1.1 Status LEDs 12
2.1.2 I2C Connector 15
2.1.3 Internal Management 15
2.1.4 Interfaces 15
2.2 Switch Platform Installation and Operation 16
2.2.1 Installation Safety Warnings 16
2.2.2 Mechanical Installation 18
2.2.3 Grounding the Switch 21
2.2.4 Power Connections and Initial Power On 22
2.2.5 Extracting and Inserting the Power Supply Unit 23
2.2.6 InfiniBand Cable Installation 24
2.2.7 Extracting and Inserting the Fan Unit 25
2.3 Disassembly of the Switch from the Rack 25
2.4 Disposal 26
Chapter 3 Management and Tools Overview 27
3.1 Network Management and Clustering Software 27
3.2 Internally Managed Switch System 27
3.2.1 Initializing the Switch for the First Time 28
3.2.2 Accessing the Switch Locally 28
3.2.3 Starting a Remote Connection to the Switch 28
3.2.4 Downloading Firmware 29
3.3 Externally Managed Switch Updating Firmware 29
3.3.1 Instructions for Reprogramming Over the InfiniBand Network 29
3.3.2 Reprogramming the Switch Through The I2C Port 32
3.4 Externally Managed Switch System 34
3.4.1 How to Get Mellanox Firmware Tools (MFT) 34
3.4.2 Open SM 34
3.5 Updating Firmware 34
3.5.1 Instructions for Reprogramming Over the InfiniBand Network 34
Chapter 4 Switch Management Tools 39
4.1 Chassis Management 39
4.2 FabricIT Management 39
Chapter 5 Troubleshooting 40
Appendix A Specification 42
Appendix B Switch Hardware Components 46
Appendix C QSFP Interface 47
Appendix D Replacement Parts Ordering Numbers 48
Appendix E Avertissements de sécurité d’installation (French) 49

Rev 2.4
Mellanox Technologies
4
Appendix F Installation - Sicherheitshinweise (German) 51
Appendix G Special Regulations Regarding Finland, Sweden, Denmark, and Norway 53

Rev 2.4
Mellanox Technologies 5
Figure 1: Generic Product label 10
Figure 2: Management Module Label 11
Figure 3: QSFP Switch System Power and Connector Side Panels 12
Figure 4: Power, Fan, and System LEDs 13
Figure 5: MTS3600 QSFP Power Side Panel 14
Figure 6: Management Interfaces 15
Figure 7: Rack Installation Kit Parts 19
Figure 8: Which Side of the Rack Do You Want the Connectors? 20
Figure 9: Screwing the Rail Slide onto the switch 20
Figure 10: Screwing in the Rails 21
Figure 11: Two Power Inlets - Electric Caution Notifications 22
Figure 12: Power Supply Unit Extraction 23
Figure 13: Cable connections to the Switch 24
Figure 14: MTUSB-1 with Cables 33
Figure 15: I2C Cable Connected to MTS3600 33

Rev 2.4
Mellanox Technologies
6
List of Tables
Table 1: Revision History Table 7
Table 2: Reference Documents 8
Table 3: Switch Products Covered by this User Manual 9
Table 4: Part Numbering Legend 9
Table 5: Fans, System Status, LED Configurations 13
Table 6: Fan Status LED Configurations 14
Table 7: MTS3600Q Specification Data 42
Table 8: MTS3600R Specification Data 43
Table 9: Switch Certification Status 44
Table 10: InfiniBand QSFP Connector Pinout 47
Table 11: Replacement Parts Ordering Numbers 48

Rev 2.4
Mellanox Technologies 7
Revision History
Table 1 - Revision History Table
Date Revision Description
August 2009 Rev 2.4 Added that CONSOLE,Ethernet , and USB interfaces do not work in unmanages
switches
August 2009 Rev 2.3 Updated Figures 4 and 5. Updated interfaces to reflect CONSOLE interface.
August 2009 Rev 2.2 Updated references to initial start up connecting to a host PC.
August 2009 Rev 2.1 Removed procedure to replace the battery.
June 2009 Rev 2.0 Added Management module label and RS232 DB9 to DB9 harness
Added part numbers for the two management modules.
June 2009 Rev 1.9 Made fixes to management section
added Disassembly instructions
June 2009 Rev 1.8 Added Fabric IT section
April 2009 Rev 1.7 inserted procedure to replace the battery.
April 2009 Rev 1.6 Changes to installation kit and procedure
January 2009 Rev 1.5 added installation instructions
added ECM
Removed Watermark
November 2008 Rev 1.4 fixed weight in specs
October 2008 Rev 1.3 Minor Formatting changes.
September 2008 Rev 1.2 Added Reprogramming the MT3600 Through The I2C Port section.
Added power cord OPNs to table 11 in Appendix D.
September 2008 Rev 1.1 Fixed link to Software package page.
September 2008 Rev 1.0 Initial release

Rev 2.4
Mellanox Technologies
8
About this Manual
This manual describes the installation and basic use of the Mellanox MTS3600 switch, which is
based on the InfiniScale IV InfiniBand switch device.
Intended Audience
This manual is intended for users and system administrators responsible for installing and setting
up the switch platforms listed above.
The manual assumes familiarity with the InfiniBand®Architecture Specification.
Related Documentation
Additional Documentation available from Mellanox:
Table 2 - Reference Documents
Switch Firmware and
Firmware Update
Tools
See
http://www.mellanox.com > Downloads > Firmware Tools.
Note that the Switch System described in this manual is based on Mellanox
Technologies’ InfiniScale®IV switch device.
Mellanox OFED
Stack for Linux
User’s Manual
See
http://www.mellanox.com > Downloads > InfiniBand SW/Drivers,
Click “Mellanox OpenFabrics Enterprise Distribution for Linux
(MLNX_OFED)”
Select the Linux User’s Manual
The embedded OS and tools on the CPU in the management module is a
subset of the Mellanox OFED stack.
FabricIT Enterprise
Fabric Management
Software CLI User’s
Manual
Talk to your Mellanox representative for information regarding licensing
and implementation of the FabricIT Enterprise Fabric Management Soft-
ware System.
Mellanox Firmware
Tools (MFT)
User’s Manual
Document # 2329
The MFT (Mellanox Firmware Tools) package is a set of firmware tools.
The manual supplied with this package provides an overview of the firm-
ware its installation and replacement. The MFT can be downloaded with its
documentation at:
http://www.mellanox.com > Downloads > Firmware Tools.

Rev 2.4
Mellanox Technologies 9
Conventions
Throughout this manual, the name MTS3600 and the term switch are used to describe both the 36-
port QSFP 40Gb/s InfiniBand Switch and the 36-port QSFP 20Gb/s InfiniBand Switch, unless
explicitly indicated otherwise.
Switch Products Covered in this User’s Manual
Mellanox Part Numbering Legend
Table 3 - Switch Products Covered by this User Manual
Product Number Description
MTS3600Q-1BNC 36-port Managed QSFP 40Gb/s InfiniBand switch with one power supply
MTS3600R-1BNC 36-port Managed QSFP 20Gb/s InfiniBand switch with one power supply
Table 4 - Part Numbering Legend
Place Field Decoder
MT Mellanox Technologies
SSystem Type S = Switch
MM Model 36 = InfiniScale IV
FF Form factor 00 = Rack mounted switch, 10 = Chassis
CInfiniBand Port Config. G = CX4 DDR, J = CX4 QDR, Q = QSFP QDR, R = QSFP DDR,
-Separator
P# Power Supplies 0 = 0, 1 = 1, 2 = 2
MManagement U = Unmanaged, A = Managed (Celeron), B = Managed (Freescale)
YModule Config. N = IB modules
RRoHS C = RoHS w/ Exemption, R = RoHS Lead-Free

OverviewRev 2.4
Mellanox Technologies
10
1 Overview
Mellanox MTS3600M switch systems provide the highest performing fabric solution by deliver-
ing high bandwidth and low latency to Enterprise Data Centers, High-Performance Computing and
Embedded environments. Networks built with MTS3600 systems can carry converged traffic with
the combination of assured bandwidth and granular quality of service. Built with Mellanox’s 4th
generation InfiniScale® IV InfiniBand switch device, MTS3600 systems provide up to 40Gb/s
full bidirectional bandwidth per port. With 36 ports, these systems are among the densest switch-
ing systems available. These stand-alone switches are an ideal choice for top of-rack leaf connec-
tivity or for building small to medium size clusters.
The switch comes pre-installed with all necessary firmware and is configured for standard opera-
tion within an InfiniBand fabric. This switch requires an InfiniBand compliant Subnet Manager
running from one of the hosts or Fabric Management software running on the switch. All that is
required for normal operation is to follow the usual precautions for installation and to connect the
switch to the HCAs. Once connected, the Subnet Management software automatically configures
and begins utilizing the switch.
It is recommended that Mellanox OpenFabrics software package be installed on all nodes con-
nected to the MTS3600. The software package provides a subnet manager and network manage-
ment tools as well as connectivity software for servers and storage, and is available on the
Mellanox web site. See Chapter 3 for more information.
Basic installation, hot-swapping components and hardware maintenance is covered in “Installation
and Basic Operation” on page 12.
1.1 Serial Number and Product Version Information
The serial number and product version information are found on the label seen in the figure below.
Figure 1: Generic Product label
Also on this label is the GUID identifier for the switch. There is also a label with the MAC for the
management module.

Rev 2.4
Mellanox Technologies 11
Figure 2: Management Module Label

Installation and Basic OperationRev 2.4
Mellanox Technologies
12
2 Installation and Basic Operation
2.1 Switch Platform Hardware Overview
Figure 3 shows the power side panel and connector side panel views of the QSFP switch. The fig-
ure shows port configurations for the switch systems, including the dual hot-swap power supplies,
and hot-swap fan module, I2C connector, USB connectors, VGA port connector, and various sta-
tus LEDs.
Figure 3: QSFP Switch System Power and Connector Side Panels
All InfiniBand connectivity is via the connector side panel. All connectors can support active
cables.
2.1.1 Status LEDs
2.1.1.1 System Status Indicators
The System Status Indicators are located just to the left of the primary power supply unit, and
labeled “Sys.”.
The system status indicators should display as follows:
• When the switch is plugged in, within a few seconds the STATUS LED should light up green.
• The PSU 1 LED should light up green.
• The PSU 2 LED should light up green only if a second PSU is installed in the switch for redun-
dancy and Hot-Swap ability. If only one PSU is installed in the switch, the PSU 2 LED will be
off.
• The FAN LED should light up green.
If the STATUS LED shows red unplug the switch and call your Mellanox representative for assis-
tance.
If the FAN LED shows yellow, troubleshoot the fan module.
Note: Do not run the switch for more than 2 minutes should the FAN LED show yellow.
24 25 26 27 28
21
2019 22 23 29 30 32 33 34 35
System
Fans
PSU 1
PSU 2
31 MTS3600
InfiniBand
Switch
OK !
36
11 12 13 1487
654
3
21 910 1817 16
15
I2C
CONSOLE ~AC
OK
Power Side panel
Connector Side Panel

Rev 2.4
Mellanox Technologies 13
If the PSU LEDs are not green, this indicates a problem with the power supplies. Do not run the
switch if the PSU LEDs are not green.
Figure 4: Power, Fan, and System LEDs
Table 5 - Fans, System Status, LED Configurations
If the FAN LED shows red, shut down the switch and troubleshoot the fan
module outside of the switch.
LED Configuration Status
OK – The system is up and running.
Error –A fault in the system.
Off – The system has no power.
DC output
from
Latch release mechanism Handle
There is a
fault in the
Input AC volt-
age is good
PSU Status
Indicators
Switch Fan
Status
Indicators
System
Status
Indicators

Installation and Basic OperationRev 2.4
Mellanox Technologies
14
2.1.1.2 Power Supply Status Indicators
The MTS3600 36 Port Switch is available with one or two factory installed Power Supply Units.
For switches with only one unit installed, a second Power Supply Unit can be added to increase
security, hot-swap ability and to add redundancy. See Section D, “Replacement Parts Ordering
Numbers,” on page 48 for ordering part numbers.
Figure 5: MTS3600 QSFP Power Side Panel
The primary power supply unit (PSU1) is located on the right side of the power side panel, with
PSU2 on the left side.Status indicators are located just to the left of where the primary power sup-
ply unit is located. Each PSU also has three status LEDs on the right side of the PSU, that indicate
the internal status of the unit.
~AC– This LED when lit indicates input voltage between 90 and 264 Volts.
–This LED when lit indicates a fault in the power supply.
OK – This LED when lit indicates that the output from the power supply is +12VDC.
The PSU status indicators are also located just to the left of the primary power supply unit, and
labeled “PSU1” and “PSU2”. Table 2.1.1.3 Shows the explanation of the PSU Status LED colors.
2.1.1.3 Fan Status Indicators
The indicator labeled “Fans” is located just to the left of the primary power supply unit, next to
PSU 1. The following fan status conditions are possible:
Table 6 - Fan Status LED Configurations
LED Configuration Status
OK – All fans are operating.
Error – One or more fans is not operating. Repair
or replace the fan unit.
Off – The fan unit is not receiving any power.
Check that the fan unit is properly and com-
pletely inserted.
Primary Power
Supply Unit
Place for secondary
Power Supply Unit Fan Unit

Rev 2.4
Mellanox Technologies 15
2.1.2 I2C Connector
The MTS3600 supports an I2C compatible interface on the front panel. This interface is used to
program firmware into the Flash Memory. Firmware can also be programmed in-band over the
InfiniBand network. See Section 3 for more details.
2.1.3 Internal Management
Internal management refers to a switch that has an internal PPC management board. Switches
without this management board can be managed using an external management system. The fol-
lowing interfaces allow access to the internal management module. These interfaces have no func-
tion in externally managed systems.
2.1.4 Interfaces
There are three five interfaces to connect to the MTS3600. They are:
• 1 I2C DB9 interface is labelled “I2C”. This interface can be used to access and modify the Flash
and EEPROM.
• 1 Ethernet connector is labelled . Use this connector to connect to the Ethernet for remote
management. This connector does not work in unmanaged (externally managed) switches.
• 2 USB ports is labelled . This connector does not work in unmanaged (externally man-
aged) switches.
• 1 serial CONSOLE port. Use this to connect to a local host PC.This connector does not work in
unmanaged (externally managed) switches.
Figure 6: Management Interfaces
2.1.4.1 Serial CONSOLE Interface
The port labelled “CONSOLE” is for local host connection to the management module. This is
used the first time the switch is connected. Connecting to a local host PC and following the
All fans must be operating while the power supply is plugged in. Power must
be removed within two minutes of the fans failing, to prevent damage to the
switch.
System
Fans
PSU 1
PSU 2
OK !
I2C
CONSOLE ~AC
OK

Installation and Basic OperationRev 2.4
Mellanox Technologies
16
instruction in the Installation Guide, “Configuring the Switch for the First Time ”, must be done
before any remote management is available.
This connector does not work in unmanaged (externally managed) switches.
2.1.4.2 RJ-45 Ethernet Connector
The Ethernet connection provides access for remote management.
This connector does not work in unmanaged (externally managed) switches.
2.1.4.3 I2C Connector
The I2C connection provides access to Flash and EEPROMs.
2.2 Switch Platform Installation and Operation
Installation and initialization of the switch platform are straightforward processes, requiring atten-
tion to the normal mechanical, power, and thermal precautions for rack-mounted equipment.
The switch platform does not require any programming or configuration to operate as a basic
InfiniBand switch and includes all of the necessary functionality to operate with external standard
InfiniBand Subnet Management software.
An Internal Management option is available for the switch platform which includes InfiniBand
Subnet Management software embedded internally in the switch. This section describes the instal-
lation process and basic operation of the switch platform. Please first read the warnings sub-sec-
tion carefully before carrying on with installation procedures.
2.2.1 Installation Safety Warnings
For Safety Warnings in French see Section E,“Avertissements de sécurité d’installation (French),”
on page 49, and for German see Section F,“Installation - Sicherheitshinweise (German),” on page
51.
For Special Regulations Regarding Finland, Sweden, Denmark, and Norway see
Section G,“Special Regulations Regarding Finland, Sweden, Denmark, and Norway,” on page 53.
1. Installation Instructions
Read all installation instructions before connecting the equipment to the
power source.

Rev 2.4
Mellanox Technologies 17
2. Over-temperature
3. Stacking the Chassis
4. Redundant Power Supply Connection - Electrical Hazard
5. During Lightning - Electrical Hazard
6. Copper InfiniBand Cable Connecting/Disconnecting
7. Rack Mounting and Servicing
8. Equipment Installation
This equipment should not be operated in an area with an ambient tem-
perature exceeding the maximum recommended: 45°C (113°F). More-
over, to guarantee proper air flow, allow at least 8cm (3 inches) of
clearance around the ventilation openings.
The chassis should not be stacked on any other equipment. If the chassis
falls, it can cause bodily injury and equipment damage.
This product includes a blank cover over the space for the redundant
power supply. Do not operate the product if the blank cover is not
securely fastened or if it is removed.
During periods of lightning activity, do not work on the equipment or
connect or disconnect cables.
Copper InfiniBand cables are heavy and not flexible, as such they should
be carefully attached to or detached from the connectors. Refer to the
cable manufacturer for special warnings and instructions.
When this product is mounted or serviced in a rack, special precautions
must be taken to ensure that the system remains stable. In general you
should fill the rack with equipment starting from the bottom to the top.
This equipment should be installed, replaced, or serviced only by trained
and qualified personnel.

Installation and Basic OperationRev 2.4
Mellanox Technologies
18
9. Equipment Disposal
10. Local and National Electrical Codes
2.2.2 Mechanical Installation
The switch platform can be rack mounted and is designed for installation in a standard 19” rack.
The power side of the switch includes a hot-swap power supply module, a blank cover for an
optional second PSU for redundancy, a hot-swap fan tray, system LEDs, and management connec-
tion ports. There are two possible air flow directions. Be sure that the switch air flow direction is
compatible with your system and rack. The connector side of the switch has the QSFP ports.
The switch platform contains auto-sensing 90 - 264 VAC connections for all possible PSUs.
The installer should use a rack capable of supporting the mechanical and environmental character-
istics of a fully populated platform.
2.2.2.1 Minimum and Maximum Rack Depth for this Switch
This switch can be installed in any standard 19”racks with depths of 670mm to 900mm.
2.2.2.2 Installing the Switch in the Rack
Tools and Customer Supplied Parts
Parts included in the installation kit:
Disposal of this equipment should be in accordance to all national laws
and regulations.
This equipment should be installed in compliance with local and national
electrical codes.
The rack mounting holes conform to the EIA-310 standard for 19-inch racks.
Take precautions to guarantee proper ventilation in order to maintain good air-
flow at ambient temperature. Cable routing in particular should not impede the
air exhaust from the chassis.
• Phillips Screwdrivers #1 and #2 • Grounding screw
• ESD Strap • Grounding wire sufficient to reach a valid ground.
• ESD mat
• 2 rails • 8 recessed flat head screws
• 2 rail slides • 4 caged nuts

Rev 2.4
Mellanox Technologies 19
Figure 7: Rack Installation Kit Parts
Before you install your new MTS3600 switch, unpack the system and check to make sure that all
the parts have been sent, check this against the parts list. Check the parts for visible damages that
may have occurred during shipping.
Note: If anything is damaged or missing, contact your customer representative immediately.
Procedure
1. Place the ESD mat on the floor where you will be working and put on the ESD strap. Make sure
the ESD strap is touching your skin and that the other end is connected to a verified ground.
2. Choose which side of the switch you want even with the vertical rack support. Either the side
with the power supply units or the side with the IB connectors can be even with the vertical rack
support. The other side of the switch will be further inside of the rack.
Things to consider before choosing where to mount the rails and rail slides.
• 8 pan head screws M6 • 2 metal washers
rail
rail slide

Installation and Basic OperationRev 2.4
Mellanox Technologies
20
Figure 8: Which Side of the Rack Do You Want the Connectors?
• Do you want the connector side recessed in the rack to allow for larger cable bending radius?
• Will the connector side be recessed past other equipment in the rack and will this be problem-
atic?
3. Install the rail slides onto the switch. Place the end of the rail slide with the 90oangle on the side
of the switch that you want to be even with the vertical support of the 19” rack. Use four of the
flat head screws for each rail slide. There are two sets of holes in the rail slide. Select the set of
holes to either mount the switch closer or farther away from the rack vertical support. Tighten
the screws to 3Nm or 26.5 pound inches.
Figure 9: Screwing the Rail Slide onto the switch
The distance between the rack and the door can be as little as 4 cm on one side
of the rack and as much as 18 cm on the other side of the rack. Keep in mind that
there can be as many as 36 cables connected to the switch.
This side of the
switch will be
next to the ver-