Mellanox Technologies SwitchX-2 MSX6005F-1BFS Instructions for use

www.mellanox.com
SwitchX®-2 12 Port InfiniBand Switch System
Hardware User Manual
P/N: MSX6005F-1BFS, MSX6005T-1BFS, MSX6012F-1BFS, MSX6012F-2BFS, MSX6012T-1BFS,
MSX6012T-2BFS, MSX6005F-1BRS, MSX6005T-1BRS, MSX6012F-1BRS, MSX6012T-1BRS
Rev 1.0

Document Number: MLNX-15-725
Mellanox Technologies
2
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
Beit Mellanox
PO Box 586 Yokneam 20692
Israel
www.mellanox.com
Tel: [phone]
Fax: [phone]
© Copyright 2013. Mellanox Technologies. All Rights Reserved.
Mellanox®, Mellanox logo, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®,
MLNX-OS®, PhyX®, SwitchX®, UFM®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of
Mellanox Technologies, Ltd.
Connect-IB™, FabricIT™, Mellanox Open Ethernet™, Mellanox Virtual Modular Switch™, MetroX™, MetroDX™,
ScalableHPC™, Unbreakable-Link™ are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE
THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT
HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE
PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND
DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST
QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES
FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND
(INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED
DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

SwitchX®-2 12 Port InfiniBand Switch System Hardware User Manual Rev 1.0
Mellanox Technologies 3
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
Mellanox Part Numbering Legend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
Chapter 1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.2 Serial Number and Product Version Information . . . . . . . . . . . . . . . . . . . . . . . . 14
1.3 FDR Transmission Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4 Internally Managed vs. Externally Managed. . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Chapter 2 Basic Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.1 Switch Platform Hardware Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 Port Connector LEDs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4 Air Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.5 QSFP Cable Power Budget Classification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.6 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Chapter 3 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.1 Installation Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2 Package Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3 Mechanical Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.4 Grounding the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.5 Power Connections and Initial Power On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.6 Externally Managed Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.7 Internally Managed Switch Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Chapter 4 Cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.1 Cable Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.2 Splitting the Data Stream Using Breakout (Fanout) Cables . . . . . . . . . . . . . . . . 48
Chapter 5 Switch Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.1 InfiniBand Subnet Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.2 Fabric Inspector (Diagnostics). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.3 Upgrading Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.4 Updating Firmware Internally Managed Switch . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.5 Updating Firmware Externally Managed Switch . . . . . . . . . . . . . . . . . . . . . . . . 51

Rev 1.0
Mellanox Technologies
4
Chapter 6 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Appendix A Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
A.1 Approved Cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
A.2 EMC Certifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
A.3 China CCC Warning Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Appendix B Thermal Threshold Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Appendix C QSFP Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Appendix D RJ-45 CONSOLE and I2C Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Appendix E Accessory and Replacement Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Appendix F Disassembly of the Switch from the Rack . . . . . . . . . . . . . . . . . . . . . . . . . 63
F.1 Disassembly Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
F.2 Disposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Appendix G Avertissements de sécurité pour l'installation (French) . . . . . . . . . . . . . . 64
Appendix H Installation – Sicherheitshinweise(German) . . . . . . . . . . . . . . . . . . . . . . . 69
Appendix I Advertencias de seguridad de instalación(Spanish) . . . . . . . . . . . . . . . . . 73
Appendix J Предупреждения по технике безопасности при установке(Russian) 78
Appendix K Avertismente privind siguranţa la instalare (Romanian) . . . . . . . . . . . . 83
Appendix L 安裝安全性警告 (Chinese) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

SwitchX®-2 12 Port InfiniBand Switch System Hardware User Manual Rev 1.0
Mellanox Technologies 5
List of Figures
Figure 1: Internally Managed Switch SX6012 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Figure 2: Externally Managed Switch SX6005 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Figure 3: Connector Side View of the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Figure 4: Pull out tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Figure 5: Switch System Connector Side Panels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
Figure 6: Externally Managed System Status LEDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Figure 7: Switch Status LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Figure 8: Fan Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19
Figure 9: Power Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Figure 10: Bad Port LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21
Figure 11: Identifier LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Figure 12: Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Figure 13: Front Side Interfaces for the Internally Managed Switch . . . . . . . . . . . . . . . . . . . . . . . . .24
Figure 14: Interface for Externally Managed Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Figure 15: Rear of Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26
Figure 16: Reset Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
Figure 17: Port Numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
Figure 18: Top and Bottom QSFP Port Orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
Figure 19: Placing the Bumpers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33
Figure 20: Installation Kit Parts for a Side by Side Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . .34
Figure 21: Screw on the Switch Mounted Rails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35
Figure 22: Two Switch Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36
Figure 23: Placement of Frame in the Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36
Figure 24: Placing the Spacer in the Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Figure 25: Using the Spacer Bushings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37
Figure 26: Insert Switch into Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38
Figure 27: Status LEDs 5 Minutes After Power On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39
Figure 28: Two Power Inlets - Electric Caution Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40
Figure 29: Console Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41
Figure 30: Top and Bottom QSFP Cable Orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48
Figure 31: Breakout or Fanout Cable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49
Figure 32: QSFP Connector Male and Female Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60

Rev 1.0
Mellanox Technologies
6
List of Tables
Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
Table 2: Reference Documents and Web Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Table 3: General System Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
Table 4: Management Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
Table 5: Switch Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
Table 6: System Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Table 7: Switch Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Table 8: Fan Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19
Table 9: PS Unit Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20
Table 10: Bad Port LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Table 11: Port Connector Physical and Logical Link Assignments for Ethernet Mode . . . . . . . . . .22
Table 12: Connector Physical and Logical Link Assignments for IB Mode . . . . . . . . . . . . . . . . . .23
Table 13: Serial Terminal Program Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42
Table 14: Configuration Wizard Session - DHCP Configuration (Example) . . . . . . . . . . . . . . . . .42
Table 15: Configuration Wizard Session - Static IP Configuration . . . . . . . . . . . . . . . . . . . . . . . . .44
Table 16: Configuration Wizard Session - Zeroconf Configuration . . . . . . . . . . . . . . . . . . . . . . . .45
Table 17: SX6012/6005 Specification Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Table 18: RJ-45 CONSOLE Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61
Table 19: RJ-45 I2C Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Table 20: OPNs for Replacement Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62

SwitchX®-2 12 Port InfiniBand Switch System Hardware User Manual Rev 1.0
Mellanox Technologies 7
Revision History
Table 1 - Revision History Table
Date Revision Description
July 2013 1.0 Initial Draft

Rev 1.0
Mellanox Technologies
8
About this Manual
This manual describes the installation and basic use of the Mellanox MSX6012/6005 switch
which is based on the SwitchX® -2 switch device.
Intended Audience
This manual is intended for users and system administrators responsible forwitch platforms.
The manual assumes familiarity with the InfiniBand® Architecture Specificationand with the
Ethernet specification.
Related Documentation
Additional Documentation available from Mellanox:
All of these documents can be found on the Mellanox Website at www.mellanox.com. They are
available either through the product pages www.mellanox.com Products InfiniBand/VPI
Switch Systems or through the support page with a login and password.
Conventions
Throughout this manual, the name SX6012/6005 and the term switch are used to describe the 12-
port 56Gb/s IB switch unless explicitly indicated otherwise.
The following icons are used throughout this document to indicate information that is important
to the user.
Table 2 - Reference Documents and Web Sites
Document Name Description
InfiniBand Architecture Specifica-
tion, Volume 1, Release 1.2.1 The InfiniBand Architecture Specification that is provided by IBTA
SwitchX®-2 Switch Product Hardware
Release Notes For possible hardware issues see the switch support product page. This
document can be found on the support web page for this product.
This requires a customer support login. Look up the relevant SwitchX®-
based switch system/series release note file.
Mellanox MLNX-OS® Software User
Manual This document contains information regarding configuring and manag-
ing Mellanox Technologies Switch Platforms.
MLNX-OS® Software Command Ref-
erence Guide Command Reference Guide for MLNX-OS® listing all of the commands
available through MLNX-OS® with explanations and examples.
This icon makes recommendations to the user.

SwitchX®-2 12 Port InfiniBand Switch System Hardware User Manual Rev 1.0
Mellanox Technologies 9
Abbreviations
FDR — Fourteen Data Rate – Used to indicate a 4X port running at 56 Gb/s.
This icon indicates information that is helpful to the user.
This icon indicates a situation that can potentially cause damage to hardware or soft-
ware.
BEWARE! This icon indicates a situation that can potentially cause personal injury or
damage to hardware or software.

Rev 1.0
Mellanox Technologies
10
Mellanox Part Numbering Legend
Place Field Decoder
MMellanox Technologies
SX System Type SwitchX® switch Family
PData Transfer Protocol 6 = InfiniBand
RSize of box 0 = 1U
1 = 1.5U
2 = 2U
FF Management Capabilities 05 = 12 Ports externally managed
12 = 12 Ports internally managed
15 = 18 Ports externally managed
18 = 18 Ports internally managed
25 = 36 Ports externally managed
36 = 36 Ports internally managed
CData Rate F = FDR, T = FDR10, Q = QDR, D = DDR
-Separator
P# Power Supplies 1=1, 2=2....
MDepth of the Unit S = standard depth, B = short depth
YAir Flow direction R= Connector side to Power side airflow 
F= Power Side side to Connector side airflow
RChip Generation R – SwitchX
S – SwitchX-2

SwitchX®-2 12 Port InfiniBand Switch System Hardware User Manual Rev 1.0
Mellanox Technologies 11
1 Overview
Mellanox SX6012 and SX6005 switch systems provide the highest performing fabric solution in
a 1U half-width form factor by delivering up to 1.3Tb/s of non-blocking bandwidth with 170ns
port-to-port latency. These switches are the industry's most cost-effective building blocks for
embedded systems and storage with the need for a low port density switch. Whether looking at
price-to-performance or energy-to-performance, the Mellanox 12 port FDR switches offer supe-
rior performance, power and space reducing capital and operating expenses, providing the best
return-on-investment.
The VPI switch running as an InfiniBand system switch is built with Mellanox’s 6th generation
switch device (SwitchX®-2 switch device), and provides up to 56Gb/s full bidirectional band-
width per port. These stand alone switches are an ideal choice for smaller departmental or back-
end clustering uses with high-performance needs, such as storage, data base and GPGPU clus-
ters.
The switch comes pre-installed with all necessary firmware and is configured for standard opera-
tion within an InfiniBand fabric. This switch requires an InfiniBand compliant Subnet Manager
running from one of the hosts or Fabric Management software running on the fabric. All that is
required for normal operation is to follow the usual precautions for installation and to connect the
switch to the adapter cards. Once an unmanaged switch is connected, the Subnet Management
software automatically configures and begins utilizing the switch. Managed switches need initial
configuration before they will start working.
It is recommended that the Mellanox OpenFabrics software package be installed on all nodes
connected to the switch. The software package provides a subnet manager and network manage-
ment tools as well as connectivity software for servers and storage, and is available on the Mella-
nox web site. See Section 5 on page 50 for more information.
Installation, hot-swapping components and hardware maintenance is covered in “Basic Opera-
tion” on page 16.
Figure 1: Internally Managed Switch SX6012
Figure 2: Externally Managed Switch SX6005
RST
PS1
UID
Mellanox
SX6012
PS2
12345678910 11 12
RST
PS1
UID
Mellanox
SX6005
PS2
I2C
12345678910 11 12

OverviewRev 1.0
Mellanox Technologies
12
Figure 3: Connector Side View of the Switch
1.1 Features
The switch includes the following features:
* License required.
** This feature will be available to customers in the near future.
Table 3 - General System Features
Feature Description
Hardware features Network Interfaces • 12 QSPF+ VPI interfaces
• Interface type –InfiniBand or Ethernet*
• InfiniBand speed: FDR (56Gb/s); FDR10/QDR (40Gb/s);
DDR (20Gb/s); SDR (10Gb/s)
• Ethernet* speed: 10/40/56 GbE
Switch options • InfiniBand – Inbox
• Ethernet*
• InfiniBand to Ethernet Gateway*
Compliance • Compliant with IBTA 1.21 and 1.3
• RoHS 6
Advanced Features • Congestion control**
• Adaptive routing**
• Port mirroring**
• Forward Error Correction -(FEC)
• Link Layer Retransmission -(LLR)
• Multiple SWIDs **
RST
PS1
UID
SX6012
12345678910 11 12
PS2

SwitchX®-2 12 Port InfiniBand Switch System Hardware User Manual Rev 1.0
Mellanox Technologies 13
Table 4 - Management Software
Feature Description
Management Software
(MLNX-OS®)
Software Management • Dual software image
• Software and firmware updates
File management •FTP
• TFTP
•SCP
Logging • Event history log
• SysLog support
Management Interface • DHCP/Zeroconf
•IPv6
Chassis Management • Monitoring environmental controls
• Power management
• Auto temperature control
• High availability
Network Management Inter-
faces
• SNMP v1,v2c,v3
• REST interfaces (XML Gateway)
Security •SSH
•Telnet
•RADIUS
• TACACS+
Date and Time •NTP
Cables & Transceivers • Transceiver info
Virtual Port Interconnect®
(VPI)
• Ethernet
• InfiniBand

OverviewRev 1.0
Mellanox Technologies
14
1.2 Serial Number and Product Version Information
The serial number and GUID for the switch are found on the pull out tab below the CONSOLE
connector. On managed switches only, the MAC for the Management PC is also placed on this
tab.
Figure 4: Pull out tab
1.3 FDR Transmission Rate
Mellanox switch systems support FDR (fourteen data rate), an InfiniBand data rate, where each
lane of a 4X port runs a bit rate of 14.0625 Gb/s with a 64b/66b encoding, resulting in an effec-
tive bandwidth of 56.25 Gb/s. The FDR physical layer is an IBTA specified physical layer using
advanced block types, deskew mechanism and framing rules for supporting higher speeds and
increased reliability.
The switches also supports FDR10, a non-standard InfiniBand data rate, where each lane of a 4X
port runs a bit rate of 10.3125 Gb/s with a 64b/66b encoding, resulting in an effective bandwidth
of 40 Gb/s.
FDR10 supports 20% more bandwidth over QDR using the same QSFP cables/connectors.
Both FDR and FDR10 support Forward Error Correction (FEC), as described in IEEE 802.3ap
chapter 74.
FDR and FDR10 are only guaranteed to work with approved Mellanox Cables.
FDR10 is only guaranteed to work with approved Mellanox ConnectX®-3 adapters.
Mellanox
SX6012
910 11 12
S/N:MT1117X00014
P/N:MSX6012F‐1BFS
GUID:0002C9020040DDB0
MadeinIsrael
Rev:A1
MGT1:
00:02:C9:11:A2:01
322X6519220002
Pull out tab

SwitchX®-2 12 Port InfiniBand Switch System Hardware User Manual Rev 1.0
Mellanox Technologies 15
1.4 Internally Managed vs. Externally Managed
All internally managed switches can support IB fabric of up to 648 nodes. Internally managed
switches need an initial configuration before they will start working.
The externally (unmanaged) managed switch system is designed as a low cost switch and as a
result does not have any field replaceable units. This switch has no Ethernet or console inter-
faces. The only management interface is an I2C interface on the connector side.
Externally Managed (Unmanaged) switches are plug and play out of the box. Updating the firm-
ware stack for the externally managed switches is done in-band only. You will need to confirm
that your switch has the latest firmware. For externally managed switches, confirm that the latest
FW revision is installed on this device. Use the CLI to get the currently installed revision and
then go to Mellanox.com to check the revision of the latest version.
The following table shows which switches come with a management CPU and which do not.
Externally managed switches can only run in an IB fabric. Externally managed
switches cannot run Ethernet protocol.
Table 5 - Switch Management
Family
Externally /
Internally
Managed
Management Connections
SX6005 Externally
Managed
(Unmanaged)
Plug and Play
All firmware updates should be done in-band using Mellanox firmware man-
agement tools. I2C port access using MTUSB-1 device is required for firm-
ware updates if in-band burning is not possible.
SX6012 Internally
Managed
RS232 cable DB9 to RJ-45 included in the box to connect to host PC for ini-
tial configuration of the switch. After initial configuration, the switch can be
managed through the Ethernet port using a remote connection.

Basic OperationRev 1.0
Mellanox Technologies
16
2Basic Operation
2.1 Switch Platform Hardware Overview
switchFigure 5 shows the connector side panel views of the internally managed switches. The
figure shows port configurations for the switch systems.Managed systems have:
• 1 – Ethernet RJ-45 connector for management
• 1 – RJ-45 connector for connecting to a host PC
• 1 – USB connector.
Externally managed switches only have a single I2C RJ-45 connector. All switches have various
status LEDs for on site status information.
Figure 5: Switch System Connector Side Panels
2.2 Status LEDs
The System Status LEDs are located to the left of the connectors on the connector side panel.
2.2.1 System Status LEDs
Figure 6: Externally Managed System Status LEDS
RST
PS1
UID
Mellanox
SX6012
PS2
12345678910 11 12
RST
PS1
UID
Mellanox
SX6005
PS2
I2C
12345678910 11 12

SwitchX®-2 12 Port InfiniBand Switch System Hardware User Manual Rev 1.0
Mellanox Technologies 17
Switch The system status LEDs should display as follows:
• when the switch is plugged in, within five minutes the switch status LED should light
up green
• the FAN LED should light up green. If the FAN LED shows red, replace the switch.
• the PS1 LED for the plugged in PS unit should light up green
• the PS2 LED for the second PS unit should light up green only if a second PS unit is
installed in the switch for redundancy and it is connected to a power source
• if two PS units are installed and only one PS unit is connected to a power supply the PS2
LED will be red
• if only one PS unit is installed in the switch, the PS2 LED will be off
If the PS LEDs are not green, this indicates a problem with the power supplies. The switch is
operational only if at least one of the PS LEDs is green.
Table 6 - System Status LEDs
Symbol Name Description Normal Conditions
Switch Status LED Shows the health of the switch Solid green after 5 minutes
Fan Status LED Shows the health of the fans Green
Power supply #1 Shows the health of the right side power supply unit Green
Power supply #2 Shows the health of the left side power supply unit Depends on switch
Bad Port LED lights up when a symbol error is detected Off
Unit Identifier LED Lights up on command through the CLI Off
As long as there is power to the switch (one PS unit is connected), and the switch is
booted up and running, the switch status LED should be green.
If the status LED shows red after five minutes, unplug the switch and call your Mella-
nox representative for assistance.
If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes
and replug in the switch. For more information See page 53.
UID
PS2
PS1

Basic OperationRev 1.0
Mellanox Technologies
18
2.2.2 Switch Status LED
Figure 7: Switch Status LED
2.2.3 Fan Status LED
Table 7 - Switch Status LED Assignments
LED Configuration STATUS/ System Health LED
Solid Green OK – The system is up and running.
Flashing Green The system is booting up. This assignment is valid on managed systems only.
Solid Yellow To be implemented in a future SW release.
Solid Red Major Error – Possible damage can result to the switch. Turn off immediately.
e.g. bad firmware, can’t boot, overheated. If the system is booting up, it can take
up to 5 minutes for the status led to change to green.
Off Off – The system has no power.
All fans must be operating while the power supply is plugged in.
If the switch shuts down due to over temperature, unplug the switch , wait 5 minutes
and replug in the switch . For more information See page 53.
RST
UID
PS2
PS1

SwitchX®-2 12 Port InfiniBand Switch System Hardware User Manual Rev 1.0
Mellanox Technologies 19
Figure 8: Fan Status LED Connector Side
Table 8 shows the fan status LED assignment.
Table 8 - Fan Status LED Assignments
LED Configuration FAN LED
Solid Green OK – All fans are up and running.
Solid Red Error – One or more fans is not operating properly. The system should be powered
down and troubleshoot the fan module. For the SX6005 replace the switch.
Off Off – The fan unit is not receiving any power. Check that the fan unit is properly
and completely inserted.
RST
UID
PS2
PS1

Basic OperationRev 1.0
Mellanox Technologies
20
2.2.4 Power Supply Status LEDs
Figure 9: Power Status LED Connector Side
This switch can come with two power supply units for redundancy.
The primary power supply (PS) unit is located on the left side and if there is a secondary PS unit
it will be on the right side.
Table 9 shows the power supply unit status LED assignment.
Table 9 - PS Unit Status LED Assignments
LED Color Status
Solid Green OK – The Power supply is delivering the correct voltage – 12VDC.
Solid Red Error – The PS unit is not operational.
Off Off – There is no power to the system (neither PS unit is receiving power). If one
PS unit is showing green and the second PS unit is unplugged it will show a red
indication.
RST
UID
PS2
PS1