Mellanox Technologies SwitchX-2 MSX6018F-1SFS Instructions for use

www.mellanox.com
SwitchX®-2 18-Port QSFP FDR InfiniBand
System Hardware User Manual
P/N: MSX6018F-1SFS, MSX6018T-1SFS, MSX6018F-1BRS, MSX6018T-1BRS, MSX6015F-1SFS,
MSX6015T-1SFS, MSX6015F-1BRS, MSX6015T-1BRS
Rev 1.0
Downloaded from Arrow.com.

Document Number: 4161
Rev 1.0
Mellanox Technologies
2
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
Beit Mellanox
PO Box 586 Yokneam 20692
Israel
www.mellanox.com
Tel: [phone]
Fax: [phone]
© Copyright 2013. Mellanox Technologies. All Rights Reserved.
Mellanox®, Mellanox logo, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®,
MLNX-OS®, PhyX®, SwitchX®, UFM®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of
Mellanox Technologies, Ltd.
Connect-IB™, FabricIT™, Mellanox Open Ethernet™, MetroX™, MetroDX™, ScalableHPC™, Unbreakable-Link™ are
trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE
THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT
HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE
PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND
DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST
QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES
FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND
(INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED
DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Downloaded from Arrow.com.Downloaded from Arrow.com.

SwitchX®-2 18-Port QSFP FDR InfiniBand System Hardware User Manual Rev 1.0
Mellanox Technologies 3
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
Mellanox Part Numbering Legend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
Chapter 1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2 Serial Number and Product Version Information . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3 FDR Transmission Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4 Internally Managed vs. Externally Managed. . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Chapter 2 Basic Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1 Switch Platform Hardware Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Air Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4 QSFP Cable Power Budget Classification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Chapter 3 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.1 Installation Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 Package Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.3 Grounding the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.4 Power Connections and Initial Power On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.5 Battery Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.6 Configuring the Switch the First Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.7 Externally Managed Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.8 Managed (Internally Managed) Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.9 Rerunning the Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.10 Connecting to the Switch Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.11 Starting an SSH Connection to the Switch (CLI) . . . . . . . . . . . . . . . . . . . . . . . . 38
Chapter 4 Extracting and Inserting FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.1 Extracting and Inserting the Power Supply Unit. . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2 Extracting and Inserting the Fan Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Chapter 5 Cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.1 SX6018 Splitting the Data Stream Using Breakout (Fanout) Cables . . . . . . . . . 43
5.2 Cable Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Rev 1.0
Mellanox Technologies
4
Chapter 6 Switch Shut Down Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Chapter 7 Disassembly of the Switch from the Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
7.1 Disposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Chapter 8 Management and Tools Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
8.1 Switch Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
8.2 Fabric Inspector (Diagnostics). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
8.3 Upgrading Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
8.4 Externally Managed Switch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Chapter 9 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Appendix A Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54
A.1 Approved Cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
A.2 EMC Certifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
A.3 China CCC Warning Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Appendix B QSFP Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60
Appendix C RJ-45 CONSOLE and I2C Interface . . . . . . . . . . . . . . . . . . . . . . . . .62
Appendix D Accessory and Replacement Parts . . . . . . . . . . . . . . . . . . . . . . . .63
Appendix E Avertissements de sécurité d’installation (French) . . . . . . . . . . .64
Appendix F Installation - Sicherheitshinweise (German) . . . . . . . . . . . . . . . . .68
Appendix G Advertencias de seguridad para la instalación (Spanish) . . . . . .71
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

SwitchX®-2 18-Port QSFP FDR InfiniBand System Hardware User Manual Rev 1.0
Mellanox Technologies 5
List of Figures
Figure 1: Connector Side View of the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
Figure 2: Pull out tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11
Figure 3: QSFP Switch System Power and Connector Side Panels . . . . . . . . . . . . . . . . . . . . . . . . .13
Figure 4: QSFP Switch System Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
Figure 5: Externally Managed Switch System Power and Connector Side Panels . . . . . . . . . . . . .14
Figure 6: System Status Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
Figure 7: System LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
Figure 8: Fan Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
Figure 9: Fan Status LED Power side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
Figure 10: Power Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Figure 11: Power Side Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Figure 12: PS Unit Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18
Figure 13: Bad Port LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18
Figure 14: Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19
Figure 15: Component Air Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21
Figure 16: Component Airflow Mismatch (Wrong) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21
Figure 17: Port Numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22
Figure 18: Interface for Internally Managed Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23
Figure 19: Interface for Externally Managed Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23
Figure 20: RJ-45 I2C Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25
Figure 21: Reset Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25
Figure 22: Installing the Power Cord Wire Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30
Figure 23: Two Power Inlets - Electric Caution Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31
Figure 24: Console Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33
Figure 25: Fan and PS unit Air Flow Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39
Figure 26: Power Supply Unit Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39
Figure 27: PS Unit Pulled Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40
Figure 28: Air Flow Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41
Figure 29: Fan Module Latches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41
Figure 30: Breakout or Fanout Cable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43
Figure 31: SX6018 Port Splitting Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43
Figure 32: QSFP Connector Male and Female Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Rev 1.0
Mellanox Technologies
6
List of Tables
Table 1 - Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
Table 2 - Reference Documents and Web Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Table 3 - Switch Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
Table 4 - System Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
Table 5 - Fan Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
Table 6 - PS Unit Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18
Table 7 - Bad Port LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19
Table 8 - Port Connector Physical and Logical Link Assignments for Ethernet . . . . . . . . . . .19
Table 9 - Connector Physical and Logical Link Assignments for IB . . . . . . . . . . . . . . . . . . . .20
Table 10 - Air Flow Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22
Table 11 - Serial Terminal Program Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33
Table 12 - Configuration Wizard Session - IP Configuration by DHCP . . . . . . . . . . . . . . . . . .34
Table 13 - Configuration Wizard Session - IP Zeroconf Configuration . . . . . . . . . . . . . . . . . . .35
Table 14 - Configuration Wizard Session - Static IP Configuration . . . . . . . . . . . . . . . . . . . . . .36
Table 15 - Port Splitting Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44
Table 16 - SX6015 Specification Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54
Table 17 - SX6018 Specification Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56
Table 18 - InfiniBand QSFP Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60
Table 19 - RJ-45 CONSOLE Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62
Table 20 - RJ-45 I2C Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62
Table 21 - OPNs for Replacement Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

SwitchX®-2 18-Port QSFP FDR InfiniBand System Hardware User Manual Rev 1.0
Mellanox Technologies 7
Revision History
Table 1 - Revision History Table
Date Revision Description
March 2013 1.0 Initial release
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Rev 1.0
Mellanox Technologies
8
About this Manual
This manual describes the installation and basic use of the Mellanox switch which is based on the
SwitchX®-2 switch device.
Intended Audience
This manual is intended for users and system administrators responsible for installing, running,
and managing switch platforms.
The manual assumes familiarity with the InfiniBand®Architecture Specification.
Related Documentation
Additional Documentation available from Mellanox:
All of these documents can be found on the Mellanox Website. They are available either through
the product pages or through the support page with a login and password.
Conventions
Throughout this manual, the names SX6018 and SX6015 and the term switch are used to
describe the 18-port 56Gb/s IB switch unless explicitly indicated otherwise.
The following icons are used throughout this document to indicate information that is important
to the user.
Table 2 - Reference Documents and Web Sites
Document Name Description
InfiniBand Architecture Specifica-
tion, Volume 1, Release 1.2.1 The InfiniBand Architecture Specification that is provided by IBTA
Mellanox SwitchX®-2 Switch Prod-
ucts Release Notes For possible hardware issues see the switch support product page. This
requires a customer support login.
Mellanox MLNX-OS® SwitchX®-2
Software User Manual This document contains information regarding configuring and manag-
ing Mellanox Technologies' SwitchX®-2 Switch Platforms.
MLNX-OS® Software Command Ref-
erence Guide Command Reference Guide for MLNX-OS® listing all of the commands
available through MLNX-OS® with explanations and examples.
MLNX-OS® Software Configuration
Guide Configuration Guide for MLNX-OS® displaying different configuration scenarios.
This icon makes recommendations to the user.
This icon indicates information that is helpful to the user.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

SwitchX®-2 18-Port QSFP FDR InfiniBand System Hardware User Manual Rev 1.0
Mellanox Technologies 9
Abbreviations
FDR — Fourteen Data Rate – Used to indicate a 4X port running at 56Gb/s
Mellanox Part Numbering Legend
This icon indicates a situation that can potentially cause damage to hardware or soft-
ware.
BEWARE! This icon indicates a situation that can potentially cause personal injury or
damage to hardware or software.
Place Field Decoder
M Mellanox Technologies
SX System Type SwitchX® switch Family
P Data Transfer Protocol 6 = InfiniBand
R Size of box 0 = 1U
1 = 1.5U
2 = 2U
FF Management Capabilities 15 = 18 Ports externally managed
18 = 18 Ports internally managed
25 = 36 Ports externally managed
36 = 36 Ports internally managed
C Data Rate F = FDR, T = FDR10, Q = QDR, D = DDR
- Separator
P # Power Supplies 0=0, 1=1, 2=2....
M Depth of the Unit S = standard depth, B = short depth
Y Air Flow direction R= Connector side to Power side airflow 
F= Power Side side to Connector side airflow
R Chip Generation R – SwitchX
S – SwitchX-2
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

OverviewRev 1.1
Mellanox Technologies
10
1 Overview
The SX6015 and SX6018 switch systems provides the highest performing fabric solution in a
1RU form factor by delivering 2Tb/s of non-blocking bandwidth with 170ns port-to-port latency.
Built with Mellanox’s 6th generation SwitchX®-2 InfiniBand switch device, these switches pro-
vides up to eighteen 56Gb/s full bi-directional bandwidth per port. These stand-alone switches
are an ideal choice for top-of-rack leaf connectivity or for building small to extremely large sized
clusters.
The SX6018 with Virtual Protocol Interconnect (VPI) supporting InfiniBand and Ethernet con-
nectivity provide the highest performing and most flexible interconnect solution for PCI Express
Gen3 servers. VPI simplifies system development by serving multiple fabrics with one hardware
design. VPI simplifies today network by enabling one platform to run both InfiniBand and Ether-
net subnets on the same chassis.
The SX6000 switch family enables efficient computing with features such as static routing, adap-
tive routing, and advanced congestion management. These features ensure the maximum effec-
tive fabric bandwidth by eliminating congestion hot spots.
Figure 1: Connector Side View of the Switch
1.1 Features
• 18 FDR (56Gb/s) ports in 1U switch
• IBTA 1.3 and 1.21 compliant
• SDR/DDR/QDR/FDR10/FDR link speed
• Congestion control
• Adaptive routing*
• Forward Error Correction (FEC)
• Port mirroring*
• IPoIB Bridging*
• 19” rack mountable chassis, 1U with redundant power supplies and Fan units
* This feature will be available to customers in the near future.
1.2 Serial Number and Product Version Information
The serial number and GUID for the switches are found on the pull out tab below the USB con-
nector. On managed switches only, the MAC for the Management PC is also on this tab. This
MAC is for the mgmt0 management interface (the top Ethernet port). The second interface
mgmt1 has the next consecutive MAC. Externally managed switches do not have a MAC on the
tab.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

SwitchX®-2 18-Port QSFP FDR InfiniBand System Hardware User Manual Rev 1.1
Mellanox Technologies 11
Figure 2: Pull out tab
1.3 FDR Transmission Rate
Mellanox SX60XX switch systems support FDR (fourteen data rate), an InfiniBand data rate,
where each lane of a 4X port runs a bit rate of 14.0625Gb/s with a 64b/66b encoding, resulting in
an effective bandwidth of 54.54Gb/s. The FDR physical layer is an IBTA specified physical layer
using advanced block types, deskew mechanism and framing rules for supporting higher speeds
and increased reliability.
SX60XX switches also supports FDR10, a non-standard InfiniBand data rate, where each lane of
a 4X port runs a bit rate of 10.3125Gb/s with a 64b/66b encoding, resulting in an effective band-
width of 40Gb/s.
FDR10 supports 20% more bandwidth over QDR using the same QSFP cables/connectors.
Both FDR and FDR10 support Forward Error Correction (FEC), as described in IEEE 802.3ap
chapter 74.
FDR and FDR10 are only guaranteed to work with approved Mellanox Cables.
FDR10 is only guaranteed to work with approved Mellanox ConnectX®-3 adapters.
S/N:MT1117X00014
P/N:MSX6018F‐1BFS
GUID: 0002C9020040DDB0
MadeinIL
MGT2:
00:02:C9:11:A2:02
Rev:A1
MGT1:
00:02:C9:11:A2:01
SX6018
CONSOLE
MGT
16 17 18
Pull out tab
SX6015
I2C
16 17 18
S/N:MT1117X00014
P/N:MSX60015F‐1BFS
GUID:0002C9020040DDC0
MadeinIL
Rev:A1
Pull out tab
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

OverviewRev 1.1
Mellanox Technologies
12
1.4 Internally Managed vs. Externally Managed
The following table shows which switches come with a management CPU and which do not.
All internally managed switches have internal chassis management and can support IB fabric of
up to 648 nodes. Internally managed switches need an initial configuration before they will start
working.
Externally Managed (Unmanaged) switches are plug and play out of the box. Updating the firm-
ware stack for the externally managed switches is done in-band only. You will need to confirm
that your switch has the latest firmware. For externally managed switches, confirm that the latest
FW revision is installed on this device. Use the CLI to get the currently installed revision and
then go to Mellanox.com to check the revision of the latest version.
Externally managed switches can only run in an IB fabric. Externally managed switches cannot
run Ethernet protocol.
Table 3 - Switch Management
Family Externally /
Internally
Managed Management Connections
SX6015 Externally
Managed
(Unmanaged)
Plug and Play
All firmware updates should be done in-band using Mellanox firmware man-
agement tools. I2C port access using MTUSB-1 device is required for firm-
ware updates if in-band burning is not possible.
SX6018 Internally
Managed RS232 cable DB9 to RJ-45 included in the box to connect to host PC for ini- tial configuration of the switch. After initial configuration, the switch can be
managed through the Ethernet port using a remote connection.
Gateway functionality is not supported in the externally managed SwitchX® switches.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

SwitchX®-2 18-Port QSFP FDR InfiniBand System Hardware User Manual Rev 1.1
Mellanox Technologies 13
2Basic Operation
2.1 Switch Platform Hardware Overview
Figure 3 shows the power side panel view of the switch. The figure shows the hot-swap power
supply locations and the hot-swap fan module, and an I2C banana connector.
Figure 3 shows the connector side panel views of the internally managed and externally managed
switches. The figure shows port configurations for the switch systems. Managed systems have 2
– Ethernet RJ-45 connectors for management, 1 – RJ-45 connector for connecting to a host PC, 1
– USB connector. Externally managed switches only have a single I2C RJ-45 connector.All
switches have various status LEDs for on site switch status information.
Figure 3: QSFP Switch System Power and Connector Side Panels
Figure 4: QSFP Switch System Connector Side
All connectivity except for power cords is via the connector side panel. All connectors can sup-
port fiber cables.
2.2 Status LEDs
2.2.1 System Status Indicators
The System Status Indicators are located to the left of the connectors on the connector side panel,
and on the power side at the far right. Both of these LEDs give identical information.
OK
!
!
OK
OK
!
!
OK
12C
Power Side Panel
SX6018
CONSOLE
MGT
RST
UID
PS2
PS1
12 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
SX6018 Connector Side Panel
SX6015
I2C
RST
UID
PS2
PS1
12 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
SX6015 Connector Side Panel
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Basic OperationRev 1.1
Mellanox Technologies
14
Figure 5: Externally Managed Switch System Power and Connector Side Panels
Figure 6: System Status Indicators
The system status indicators should display as follows:
• when the switch is plugged in, within five minutes the STATUS LED should light up
green
• the PS1 LED for the plugged in PS unit should light up green
• the PS2 LED for the second PS unit should light up green only if a second PS unit is
installed in the switch for redundancy and it is connected to a power source
• if two PS units are installed and only one PS unit is connected to a power supply the PS2 LED will
be red
• if only one PS unit is installed in the switch, the PS2 LED will be off
• the FAN LED should light up green. If the FAN LED shows red, troubleshoot the fan
module. See “Troubleshooting” on page 51.
OK
!
!
OK
OK
!
!
OK
12C
SX6015
I2C
RST
UID
PS2
PS1
12 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Power Side Panel
Connector Side Panel
These two LEDs indicate the status of the switch as assigned in Table 4
on page 15.
Power side 
status LED
Connector side 
status LED
Both of these LEDs show the
switch status.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

SwitchX®-2 18-Port QSFP FDR InfiniBand System Hardware User Manual Rev 1.1
Mellanox Technologies 15
If the PS LEDs are not green, this indicates a problem with the power supplies. The switch is
operational only if at least one of the PS LEDs is green.
Figure 7: System LEDs
Table 4 shows the system status LED assignment.
As long as there is power to the switch (one PS unit is connected), and the switch is
booted up and running, the status LED should be green.
If the STATUS LED shows red after five minutes, unplug the switch and call your
Mellanox representative for assistance.
If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes
and replug in the switch. For more information See “Troubleshooting” on page 51.
Table 4 - System Status LED Assignments
LED Configuration STATUS/ System Health LED
Solid Green OK – The system is up and running.
Flashing Green The system is booting up. This assignment is valid on managed systems only.
Solid Yellow To be implemented in a future SW release.
Solid Red Major Error – Possible damage can result to the switch. Turn off immediately.
e.g. bad firmware, can’t boot, overheated. If the system is booting up, it can take
up to 5 minutes for the status led to change to green.
Off Off – The system has no power.
RST
UID
PS2
PS1
123 4
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Basic OperationRev 1.1
Mellanox Technologies
16
2.2.2 Fan Status Indicators
The indicator is located to the left of the connectors on the connector side panel.
Figure 8: Fan Status LED Connector Side
The LED indicator on the Fan Module is on the right side of the module.
Figure 9: Fan Status LED Power side
Table 5 shows the fan status LED assignment.
Table 5 - Fan Status LED Assignments
LED Configuration FAN LED
Solid Green OK – All fans are up and running.
Solid Red Error – One or more fans is not operating properly. The system should be powered
down and troubleshoot the fan module.
Off Off – The fan unit is not receiving any power. Check that the fan unit is properly
and completely inserted.
RST
UID
PS2
PS1
This is a two color LED
!
This is a two color LED
Fan air flow direction
indicator
UID
RST
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

SwitchX®-2 18-Port QSFP FDR InfiniBand System Hardware User Manual Rev 1.1
Mellanox Technologies 17
2.2.3 Power Supply Status Indicators
The indicator is located to the left of the connectors on the connector side panel.
Figure 10: Power Status LED Connector Side
The switch is available with one factory installed Power Supply Unit. A second Power Supply
Unit can be added to support hot-swap ability and to add redundancy. See “Accessory and
Replacement Parts” on page 63 for ordering part numbers.
Figure 11: Power Side Panel
The primary power supply unit (PS1) is located on the right side of the power side panel, with
PS2 on the left side. Each PS unit has a single 2 color LED on the right side of the PS unit, that
indicates the internal status of the unit.
All fans must be operating while the power supply is plugged in.
If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes
and replug in the switch. For more information See “Troubleshooting” on page 51.
RST
UID
PS2
PS1
These are two color LEDs
OK
!
!
OK
OK
!
!
OK
12C
Primary Power Supply Unit
Fan Unit
PS unit 1 status LED
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Basic OperationRev 1.1
Mellanox Technologies
18
Figure 12: PS Unit Status LEDs
Table 6 shows the power supply unit status LED assignment.
2.2.3.1 Bad Port LED
Figure 13: Bad Port LED
The Bad Port indicator is located on the left side of the connector side panel of the unit. Table 7
shows the fan status LED assignment.
Table 6 - PS Unit Status LED Assignments
LED Color Status
Solid Green OK – The Power supply is delivering the correct voltage – 12VDC.
Solid Red Error – The PS unit is not operational.
Off Off – There is no power to the system (neither PS unit is receiving power). If one
PS unit is showing green and the second PS unit is unplugged it will show a red
indication.
This is a two color LED
Power supply fan air flow
direction indicator
RST
UID
PS2
PS1
This is a one color LED
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

SwitchX®-2 18-Port QSFP FDR InfiniBand System Hardware User Manual Rev 1.1
Mellanox Technologies 19
This LED lights up when one or more ports is receiving a symbol error. The LED immediately
goes off until the next symbol error is received.
2.2.3.2 UID LED Switch Identifier
The UID LED is a debug feature that lights a blue LED on the switch connector side panel for
ease in finding a particular switch.
2.2.3.3 Port Connector LED Indicators
Above the ports are LEDs, one for each port . Table 8 shows the port status according to the
LED indication for Ethernet protocol. Table 9 shows the port status according to the LED indica-
tion for InfiniBand protocol.
Figure 14: Port LEDs
Table 7 - Bad Port LED Assignments
LED Configuration Description
Off OK – No symbol errors have been received in last few seconds.
Flashing Orange Error – One or possibly more ports have just received a symbol error
Possible causes are:
• Bad cable
• Bad connection
• Bad connector
Table 8 - Port Connector Physical and Logical Link Assignments for Ethernet
LED Configuration LED Description
Off Physical link is down / Default.
Solid Green Physical link is up with no traffic.
Flashing Green Physical link is up with traffic.
Flashing Orange Physical errors.
RST
UID
PS2
PS1
123 456
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Basic OperationRev 1.1
Mellanox Technologies
20
The LED indicator, corresponding to each data port, will light orange when the physical connec-
tion is established (that is, when the unit is powered on and a cable is plugged into the port with
the other end of the connector plugged into a functioning port). When a logical connection is
made the LED will change to green. When data is being transferred the light will blink green.
2.3 Air Flow
These switches can come with two air flow patterns. The two patterns are:
• Connector side inlet to power side outlet
• Power side inlet to connector side outlet
The air flow is specified in the product model number. See “Mellanox Part Numbering Legend”
on page 9.On the PS unit and fan modules the air flow direction can be seen on the power side
panel. All components need to have the same air flow. A mismatch in the air flow will affect the
heat dissipation.
Table 9 - Connector Physical and Logical Link Assignments for IB
LED Status LED Description
Off No power to the port.
Solid Green Logical link is up.
Flashing Green Data activity flashing speed is proportional to data transfer speed.
Solid Orange Physical link is up.
Flashing Orange A problem with the physical link.
All PS units must have the same air flow as the fan module.
Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.