Mellanox Technologies TX6100 MetroX Series Instructions for use

www.mellanox.com
MetroX™ InfiniBand Long Haul System
Hardware User Manual
P/N: MTX6100-2SFS
Rev 1.1

Document Number: 4067
Rev 1.1
Mellanox Technologies
2
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
Beit Mellanox
PO Box 586 Yokneam 20692
Israel
www.mellanox.com
Tel: [phone]
Fax: [phone]
© Copyright 2013. Mellanox Technologies. All Rights Reserved.
Mellanox®, Mellanox logo, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®, PhyX®,
SwitchX®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of Mellanox Technologies, Ltd.
Connect-IB™, FabricIT™, MLNX-OS™, MetroX™, ScalableHPC™, Unbreakable-Link™, UFM™ and Unified Fabric
Manager™ are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE
THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT
HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE
PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND
DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST
QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES
FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND
(INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED
DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

MetroX 12-Port QSFP InfiniBand Long Haul System Hardware User Manual Rev 1.1
Mellanox Technologies 3
Table of Contents
Table of Contents 3
List of Figures 5
List of Tables 6
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
Mellanox Part Numbering Legend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
Chapter 1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1.1 Full Feature List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2 Serial Number and Product Version Information. . . . . . . . . . . . . . . . . . . 11
Chapter 2 Basic Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1 Switch Platform Hardware Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.1 Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Air Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 QSFP Cable Power Budget Classification. . . . . . . . . . . . . . . . . . . . . . . . 20
2.4 Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4.1 Port Connector Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4.2 Management and Firmware Updating Interfaces . . . . . . . . . . . . . . . . . . . 21
2.4.3 Reset Button. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Chapter 3 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.1 Installation Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.1.1 Installation Kits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 Package Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3 Mechanical Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.4 Grounding the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.5 Power Connections and Initial Power On . . . . . . . . . . . . . . . . . . . . . . . . 32
3.6 Configuring the Switch the First Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.7 Rerunning the Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.8 Connecting to the Switch Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.8.1 Starting a Remote Connection to the Switch . . . . . . . . . . . . . . . . . . . . . . 40
3.9 Starting an SSH Connection to the Switch (CLI). . . . . . . . . . . . . . . . . . . 40
Chapter 4 Extracting and Inserting FRUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.1 Extracting and Inserting the Power Supply Unit . . . . . . . . . . . . . . . . . . . 41
4.2 Extracting and Inserting the Fan Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Chapter 5 Cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Rev 1.1
Mellanox Technologies
4
5.1 Supported Approved Cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.2 Splitting the Data Stream Using Breakout (Fanout) Cables . . . . . . . . . . 44
5.3 Cable Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Chapter 6 Switch Shut Down Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Chapter 7 Disassembly of the Switch from the Rack. . . . . . . . . . . . . . . . . . . . 47
7.1 Disposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Chapter 8 Resetting the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Chapter 9 Management and Tools Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 49
9.1 Switch Management Using the MLNX-OS™ Software . . . . . . . . . . . . . . 49
9.2 Switch Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
9.2.1 InfiniBand Subnet Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
9.3 Fabric Inspector (Diagnostics). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
9.3.1 Upgrading Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Chapter 10 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Appendix A Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
A.1 Approved Cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55
A.2 EMC Certifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55
A.3 China CCC Warning Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56
Appendix B QSFP Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Appendix C RJ-45 CONSOLE and Ethernet interfaces. . . . . . . . . . . . . . . . . . . 59
Appendix D Accessory and Replacement Parts . . . . . . . . . . . . . . . . . . . . . . . . 60
Appendix E Avertissements de sécurité d’installation (French). . . . . . . . . . . 61
Appendix F Installation - Sicherheitshinweise (German). . . . . . . . . . . . . . . . . 65
Appendix G Advertencias de seguridad para la instalación (Spanish) . . . . . 69

MetroX 12-Port QSFP InfiniBand Long Haul System Hardware User Manual Rev 1.1
Mellanox Technologies 5
List of Figures
Figure 1: Connector Side View of the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
Figure 2: Pull out tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11
Figure 3: QSFP Switch System Power and Connector Side Panels . . . . . . . . . . . . . . .12
Figure 4: System Status Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13
Figure 5: Fan Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
Figure 6: Fan Status LED Power side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
Figure 7: Power Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
Figure 8: Power Side Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
Figure 9: PS Unit Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
Figure 10: Bad Port LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Figure 11: Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18
Figure 12: All Components Need the Same Air Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19
Figure 13: Port Numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21
Figure 14: Top and Bottom QSFP Port Orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21
Figure 15: Management Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21
Figure 16: Reset Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23
Figure 17: Rack Rail Kit Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28
Figure 18: Placement of Switch in Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29
Figure 19: Mounting Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30
Figure 20: Screwing on the Rail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30
Figure 21: Inserting the Caged Nuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31
Figure 22: Slide the Rail into the Rail Slide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31
Figure 23: Status LEDs 5 Minutes After Power On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32
Figure 24: Two Power Inlets - Electric Caution Notifications . . . . . . . . . . . . . . . . . . . . . . . .33
Figure 25: Console Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34
Figure 26: Fan and PS unit Air Flow Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41
Figure 27: Power Supply Unit Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41
Figure 28: PS Unit Pulled Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42
Figure 29: Air Flow Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43
Figure 30: Fan Module Latches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43
Figure 31: Breakout or Fanout Cable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44
Figure 32: Port Splitting Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44
Figure 33: Reset Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48
Figure 34: QSFP Connector Male and Female Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58

Rev 1.1
Mellanox Technologies
6
List of Tables
Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
Table 2: Reference Documents and Web Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Table 3: System Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
Table 4: Fan Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
Table 5: PS Unit Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Table 6: Bad Port LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Table 7: Connector Physical and Logical Link Assignments for IB . . . . . . . . . . . . . . . . . .18
Table 8: Air Flow Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20
Table 9: Installation Kit According to Rack Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27
Table 10: Serial Terminal Program Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34
Table 11: Configuration Wizard Session - IP Configuration by DHCP . . . . . . . . . . . . . . . .35
Table 12: Configuration Wizard Session - IP Zeroconf Configuration . . . . . . . . . . . . . . . . .37
Table 13: Configuration Wizard Session - Static IP Configuration . . . . . . . . . . . . . . . . . . .38
Table 14: TX6100 Specification Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54
Table 15: InfiniBand QSFP Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57
Table 16: RJ-45 Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59
Table 17: Replacement Parts Ordering Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60

MetroX 12-Port QSFP InfiniBand Long Haul System Hardware User Manual Rev 1.1
Mellanox Technologies 7
Revision History
Table 1 - Revision History Table
Date Revision Description
January 2013 1.1 Fixed - 2 power supplies
no wire bale
December 2012 1.0 Initial release

Rev 1.1
Mellanox Technologies
8
About this Manual
This manual describes the installation and basic use of the Mellanox MetroX Long Haul switch
which is based on the SwitchX®-2 switch device.
Intended Audience
This manual is intended for users and system administrators responsible for designing and testing
switch platforms.
Related Documentation
Additional Documentation available from Mellanox:
All of these documents can be found on the Mellanox Website. They are available either through
the product pages or through the support page with a login and password.
Conventions
Throughout this manual, the name TX6100 and the term switch are used to describe the , switch
unless explicitly indicated otherwise.
The following icons are used throughout this document to indicate information that is important
to the user.
Table 2 - Reference Documents and Web Sites
Document Name Description
InfiniBand Architecture Specifica-
tion, Volume 1, Release 1.2.1 The InfiniBand Architecture Specification that is provided by IBTA
Mellanox SwitchX®-2 Switch Prod-
ucts Release Notes For possible hardware issues see the switch support product page. This
requires a customer support login.
Mellanox MLNX-OS™ SwitchX-2
Software User Manual This document contains information regarding configuring and manag-
ing Mellanox Technologies' SwitchX-2 Switch Platforms.
MLNX-OS Software Command Refer-
ence Guide Command Reference Guide for MLNX-OS listing all of the commands
available through MLNX-OS with explanations and examples.
MLNX-OS Software Configuration
Guide Configuration Guide for MLNX-OS displaying different configuration scenarios.
This icon makes recommendations to the user.
This icon indicates information that is helpful to the user.

MetroX 12-Port QSFP InfiniBand Long Haul System Hardware User Manual Rev 1.1
Mellanox Technologies 9
Abbreviations
FDR — Fourteen Data Rate
Mellanox Part Numbering Legend
This icon indicates a situation that can potentially cause damage to hardware or soft-
ware.
BEWARE! This icon indicates a situation that can potentially cause personal injury or
damage to hardware or software.
Place Field Decoder
MMellanox
TX System Type MetroX based systems
6Family generation
1Data Transfer Protocol 1 – IB, 2 – ETH
0Height of box Height 0 – 1U, 1 – 1.5U, 2 – 2U
0Switch generation
-Separator
2# Power Supplies # of PSUs
SDepth of the Unit S – standard, B – short
FAir Flow direction F – forward air flow, R – Reverse air flow
SChip Generation S – SwitchX-2

OverviewRev 1.1
Mellanox Technologies
10
1 Overview
The TX6100 MetroX™ series extends InfiniBand from a single-location data center network to a
high-performance technology for local, campus and even metro applications.
Data centers, compute clusters and supercomputers are overwhelmed by unprecedented growth
in data volume, fueled by strong application and technology trends.
While InfiniBand products have been traditionally deployed for their high-performance intercon-
nect benefits within the data center, Mellanox MetroX switches, implementing long-haul Infini-
Band, enable connecting between data centers deployed across multiple geographically
distributed sites, extending the same world-leading interconnect benefits of InfiniBand beyond
local data centers and storage clusters.
Mellanox’s MetroX is the perfect cost-effective, low power, easily managed and scalable solu-
tion that enables today’s data centers and storage to run over local and distributed InfiniBand fab-
rics, managed as a single unified network infrastructure.
MetroX switches, which implement long-haul InfiniBand, can transfer data to distances of up to
10km. The switches enable aggregate data and storage networking over a single, consolidated
InfiniBand fabric. The long-haul InfiniBand technology guarantees high-performance, high-vol-
ume data sharing between distant sites, enabling existing data centers expansion, disaster recov-
ery, data mirroring and campus connectivity.
MetroX enables a campus network to assemble large aggregate clusters, all connected and easily
managed by an InfiniBand Subnet Manager - an embedded manager, OpenSM, or using Mella-
nox’s Unified Fabric Manager™ (UFM™).
MetroX extends InfiniBand protocol RDMA capabilities beyond the local cluster and ensures
RDMA enhancements to both data centers and storage clusters alike.
The switch comes pre-installed with all necessary firmware and is configured for standard oper-
ation within an InfiniBand fabric. See Section 9 on page 49 for more information.
Installation, hot-swapping components and hardware maintenance is covered in “Basic Opera-
tion” on page 12.
Figure 1: Connector Side View of the Switch
1.1 Features
1.1.1 Full Feature List
• 6 FDR10 (40Gb/s) QSFP Long haul ports (aggregate data throughput up to 240Gb/s)
• 6 FDR (56Gb/s) QSFP down link ports (aggregate data throughput up to 336Gb/s)
MT6100
CONSOLE
MGT
12
11
10
9
8
7
6
5
4
3
2
1
EXT1EXT2EXT3EXT4EXT5EXT6
RST
UID
PS2
PS1

MetroX 12-Port QSFP InfiniBand Long Haul System Hardware User Manual Rev 1.1
Mellanox Technologies 11
• IBTA 1.3 and 1.21 compliant
• FDR10 supports 20% more bandwidth over QDR using the same cables/connectors**
• QoS (2 Virtual Lanes)
• InfiniBand connectivity to up to 10km
• RDMA over long distance
• 19’’ rack mountable chassis, 1U with redundant power supplies and Fan units
• 6 Downlink QSFP+ non blocking ports with aggregate data throughput up to 336Gb/s
(FDR)
• 6 Uplink QSFP+ non-blocking ports with aggregate data throughput up to 240Gb/s
• Compliant with IBTA 1.2.1 and 1.3
• Virtual Lanes - 2 data + 1 management
1.2 Serial Number and Product Version Information
The serial number and GUIDfor the switch is found on the pull out tab below the USB connector.
TheMAC for the Management PC is also on this tab. This MAC is for the mgmt0 management
interface (the top Ethernet port). The second interface mgmt1 has the next consecutive MAC.
Figure 2: Pull out tab
TX6100
CONSOLE
MGT
12110
S/N:MT1117X00014
P/N:MTX6100‐2SFS
GUID:0002C9020040DDB0
MGT1: 00:02:C9:11:A2:01
Rev:A2
MGT2: 00:02:C9:11:A2:02
Pull out tab

Basic OperationRev 1.1
Mellanox Technologies
12
2Basic Operation
2.1 Switch Platform Hardware Overview
Figure 3 shows the power side panel view of the switch. The figure shows the hot-swap power
supply locations and the hot-swap fan module, and an I2C banana connector.
The figure also shows the connector side panel views of the managed switch. The figure shows
port configurations for the switch systems. Managed systems have 2 – Ethernet RJ-45 connectors
for management, 1 – RJ-45 connector for connecting to a host PC, 1 – USB connector. All
switches have various status LEDs for on site switch status information.
Figure 3: QSFP Switch System Power and Connector Side Panels
All connectivity except for power cords is via the connector side panel. All connectors can sup-
port active cables.
2.1.1 Status LEDs
2.1.1.1 System Status Indicators
The System Status Indicators are located to the left of the connectors on the connector side panel,
and on the power side at the far right. Both of these LEDs give identical information.
TX6100
CONSOLE
MGT
121110987654
321
EXT1EXT2EXT3EXT4EXT5EXT6
RST
UID
PS2
PS1
Connector Side Panel
Power Side Panel
OK
!
!
OK
OK
!
!
OK
12C

MetroX 12-Port QSFP InfiniBand Long Haul System Hardware User Manual Rev 1.1
Mellanox Technologies 13
Figure 4: System Status Indicators
The system status indicators should display as follows:
• When the switch is plugged in, within five minutes the STATUS LED should light up
green.
• The PS1 LED for the plugged in PS unit should light up green
• The PS2 LED for the second PS unit should light up green only if a second PS unit is
installed in the switch for redundancy and Hot-Swap ability and it is connected to a
power source.
• If two PS units are installed and only one PS unit is connected to a power supply the PS2
LED will be red.
• If only one PS unit is installed in the switch, the PS2 LED will be off.
• The FAN LED should light up green.
If the FAN LED shows red, troubleshoot the fan module. See “Troubleshooting” on page 51.
If the PS LEDs are not green, this indicates a problem with the power supplies. The switch is
operational only if at least one of the PS LEDs is green.
As long as there is power to the switch (one PS unit is connected), and the switch is
booted up and running, the status LED should be green.
If the STATUS LED shows red after five minutes, unplug the switch and call your
Mellanox representative for assistance.
If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes
and replug in the switch. For more information See “Troubleshooting” on page 51.
Power side 
status LED
Connector side 
status LED
Green – OK
Yellow – a fault in the system
Red – Major Fault or Fatal Error
LED color shows the
switch status
These three symbols show the possible
colors for the LED below, Green, Yellow,
and Red.
Both of these LEDs show the
switch status. One is on the
Power side and one is on the
Connector side

Basic OperationRev 1.1
Mellanox Technologies
14
Power, Fan, and System LEDs
2.1.1.2 Fan Status Indicators
The indicator is located to the left of the connectors on the connector side panel.
Figure 5: Fan Status LED Connector Side
The LED indicator on the Fan Module is on the right side of the module.
Table 3 - System Status LED Assignments
LED Configuration STATUS/ System Health LED
Solid Green OK – The system is up and running.
Flashing Green The system is booting up. This assignment is valid on managed systems only.
Solid Yellow To be implemented in a future SW release.
Solid Red Major Error – Possible damage can result to the switch. Turn off immediately.
e.g. bad firmware, can’t boot, overheated
Off Off – The system has no power.
321
EXT1EXT2EXT3
RST
UID
PS2
PS1
RST
UID
PS2
PS1
This is a two color LED
Green — OK
Red — Major fault
Off — No power to the fans

MetroX 12-Port QSFP InfiniBand Long Haul System Hardware User Manual Rev 1.1
Mellanox Technologies 15
Figure 6: Fan Status LED Power side
The following fan status conditions are possible:
2.1.1.3 Power Supply Status Indicators
The indicator is located to the left of the connectors on the connector side panel.
Table 4 - Fan Status LED Assignments
LED Configuration FAN LED
Solid Green OK – All fans are up and running
Solid Red Error – One or more fans is not operating properly. The system should be powered
down and troubleshoot the fan module.
Off Off – The fan unit is not receiving any power. Check that the fan unit is properly
and completely inserted.
All fans must be operating while the power supply is plugged in.
If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes
and replug in the switch. For more information See “Troubleshooting” on page 51.
!
This is a two color LED
Green — OK
Red — Major fault
Off — No power to the fans
Fan air flow indicator

Basic OperationRev 1.1
Mellanox Technologies
16
Figure 7: Power Status LED Connector Side
This switch comes with two power supply units for full N+1 redundancy.
Figure 8: Power Side Panel
The primary power supply unit (PS1) is located on the right side of the power side panel, with
PS2 on the left side. Each PS unit has a single 2 color LED on the right side of the PS unit, that
indicates the internal status of the unit.
Figure 9: PS Unit Status LEDs
RST
UID
PS2
PS1
These are two color LEDs
Green — OK
Red — Major fault
Off — No power
OK
!
!
OK
OK
!
!
OK
12C
Primary Power Supply Unit
Fan Unit
PS unit 1 status LED
Fan status LED
This is a two color LED
Green — OK
Red — Major fault
Off — No power to the system
Power supply fan air flow
indicator

MetroX 12-Port QSFP InfiniBand Long Haul System Hardware User Manual Rev 1.1
Mellanox Technologies 17
2.1.1.4 Bad Port LED
Figure 10: Bad Port LED
The Bad Port indicator is located on the left side of the connector side panel of the unit. The fol-
lowing Bad Port conditions are possible.
This LED shows symbol errors. Possible causes for this are:
• Bad cable
• Bad connection
• Bad connector
This LED lights up when one or more ports is receiving a symbol error. The LED immediately
goes off until the next symbol error is received.
2.1.1.5 UID LED Switch Identifier
The UID LED is a debug feature that lights a blue LED on the switch connector side panel for
ease in finding a particular switch.
Table 5 - PS Unit Status LED Assignments
LED Color Status
Solid Green OK – The Power supply is delivering the correct voltage – 12VDC
Solid Red Error – The PS unit is not operational
Off Off – There is no power to the system (neither PS unit is receiving power). If one
PS unit is showing green and the second PS unit is unplugged it will show a red
indication.
Table 6 - Bad Port LED Assignments
LED Configuration Description
Off OK – No symbol errors have been received in last few seconds
Flashing Orange Error – One or possibly more ports have just received a symbol error
!
PS1
PS2
!
UID
RST
This is a one color LED
Flashing Orange — received symbol
error
UID

Basic OperationRev 1.1
Mellanox Technologies
18
2.1.1.6 Port Connector LED Indicators
Above the uplink ports are LEDs, one for each port . The following table shows the port sta-
tus according to the LED indication.
Above the downlink ports are LEDs, 1 per port . The following table shows the port status
according to the LED indication.
Figure 11: Port LEDs
The LED indicator, corresponding to each data port, will light orange when the physical connec-
tion is established (that is, when the unit is powered on and a cable is plugged into the port with
the other end of the connector plugged into a functioning port). When a logical connection is
made the LED will change to green. When data is being transferred the light will blink green.
2.2 Air Flow
These switches can come with two air flow patterns. The two patterns are:
• Connector side inlet to power side outlet
• Power side inlet to connector side outlet
Table 7 - Connector Physical and Logical Link Assignments for IB
LED Status LED Description
Off No power to the port
Solid Green Logical link up
Flashing Green Data activity flashing speed is proportional to data transfer speed
Solid Orange Physical link up
Flashing Orange A problem with the physical link,
21
EXT1EXT2
RST
UID
PS2
PS1
Port LEDs
TX6100
CONSOLE
MGT
12110

MetroX 12-Port QSFP InfiniBand Long Haul System Hardware User Manual Rev 1.1
Mellanox Technologies 19
The air flow is specified in the product model number. See “Mellanox Part Numbering Legend”
on page 9. On the PS unit and fan modules the air flow direction can be seen on the power side
panel.
Figure 12: All Components Need the Same Air Flow
All PS units must have the same air flow as the fan module.
OK
!
!
OK
OK
!
!
OK
12C
These air flow labels must be the same.
Red is connector side inlet to power side outlet
Blue is power side inlet to connector side outlet

Basic OperationRev 1.1
Mellanox Technologies
20
2.3 QSFP Cable Power Budget Classification
AlMetroX downlink ports l SwitchX®-2are designed for active cables with a max power per
module of 2W. Typical power per port is 1.5W
The MetroX uplink (long haul) ports are designed for active cables with a max power per module
of 3.5W. Typical power per port is 2.5W.
2.4 Interfaces
2.4.1 Port Connector Interfaces
The connector side of the switch has 12 QSFP ports. The 6 on the left side (numbered 1 -6) are
long haul ports that require 3.5W max power per port and can drive the signal up to 10 km. The 6
ports on the right side (numbered 7 -12) are the downlink FDR ports. Four of the six downlink
ports can be split to 4 X 10 Gb/s.
Table 8 - Air Flow Direction
Direction Label OPN
Designation Description
RConnector side inlet to power side
outlet
F
Power side inlet to connector side
outlet