Mellanox Technologies MIS5022Q-1BRR User manual

www.mellanox.com
InfiniScale® IV 8-Port QSFP 40Gb/s InfiniBand
Switch User Manual
P/N:MIS5022Q-1BRR, MIS5022Q-1BFR, MIS5024Q-1BRR,
MIS5024Q-1BFR, MIS5022Q-1BFR, MIS5022Q-1BRR,
MIS5022Q-1BFC, MIS5022Q-1BRC
Rev 1.2

Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
PO Box 586 Hermon Building
Yokneam 20692
Israel
Tel: [phone]
Fax: [phone]
© Copyright 2011. Mellanox Technologies. All rights reserved.
Mellanox, BridgeX, ConnectX, Virtual Protocol Interconnect, InfiniBlast, InfiniBridge, InfiniHost, InfiniRISC, InfiniScale, and InfiniPCI
are registered trademarks of Mellanox Technologies, Ltd. CORE-Direct, FabricIT, and PhyX are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
All other marks and names mentioned herein may be trademarks of their respective companies.
QDR InfiniBand Switch Platform User Manual
Document Number: 3373
Rev 1.2
Mellanox Technologies
2
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE
PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE
OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE
CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX
TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX
TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE
DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,
PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORYOF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND
RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

QDR InfiniBand Switch Platform User Manual Rev 1.2
Mellanox Technologies
3
Contents
Contents 3
List of Figures 5
List of Tables 6
Revision History 7
About this Manual 8
Intended Audience 8
Related Documentation 8
Conventions 8
Mellanox Part Numbering Legend 9
Chapter 1 Overview 10
1.1 Serial Number and Product Version Information 10
Chapter 2 Installation Safety Warnings 11
Chapter 3 Externally Managed Switches 14
Chapter 4 Hardware Basic Operation and Installation 15
4.1 Switch Platform Hardware Overview 15
4.1.1 LED Assignments 15
4.2 AirFlow 18
4.3 QSFP Cable Power Budget Classification 18
4.4 Interfaces 19
4.4.1 Port Connector Interfaces 19
4.5 Management andFW Updating Interfaces 19
4.5.1 RJ-45 Connector (I2C) 19
Chapter 5 Installation 21
5.1 PackageContents 21
5.2 Installing the Switch in the Rack 22
5.2.1 Table Top Installation 22
5.2.2 Installation Procedure for a Single Switch Center-of-Rack Installation 23
5.2.3 Installation Procedure for a Side by Side Installation 24
5.2.4 Grounding the Switch 27
5.2.5 Power Connections and Initial Power On 27
5.2.6 InfiniBand Cable Installation 27
5.3 Disassembly of the Switch from the Rack 28
5.3.1 Disassembly Procedure — Single Switch Centered Installation 28
5.3.2 Disassembly Procedure — Side by Side Installation 29
5.4 Disposal 29
5.5 UpdatingFirmware 29
5.5.1 Current Firmware Revision 29
5.5.2 How to Get Mellanox Firmware Tools (MFT) 32
5.5.3 Open SM 32
Chapter 6 Troubleshooting 33
Appendix A Specification 35
A.1 EMC Certification Statements 36
A.2 VCCI Statements (Japan) 39
A.3 MIC Certification (Korea) 39
Appendix B QSFP Interface 40
Appendix C RJ45 I2C Interface 42

Rev 1.2
Mellanox Technologies
4
Appendix D Replacement Parts Ordering Numbers 43
Appendix E Avertissements de sécurité d’installation (French) 44
Appendix F Installation - Sicherheitshinweise (German) 46
Appendix G Advertencias de seguridad para la instalación (Spanish) 48
Appendix H Special Regulations Regarding Finland, Sweden, Denmark, and Norway 51

QDR InfiniBand Switch Platform User Manual Rev 1.2
Mellanox Technologies
5
List of Figures
Figure 1: Generic Product label 10
Figure 2: Switch System Front Panel 15
Figure 3: Status LEDs 16
Figure 4: Port Numbering 19
Figure 5: Top and Bottom Ports 19
Figure 6: RJ45 I2C Connector 20
Figure 7: Placing the Bumpers 22
Figure 8: Installation Kit for a Single Centered Installation 23
Figure 9: Connect the Mounting Brackets 24
Figure 10: Installation Kit Parts for a Side by Side Installation 25
Figure 11: Screw on the Switch Mounted Rails 25
Figure 12: Caged Nut Placement 26
Figure 13: Front Panel Orientation 26
Figure 14: Top and Bottom Ports 28
Figure 15: MTUSB-1 with Cables 31
Figure 16: I2C Cable Connected to IS5022 31
Figure 17: QSFP Male and Female Connections 41

Rev 1.2
Mellanox Technologies
6
List of Tables
Table 1: Revision History Table 7
Table 2: Reference Documents 8
Table 3: Switch Status LED Configurations 16
Table 4: Fan LED Configurations 17
Table 5: Bad Port LED Configurations 17
Table 6: Connector Physical and Logical Link Indications 18
Table 7: Installation Kit Options 21
Table 8: IS5022 Specification Data 35
Table 9: Switch Certification Status 36
Table 10: InfiniBand QSFP Connector Pinout 40
Table 11: Replacement Parts Ordering Numbers 43

QDR InfiniBand Switch Platform User Manual Rev 1.2
Mellanox Technologies
7
Revision History
Table 1 - Revision History Table
Date Revision Description
Mar. 2011 Rev 1.2 Added note and figure to blank panel installtion step.
Made changes to the installation process for the two switch frame.
Made changes to FW burning sections.
Feb. 2011 Rev 1.1 Removed UID LED support
Jan. 2011 Rev 1.0 Initial Release

Rev 1.2
Mellanox Technologies
8
About this Manual
This manual describes the installation and basic use of the Mellanox IS5022 switch, which is
based on the InfiniScale® IV InfiniBand switch device.
Intended Audience
This manual is intended for users and system administrators responsible for installing and setting
up the switch platforms listed above.
The manual assumes familiarity with the InfiniBand® Architecture Specification.
Related Documentation
Additional Documentation available from Mellanox:
Conventions
Throughout this manual, the name IS5022 and the term switch are used to describe the 8-port
QSFP 40Gb/s InfiniBand Switch unless explicitly indicated otherwise.
Table 2 - Reference Documents
Switch Firmware and
Firmware Update
Tools
See
http://www.mellanox.com > Support > Download Firmware Tools
Note that the Switch System described in this manual is based on Mellanox
Technologies’ InfiniScale
®
IV switch device.
Mellanox OFED
Stack for Linux
User’s Manual
See
http://www.mellanox.com > Support > InfiniBand Software and Drivers
Click “Mellanox OpenFabrics Enterprise Distribution for Linux
(MLNX_OFED)”
Select the Linux User’s Manual
The embedded OS and tools on the CPU in the management module is a
subset of the Mellanox OFED stack.
Mellanox Firmware
Tools (MFT)
User’s Manual
Document # 2329
The MFT (Mellanox Firmware Tools) package is a set of firmware tools.
The manual supplied with this package provides an overview of the firm-
ware its installation and replacement. The MFT can be downloaded with its
documentation at:
http://www.mellanox.com > Support > Download Firmware Tools

QDR InfiniBand Switch Platform User Manual Rev 1.2
Mellanox Technologies
9
Mellanox Part Numbering Legend
This symbol indicates information that is helpful to the user.
BEWARE! This symbol indicates a situation that can potentially cause personal injury
or damage to hardware or software.
Place Field Decoder
M
Mellanox Technologies
IS
System Type InfiniScale Switch
50
Model Family
FF
Form factor 22 = 8 Ports
-
Separator
#
Power supplies 1 = 1 power supply
M
Depth of the Unit B = short depth
Y
Air Flow direc-
tion R= Connector side to rear side airflow F= Rear side to connector side airflow
R
RoHS R=RoHS6

Overview
Rev 1.2
Mellanox Technologies
10
1Overview
This User Manual provides an overview of the InfiniBand switch product, an 8-port switch capa-
ble of up to 640Gb/s of non-blocking switching capacity. Based on the fourth generation InfiniS-
cale IV InfiniBand switch device. The platform delivers industry leading bandwidth, latency, and
scalability in a half width 1U form factor. The platform can be tiered to build larger multi stage
networks in a variety of topologies such as Fat-Tree, Mesh, 3D Torus etc. All port configurations
are covered in this manual which include switches for QSFP InfiniBand ports.
Mellanox IS5022 switch systems are an ideal choice for smaller departmental or back-end cluster-
ing uses with high-performance needs, such as storage, data base and GPGPU clusters. Networks
built with IS5022 systems can carry converged traffic with the combination of assured bandwidth
and granular quality of service. IS5022 systems provide up to 40Gb/s full bidirectional bandwidth
per port.
The switch platform comes pre-installed with all necessary firmware and is configured for stan-
dard operation within an InfiniBand fabric. This switch requires an InfiniBand compliant Subnet
Manager running from one of the hosts or another managed switch. All that is required for normal
operation is to follow the usual precautions for installation and to connect the switch to the HCAs
or other switches within the fabric. Once connected, the Subnet Management software automati-
cally configures and begins utilizing the switch.
It is recommended that the Mellanox OpenFabrics software package be installed on all nodes con-
nected to the IS5022. The software package provides a subnet manager and network management
tools as well as connectivity software for servers and storage, and is available on the Mellanox
web site. See Chapter 3 for more information.
Basic installation and hardware maintenance is covered in “Hardware Basic Operation and Instal-
lation” on page 15.
1.1 Serial Number and Product Version Information
The Serial number and GUID for the switch are found on the front panel below the Mellanox logo.
The serial number and product version information are found on the label seen in the figure below.
Figure 1: Generic Product label
Also on this label is the GUID identifier for the switch.

QDR InfiniBand Switch Platform User Manual Rev 1.2
Mellanox Technologies
11
2 Installation Safety Warnings
Warnings in French can be found on page 44. Warnings in German can be found on page 46.
Warnings in Spanish can be found on page 48.
1. Installation Instructions
2. Bodily Injury Due to Weight
3. Over-temperature
4. Stacking the Chassis
5. During Lightning - Electrical Hazard
Read all installation instructions before connecting the equipment to the power
source.
Use enough people to safely lift this product.
This equipment should not be operated in an area with an ambient temperature
exceeding the maximum recommended: 45°C (113°F). Moreover, to guarantee
proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation
openings.
The chassis should not be stacked on any other equipment. If the chassis falls, it
can cause bodily injury and equipment damage.
During periods of lightning activity, do not work on the equipment or connect or
disconnect cables.
<40 lbs
<18 kgs 40 - 70 lbs
18 - 32 kgs 70 - 121 lbs
32 - 55 kgs >121 lbs >55 kgs

Installation Safety Warnings
Rev 1.2
Mellanox Technologies
12
6. Copper InfiniBand Cable Connecting/Disconnecting
7. Rack Mounting and Servicing
8. Equipment Installation
9. Equipment Disposal
10. Local and National Electrical Codes
11. UL Listed and CSA Certified Power Supply Cord
Copper InfiniBand cables are heavy and not flexible, as such they should be
carefully attached to or detached from the connectors. Refer to the cable manu-
facturer for special warnings/instructions.
When this product is mounted or serviced in a rack, special precautions must be
taken to ensure that the system remains stable. In general you should fill the
rack with equipment starting from the bottom to the top.
This equipment should be installed, replaced, and/or serviced only by trained
and qualified personnel.
Disposal of this equipment should be in accordance to all national laws and reg-
ulations.
This equipment should be installed in compliance with local and national elec-
trical codes.
For North American power connection, select a power supply cord that is UL Listed
and CSA Certified, 3 - conductor, [16 AWG], terminated with a molded plug rated at
125 V, [13 A], with a minimum length of 1.5m [six feet] but no longer than 4.5m.
For European connection, select a power supply cord that is internationally
harmonized and marked “<HAR>”, 3 - conductor, minimum 1.0 mm 2 wire,
rated at 300 V, with a PVC insulated jacket. The cord must have a molded plug
rated at 250 V, 10 A.

QDR InfiniBand Switch Platform User Manual Rev 1.2
Mellanox Technologies
13
12. Add GND connection information
13. Installation codes
14. WEEE Directive
Before connecting this device to the power line, the protective earth terminal screws of
this device must be connected to the protective earth in the building installation.
(GND Connection Information):
The building installation shall provide a means for connection to protective earth; and
the equipment shall to be connected to that means permanently by a service person.
A SERVICE PERSON shall check whether or not the socket - outlet from which the
equipment is to be powered provides a connection to the building protective earth. If
not, the SERVICE PERSON shall arrange for the installation of a PROTECTIVE
EARTHING CONDUCTOR from the separate protective earthing terminal to the pro-
tective earth wire in the building. The equipment shall be installed in area where equi-
potential bonding exists (such as a telecommunication centre or a dedicated computer
room)
This device must be installed according to the latest version of the country national
electrical codes. For North America, equipment must be installed in accordance to the
applicable requirements in the US National Electrical Code and the Canadian Electri-
cal Code.
According to the WEEE Directive 2002/96/EC, all waste electrical and elec-
tronic equipment (EEE) should be collected separately and not disposed of with
regular household waste.
Dispose of this product and all of its parts in a responsible and environmentally
friendly way.

Externally Managed Switches
Rev 1.2
Mellanox Technologies
14
3 Externally Managed Switches
Externally managed (unmanaged) switches are plug and play out of the box. All switches come
with the latest firmware (FW) burned on the Flash. Updating the FW on externally managed
switches can be performed in-band only. When new FW is available, an email notification is sent
to registered users with the link to the Mellanox FW download site. The download site has the
Mellanox FW tool package and full instructions for updating FW.
Externally managed switches must be managed by management software installed on a node or on
another managed switch that can be anywhere in the fabric. This can be OpenSM, Mellanox
FabricIT™ EFM, Mellanox UFM™, or a third party Subnet Manager.

QDR InfiniBand Switch Platform User Manual Rev 1.2
Mellanox Technologies
15
4 Hardware Basic Operation and Installation
4.1 Switch Platform Hardware Overview
Figure 2 shows the front view of the switch. The figure shows port configurations for the switch
systems, I2C connector, and various status LEDs. Externally Managed switches come with an I2C
RJ45 connector.
Figure 2: Switch System Front Panel
All InfiniBand connectivity is via the connector side panel. All connectors can support active
cables.
4.1.1 LED Assignments
4.1.1.1 System Status Indicators
The System Status Indicators are located on the left side of the front (connector side) panel. The
system status indicators should display as follows:
When the switch is plugged in, within three seconds:
• the Status LED should light up green.
• The Fan LED should light up green.
• The Bad Port LED should be off.
• The Unit ID LED should be off.
UID

Hardware Basic Operation and Installation
Rev 1.2
Mellanox Technologies
16
Figure 3: Status LEDs
4.1.1.2 Status LED
The Status indicator is located on the left side of the front panel (connector side) of the unit. The
following status conditions are possible:
Table 3 - Switch Status LED Configurations
If the Status LED shows red after 30 seconds unplugthe switch and call your Mellanox
representative for assistance.
If the Fan LED shows red unplug the switch and call your Mellanox representative for
assistance.
If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes
and replug in the switch. For more information See “Troubleshooting” on page 33.
LED Description
Off No power to the switch
Solid Green Switch is running
All OK
Red There is a problem with power output from
the power supply, or there is a thermal shut
down.

QDR InfiniBand Switch Platform User Manual Rev 1.2
Mellanox Technologies
17
4.1.1.3 Fan Indicators
The Fan indicator is located on the left side of the front panel (connector side) of the unit. The fol-
lowing fan status conditions are possible:
Table 4 - Fan LED Configurations
4.1.1.4 Bad Port LED
The Bad Port indicator is located on the left side of the front panel (connector side) of the unit. The
following Bad Port conditions are possible:
Table 5 - Bad Port LED Configurations
This LED shows symbol errors. Possible causes for this are:
•badcable
• bad connection
• bad connector
This LED lights up when one or more ports is receiving a symbol error. The LED immediately
goes off until the next symbol error is received.
LED Configuration FAN LED
Off There is no power to the fans.
Green OK – The fans are running.
Red Error – fans are not operating properly. Replace the switch.
Fans must be operating while the power supply is plugged in.
If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes
and replug in the switch. For more information See “Troubleshooting” on page 33.
LED Configuration Description
Off OK – All ports are up and running.
Flashing Orange Error –One or possibly more ports has just received a symbol error.

Hardware Basic Operation and Installation
Rev 1.2
Mellanox Technologies
18
4.1.1.5 UID LED Switch Identifier
The UID LED is a debug feature that will become available to customers in the near future. For
details please contact Mellanox Technologies support.
4.1.1.6 Port Connector LED
Above the ports are two LEDs one for the upper port and one for the lower port . The fol-
lowing table shows the port status according to the LED indication.
Table 6 - Connector Physical and Logical Link Indications
This LED when flashing orange shows port physical errors. Possible causes for this are:
•badcable
• bad connection
• bad connector
•badcage
4.2 Air Flow
These switches can come with two air flow patterns. The two patterns are
• Rear side inlet to connector side outlet
• Connector side inlet to rear side outlet
The air flow is specified in the product model number. See “Mellanox Part Numbering Legend” on
page 9.
4.3 QSFP Cable Power Budget Classification
All IS5022 QSFP switches are designed for active cables with a max power per module of 2.0W.
This is power level 2 according to the QSFP Public Specification.
LED Description
Off No link
Orange Physical Link up
Solid Green Logical Link up
Flashing Green Data activity
flashing speed ≈data transfer speed
Flashing Orange A problem with the Physical Link

QDR InfiniBand Switch Platform User Manual Rev 1.2
Mellanox Technologies
19
4.4 Interfaces
4.4.1 Port Connector Interfaces
The Connector side of the switch has 8 QSFP ports. These are placed in two rows, 4 ports to a row.
The ports are labelled as shown in Figure 4. The bottom row ports are flipped from the top row.
See Figure 5.
Figure 4: Port Numbering
Figure 5: Top and Bottom Ports
4.5 Management and FW Updating Interfaces
This switch is externally managed. There is no CPU in this switch for management purposes. The
switch can be managed through any fabric management software such as Open SM Mellanox
UFM.
There is one interface to connect to the switch. It is an I2C RJ45 interface labelled “I2C”. Use this
connector to update firmware (as a last resort) and for advanced debug.
4.5.1 RJ-45 Connector (I2C)
There is an “I2C”interface on the front panel.
Update FW inband only.
1
2
3
4
5
6
7
8

Hardware Basic Operation and Installation
Rev 1.2
Mellanox Technologies
20
Figure 6: RJ45 I2C Connector
All firmware updates should be done in-band using Mellanox Firmware Management Tools.
The I2C connector is available only on externally managed switches and is used to update firm-
ware should in-band firmware update not work. You will need to order an MTUSB-1 USB to I2C
adapter to make use of the I2C interface. The I2C connection provides access to Flash. This is for
Advanced users and FAEs only.
The externally managed switches are Plug and Play and all firmware updates should be
done in-band. The I2C connection should only be used if the FW image was corrupted
to the point that the regular FW tools cannot successfully reburn the correct image.
Warning: Any red status LED is cause for concern and must be dealt with imme-
diately.
It can take up to 30 seconds to boot up, during which time the status LED may
indicate red.