Dell SwitchX M4001F User manual

www.mellanox.com
Thirdpartyinformationbrought
to you courtesy of Dell.
SwitchX® InfiniBand Blade Switch I/O Module
for PowerEdge M-Series User Manual
PN: M4001F, M4001T, M4001Q
Rev 1.2

Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
Beit Mellanox
PO Box 586 Yokneam 20692
Israel
Tel: [phone] ; +972
(0)74 723 7200
Fax: [phone]
© Copyright 2012.
Mellanox, Mellanox logo, BridgeX, ConnectX, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, PhyX, SwitchX, Virtual Protocol
Interconnect and Voltaire are registered trademarks of Mellanox Technologies, Ltd. FabricIT, MLNX-OS, Unbreakable-Link, UFM and
Unified Fabric Manager are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
InfiniBand Blade Switch I/O Module for PowerEdge M-Series
Document Number: 3513
Rev 1.2
Mellanox Technologies
2
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PRO-
VIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF
AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUS-
TOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES
TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CAN-
NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY.
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MER-
CHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT
SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEM-
PLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCURE-
MENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED
DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

InfiniBand Blade Switch I/O Module for PowerEdge M-Series Rev 1.2
Mellanox Technologies 3
Contents
Contents 3
Revision History 4
About this Manual 5
Chapter 1 Overview 7
1.1 InfiniBand Connectors 8
1.2 Switch Status Lights 9
1.3 I2C Interface RJ-45 Connector 10
Chapter 2 Installation and Basic Operation 11
2.1 Installation Safety Warnings 11
2.2 Mechanical Installation 13
2.2.1 Removing an Old Switch From the Chassis 13
2.2.2 Removing an I/O Module Blank From the Chassis 14
2.2.3 Installing the Mellanox M4001[F, T, Q] Switch Into the Dell Chassis 14
2.3 Power Connections and Initial Power On 14
2.4 InfiniBand Cable Installation 14
2.4.1 Supported Approved Cables 15
Chapter 3 Cluster Management and Firmware 16
3.1 Network Management and Clustering Software 16
3.2 Updating Firmware 16
3.2.1 Instructions for Reprogramming Over the InfiniBand Network 16
Chapter 4 Troubleshooting 17
Appendix A Specifications 18
Appendix B EMC Certification Statements 20
Appendix C QSFP Interface 22
Appendix D Avertissements de sécurité d’installation 24
Appendix E Installation - Sicherheitshinweise 26

Rev 1.2
Mellanox Technologies
4
Revision History
Table 1 - Revision History Table
Date Revision Description
June 2012 1.2 Added graphics to section 2.4
Added section2.4.1
May 2012 1.1 Added M4001T switch
November 2011 1.0 Initial release

InfiniBand Blade Switch I/O Module for PowerEdge M-Series Rev 1.2
Mellanox Technologies 5
About this Manual
This manual describes the installation and basic use of the SwitchX® M4001F(FDR) 56 Gb/s,
M4001T(FDR10) 40Gb/s and the M4001Q(QDR) 40 Gb/s InfiniBand blade switch I/O modules
for the PowerEdge M-Series chassis.
Intended Audience
This manual is intended for users and system administrators responsible for installing and setting
up the switch platforms listed above.
The manual assumes familiarity with the InfiniBand® Architecture Specification.
Related Documentation
The documentation set accompanying the Mellanox M4001[F, T, Q] includes the following:
Online Resources
• Mellanox Technologies Web pages: http://www.mellanox.com
• Dell Support Web pages: http://support.dell.com
Conventions
Throughout this manual, the names M4001F, M4001T, and M4001Q and the terms switch, I/O
module, and IOM are used to describe the M4001F FDR, M4001T FDR10, and the M4001Q QDR
InfiniBand blade switch I/O modules, unless explicitly indicated otherwise.
Table 2 - Reference Documents
InfiniBand Architecture Specification
Volume 1 Release 1.2.1 and Volume 2
release 1.2.1
InfiniBand architecture specification descriptions
Mellanox Firmware Tools (MFT) 
User’s Manual 
Document # 2329
The MFT (Mellanox Firmware Tools) package is a set of firmware tools. The
manual supplied with this package provides an overview of the firmware its
installation and replacement. The MFT can be downloaded with its documenta-
tion at: 
http://www.mellanox.com > Downloads > Firmware Tools.

Rev 1.2
Mellanox Technologies
6
The following pictures are used throughout this document to indicate information that is important
to the user.
This symbol makes recommendations to the user.
This symbol indicates information that is helpful to the user.
This symbol indicates a situation that can potentially cause damage to hardware or
software.
BEWARE! This symbol indicates a situation that can potentially cause personal injury
or damage to hardware or software.

InfiniBand Blade Switch I/O Module for PowerEdge M-Series Rev 1.2
Mellanox Technologies 7
1 Overview
The SwitchX® InfiniBand blade switch I/O module for the PowerEdge M-Series chassis provides
a high bandwidth, low latency fabric for Enterprise Data Centers (EDC), High-Performance Com-
puting (HPC), and Embedded environments. Based on the 5th generation InfiniBand switch device
(1st Generation SwitchX). This I/O module (IOM) delivers up to 56Gb/s at FDR speed and 
40 Gb/s at FDR10 and QDR speed full bisectional bandwidth per port. When used in conjunction
with ConnectX®-3 InfiniBand dual port Mezzanine I/O cards, clustered data bases, parallelized
applications and transactional services applications these switches will achieve significant perfor-
mance improvements resulting in reduced completion time and lower cost per operation.
The I/O module comes pre-installed with all necessary firmware, and configured for standard
operation within an InfiniBand fabric, and only requires an InfiniBand compliant Subnet Manager
running from one of the hosts. All that is required for normal operation is to follow the usual pre-
cautions for installation and connection from the switch to the HCAs or other switches. Once con-
nected, the Subnet Management software automatically configures the switch and begins utilizing
the switch.
It is recommended that Mellanox OpenFabrics software package be installed on all nodes con-
nected to the M4001[F, T, Q]. The software package provides a subnet manager and network man-
agement tools as well as connectivity software for servers and storage, and is available on the
Mellanox web site. See Chapter 3 for more information.
Basic installation, hot-swapping components and hardware maintenance is covered in “Installation
and Basic Operation” on page 11.
The M4001F, M4001T, and M4001Q switches have a Hot Swap controller and a PSOC Manage-
ment IC.

OverviewRev 1.2
Mellanox Technologies
8
Figure 1: Switch Front Panel and Locking Mechanism
1.1 InfiniBand Connectors
This switch has 16 QSFP connectors through the front panel. The remaining 16 interfaces are
through the Midplane Connector out of the back of the switch. Figure 1 shows the front 16 ports.
Each of the InfiniBand ports has two LEDs located between the connectors and the I2C interface.
The green LED, when lit, indicates that a valid physical connection to the other system (switch or
HCA port) exists. See Figure 2,“Physical and Logical Link Indication LEDs” for more details.
Locking Mechanism
Switch Status Lights
Port LEDs

InfiniBand Blade Switch I/O Module for PowerEdge M-Series Rev 1.2
Mellanox Technologies 9
Figure 2: Physical and Logical Link Indication LEDs
1.2 Switch Status Lights
The switch Status lights indicate whether the switch is receiving power from the chassis, and the
state of the switch.
Figure 3: Indicator LEDs
Table 3 - IOM states and LED configurations:
LED Switch Status
Indication Green Status
OFF OFF OFF
ON OFF Boot in Progress
Switch not ready
LED Name Connection Status
Physical Link - Green Off – No Physical Link
ON – Physical Link
Data Activity - Yellow
Blinking – indicates Data Transfer
Constant on – indicates Link exists
with no Data Transfer taking place
Off with green LED lit – indicates that
the Subnet Manager may not be running
The IO Module is on and
ready when both the blue
and green LEDs are lit.
This LED can be blue or
amber. A fault is indicated
when the amber LED is
blinking. See Table 3.

OverviewRev 1.2
Mellanox Technologies
10
1.3 I2C Interface RJ-45 Connector
There is an “I2C”interface on the front panel.
Figure 4: I2C RJ-45 Interface
The I2C connection provides access to Flash and EEPROMs. The I2C connector is used to update
firmware should in-band firmware update not work. You will need to order an RJ-45 to DB9 I2C
adapter to make use of the I2C interface. You can order this adapter from Mellanox Technologies.
The Ordering Part Number is MTUSB-1.
These switches are Plug and Play and all firmware updates should be done in-band. The I2C con-
nection should only be used if the FW image was corrupted to the point that the regular FW tools
cannot successfully reburn the correct image.
Blinking Blue
ON BLINKING BLUE The CMC is identifying
the newly installed switch
ON ON BLUE Switch is on and operating Normally
ON ON or BLINKING
AMBER Fault in System
Self-diagnosed
OFF ON or BLINKING
AMBER Fault in System
CMC-detected
All firmware updates should be done in-band using Mellanox Firmware Management
Tools. This is for Advanced users and FAEs only.
LED Switch Status
Indication Green Status

InfiniBand Blade Switch I/O Module for PowerEdge M-Series Rev 1.2
Mellanox Technologies 11
2Installation and Basic Operation
Installation and initialization of the I/O module are straightforward processes, requiring attention
to the normal mechanical, power, and thermal precautions for rack-mounted equipment. The I/O
module does not require any programming or configuration to operate as a basic InfiniBand switch
and includes all of the necessary functionality to operate with external standard InfiniBand Subnet
Management software.
This section describes the installation process and basic operation of the I/O module. Please first
read the warnings sub-section carefully before carrying on with installation procedures.
2.1 Installation Safety Warnings
1. Installation Instructions
2. Over-temperature
3. Stacking the Chassis
4. During Lightning - Electrical Hazard
Suitable electrical, mechanical and fire enclosure shall be provided by the end user.
Read all installation instructions before connecting the equipment to the power source.
This equipment should not be operated in an area with an ambient temperature exceed-
ing the maximum recommended: 40°C (104°F). Moreover, to guarantee proper air
flow, allow at least 8cm (3 inches) of clearance around the ventilation openings.
The chassis should not be stacked on any other equipment. If the chassis falls, it can
cause bodily injury and equipment damage.
During periods of lightning activity, do not work on the equipment or connect or dis-
connect cables.

Installation and Basic OperationRev 1.2
Mellanox Technologies
12
5. Copper InfiniBand Cable Connecting/Disconnecting
6. Rack Mounting and Servicing
7. Equipment Installation
8. Equipment Disposal
9. Local and National Electrical Codes
Copper InfiniBand cables are heavy and not flexible, as such they should be carefully
attached to or detached from the connectors. Support the cable’s weight using the rack
and a cable management system. Refer to the cable manufacturer for special warnings
and instructions.
When this product is mounted or serviced in a rack, special precautions must be taken
to ensure that the system remains stable. In general you should fill the rack with equip-
ment starting from the bottom to the top.
This equipment should be installed, replaced, or serviced only by trained and qualified
personnel.
Disposal of this equipment should be in accordance to all national laws and regula-
tions.
This equipment should be installed in compliance with local and national electrical
codes.

InfiniBand Blade Switch I/O Module for PowerEdge M-Series Rev 1.2
Mellanox Technologies 13
2.2 Mechanical Installation
These switches are hot pluggable. It is not necessary to power down the Dell Chassis to install a
new switch or to replace an old switch with a new one.
Figure 5: Rear View of the Dell PowerEdge M1000e Chassis
The switch can only be placed in channel slots B1, C1, C2 or B2.
2.2.1 Removing an Old Switch From the Chassis
1. Remove any locking cables or screws that secure the old switch into the chassis.
2. Disconnect all of the InfiniBand cables from the front of the switch to be removed.
3. Unlock the switch from the chassis by pushing the red latch release button.
4. Pull the locking arm down to a position perpendicular to the front of the chassis.
5. Pull the switch out of the chassis using the locking arm.
M4001[F,T, Q] blades are not allowed in the Fabric A slots.
Latch Release Buttons
Locking Arms
C2 B2 A2
Channel 2 slots
A1 B1 C1
Channel 1 slots

Installation and Basic OperationRev 1.2
Mellanox Technologies
14
2.2.2 Removing an I/O Module Blank From the Chassis
1. Unlock the I/O Module Blank by pushing the red latch release button.
2. Pull the locking arm down to a position perpendicular to the front of the chassis.
3. Pull the I/O Module Blank out of the chassis using the locking arm.
2.2.3 Installing the Mellanox M4001[F, T, Q] Switch Into the Dell Chassis
Refer to the Dell PowerEdge M1000e Hardware Owners Manual for more information.
1. Follow the instructions in Section 2.2.1 or Section 2.2.2 to remove an old switch or an I/O
Module Blank.
2. On the new switch, push the red latch release button.
3. Pull the lever forward until the lever is perpendicular to the front panel.
4. Push the switch into the slot until the latching mechanism is against the bar.
5. Push the lever on the latching mechanism up, making sure that the latching mechanism
catches the locking bar. The lever should now be parallel to the front panel.
6. Check the indicator lights to make sure the switch has power.
2.3 Power Connections and Initial Power On
The I/O module will automatically power up when AC power is applied. There is no power switch.
Immediately upon closing the latching mechanism check to make sure that the green switch LED
is lit.
2.4 InfiniBand Cable Installation
All cables can be inserted or removed with the unit powered on. To insert a cable, press the con-
nector into the port receptacle until the connector is firmly seated. The green LED indicator
accompanying each port will light when the physical connection is established (that is, when the
unit is powered on and a cable is plugged into the port with the other end of the connector plugged
into a functioning port). After plugging in a cable, lock the connector using the latching mecha-
nism particular to the cable vendor. The yellow LED will light if the subnet manager is running
Make sure the rack is stable on a solid floor and that the rack is filled from the bottom
up. This will keep the center of gravity as low as possible reducing risk of tipping.
The rack mounting is designed to fit the PowerEdge Chassis. Take precautions to guar-
antee proper ventilation for air intake at the front of the chassis and exhaust at the rear
in order to maintain good airflow at ambient temperature. Cable routing in particular
should not impede the air exhaust from the chassis.

InfiniBand Blade Switch I/O Module for PowerEdge M-Series Rev 1.2
Mellanox Technologies 15
(non blinking indicating that no data is being transferred yet). When a logical connection is made
the yellow LED will blink signifying data is being transferred.
Figure 6: Belly to Belly Cable Insertion
To remove, disengage the locks and slowly pull the connector away from the port receptacle. Both
LED indicators will turn off when the cable is unseated.
Care should be taken not to impede the air exhaust flow through the chassis.
2.4.1 Supported Approved Cables
For a list of all approved cables see:
www.mellanox.com/related-docs/user_manuals/Mellanox_approved_cables.pdf
Cables in theBelly to Belly right side should be inserted up side down in relation to the
how the cables are inserted in the left side.
Cable lengths should be used which allow for routing horizontally around to the side of
the chassis before bending upward or downward in the rack.

Cluster Management and FirmwareRev 1.2
Mellanox Technologies
16
3Cluster Management and Firmware
3.1 Network Management and Clustering Software
Download and install, on all nodes, the Mellanox OpenFabric software package for Linux, Windows, or
other operating systems from the Mellanox software website:
http://www.mellanox.com >Downloads > InfiniBand SW/Drivers.
This software package provides connectivity for server and storage systems utilizing High Perfor-
mance Computing (HPC) or enterprise data center (EDC) applications across an InfiniBand fabric.
It also provides a Subnet Manager for simple network configuration, network administration, and
diagnostic tools for network management.
3.2 Updating Firmware
The switch is delivered with the latest Firmware available at the time of production. New firmware
versions will be posted on the Mellanox firmware download page. Firmware can be updated in-
band using the Mellanox Firmware Tools (MFT).
You will need the Mellanox Firmware Tools package available in MLNX_OFED to update firm-
ware for this switch. It also can be downloaded from:
http://www.mellanox.com > Downloads > Firmware Tools.
The latest firmware can be found at:
http://www.mellanox.com > Support > Dell.
3.2.1 Instructions for Reprogramming Over the InfiniBand Network
To reprogram the switch see http://mellanox.com/content/pages.php?pg=firmware_table_dell
for the latest firmware and the burning instructions.

InfiniBand Blade Switch I/O Module for PowerEdge M-Series Rev 1.2
Mellanox Technologies 17
4 Troubleshooting
As soon as a switch is inserted in make sure that the power LED comes on.
The power LED for the switch does not come on:
1. Check that the chassis has power.
2. Remove and reinstall the switch.
The status LED for the switch is blinking amber:
1. Remove the switch from the chassis and re-insert it (verify that the switch is all the way in
the chassis and the lever is firmly closed).
2. If the amber LED continues to blink, replace the switch.
The link LED for the InfiniBand connector does not come on:
1. Check that both ends of the cable are connected.
2. Check that the locks on the ends are secured.
3. Make sure that the latest FW version is installed on both the Mezzanine I/O card and the
switch.
4. Make sure that at least one blade has a matching Mezzanine I/O card installed to support data
flow.
5. If media adapters are used check that the all connections are good, tight, and secure.
The activity LED does not come on:
Check that the Subnet Manager has been started.
The power LED for the switch shuts off:
1. Check that the there is adequate ventilation.
2. Make sure that there is nothing blocking the front or rear ventilation openings of the chassis.

Rev 1.2
Mellanox Technologies
18
Appendix A: Specifications
Table 4 - M4001[F, T, Q] Specification Data
Physical Power and Environmental
H x W x D:
Weight:
Mounting:
SerDes Speeds
Connectors:
12.2 x 1.34 x 10.7 inches
310.7 x 34.1 x 271.8 mm
2.6Kg fully configured
Vertically mounted rack
10, 20, 40, 56 Gb/s per port
QSFP
16 external QSFP connectors
16 internal backplane connectors
Typ Power:
FDR
Passive:
Active:
FDR10
Passive:
Active:
QDR
Passive:
Active:
Max Power:
FDR
Passive:
Active:
FDR10
Passive:
Active:
QDR
Passive:
Active:
Dissipated Power:
Power through
connector:
Temperature:
Humidity:
80.66W or 275.22BTUs/hr
108.26W or 369.40BTUs/hr
60.42W or 206.16BTUs/hr
88.02W or 300.34BTUs/hr
60.42W or 206.16BTUs/hr
88.02W or 300.34BTUs/hr
84.46W or 288.19BTUs/hr
112.06 or 382.36BTUs/hr
75.83W or 258.74BTUs/hr
103.43W or 352.92BTUs/hr
75.83W or 258.74BTUs/hr
103.43W or 352.92BTUs/hr
2.0 W per port maximum
0°C to 40°C
10% - 90% non-condensing

InfiniBand Blade Switch I/O Module for PowerEdge M-Series Rev 1.2
Mellanox Technologies 19
Protocol Support Regulatory Compliance
InfiniBand:
QoS:
Management:
Auto-Negotiation of 10Gb/s, 20Gb/s, or 40
Gb/s, or 56Gb/s
9 InfiniBand Virtual Lanes for all ports 
8 Data transport lanes and 1 management
lane
Baseboard, Performance, and Device man-
agement Agents for full InfiniBand In-
Band Management
Safety:
EMC:
ENVIRONMEN-
TAL
US/Canada: cTUVus
EU: IEC60950
International: CB
TUV Bauart
USA: FCC, Class A
Canada: ICES, Class A
EU: CE Mark (EN55022 Class
A, EN55024, EN61000-3-2,
EN61000-3-3)
Japan: VCCI, Class A
Korea: RRL (MIC), Class A
Australia/NewZealand:C-Tick
Class A
EU: IEC 60068-2-64: Random
Vibration
EU: IEC 60068-2-29: Shocks,
Type I / II
EU: IEC 60068-2-32: Fall Test
Scalability and Performance
Switching
Performance:
Addressing:
Switching Capacity
Simultaneous wire-speed any port to any
port
48K Unicast Addresses Max. per Subnet
16K Multicast Addresses per Subnet
2.56 Tb/s

Rev 1.2
Mellanox Technologies
20
Appendix B: EMC Certification Statements
Appendix B.1: FCC Statements (USA)
Class A Statements:
§ 15.21
Statement
Warning! Changes or modifications to this equipment not expressly approved by the party responsible for
compliance (Mellanox Technologies) could void the user's authority to operate the equipment.
§15.105(a)
Statement
NOTE: This equipment has been tested and found to comply with the limits for a Class A digital device, pur-
suant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harm-
ful interference when the equipment is operated in a commercial environment. This equipment generates,
uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instruc-
tion manual, may cause harmful interference to radio communications. Operation of this equipment in a
residential area is likely to cause harmful interference in which case the user will be required to correct the
interference at his own expense.
Appendix B.2: EN Statements (Europe)
EN55022 Class A Statement:
Warning! This is a class A product. In a domestic environment this product may cause radio 
interference, in which case the user may be required to take adequate measures.
Appendix B.3: ICES Statements (Canada)
Class A Statement:
“This Class A digital apparatus complies with Canadian ICES-003.
Cet appareil numérique de la classe A est conforme à la norme NMB-003 du Canada.”