Mellanox Technologies Vantage 8500 User manual

www.mellanox.com
Mellanox VantageTM 8500 288 Port Switch
Installation Manual
OPN
Rev A04

Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: [phone]
Fax: [phone]
Mellanox Technologies, Ltd.
PO Box 586 Hermon Building
Yokneam 20692
Israel
Tel: [phone]
Fax: [phone]
© Copyright 2011. Mellanox Technologies. All rights reserved.
Mellanox®, BridgeX®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniRISC®, InfiniScale®,
InfiniPCI®, PhyX® and Virtual Protocol Interconnect® are registered trademarks of Mellanox Technologies, Ltd. CORE-
Direct and FabricIT are trademarks of Mellanox Technologies, Ltd. All other marks and names mentioned herein may be
trademarks of their respective companies.
Mellanox VantageTM 8500 288 Port Switch Installation Manual
Document Number: DOC-00651-A04
Rev A04
Mellanox Technologies
2
MELLANOX TECHNOLOGIES, INC. AND ITS AFFILIATES (“MELLANOX”) FURNISH THIS DOCUMENT
“AS IS,” WITHOUT WARRANTY OF ANY KIND. MELLANOX DISCLAIMS ALL WARRANTIES, EXPRESS OR
IMPLIED, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT AND THOSE ARISING FROM A COURSE OF
PERFORMANCE, A COURSE OF DEALING, OR TRADE USAGE. MELLANOX SHALL NOT BE LIABLE FOR ANY
ERROR, OMISSION, DEFECT, DEFICIENCY OR NONCONFORMITY IN THIS DOCUMENT AND DISCLAIMS ALL
LIABILITY, INCLUDING LIABILITY FOR INFRINGEMENT OF ANY INTELLECTUAL PROPERTY RIGHTS
RELATED TO THE INFORMATION CONTAINED IN THIS DOCUMENT.
No license, expressed or implied, to any intellectual property rights is granted under this document. This document, as well as
the software described in it, are furnished under a separate license and shall only be used or copied in accordance with the
terms of the applicable license. The information in this document is furnished for informational use only, is subject to change
without notice, and should not be construed as any commitment by Mellanox. Except as permitted by the applicable license,
no part of this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means without
the express written consent of Mellanox.Names and logos identifying products of Mellanox in this document are registered
trademarks or trademarks of Mellanox. Voltaire is a registered trademark of Mellanox Technologies, Ltd. All other trademarks
mentioned in this document are the property of their respective owners.
Copyright © 2011 Mellanox Technologies, Inc. All rights reserved.

Rev A04
Mellanox Technologies 3
Table of Contents
About this Manual 8
Contact Us 9
Intended Audience 10
Document Conventions 11
Textual Conventions 12
Related Documentation 13
Chapter 1 Product Overview 14
1.1 The Vantage 8500 Switch 14
1.2 PhysicalDescription 15
1.2.1 Chassis 15
1.2.2 Plug-in Modules - Overview 16
1.3 BlockDiagram 21
1.4 ProductOptions 21
1.4.1 Basic Hardware Configuration 22
1.4.2 Full Hardware Configuration 22
1.5 TechnicalSpecifications 24
1.5.1 Chassis 24
1.5.2 Line Boards (sLB-1024) 24
1.5.3 Fabric Boards (sFB-8500) 25
1.5.4 Management Board (sMBE) 25
1.5.5 Control Panel sCTRL 25
1.5.6 Fan Modules 25
1.5.7 Power Supply Unit (sPSU-S) 26
1.5.8 Power Supply Calculation Example 27
1.5.9 IEEE Compliance 27
1.5.10 Datacenter Bridging (CEE) 28
1.5.11 Layer-2 Capabilities 28
1.5.12 Physical/Electrical 28
1.5.13 Cooling 28
1.5.14 Physical Characteristics 28
1.5.15 Environmental 28
1.5.16 Acoustic Data 29
1.5.17 Certifications 29
1.6 Reliability(MTBF) Data 30
1.7 Declarations 30
1.7.1 Declaration of Conformity 30
1.7.2 Hazardous Substances (RoHS 6) Compliance 30
1.7.3 Statement of Volatility 31
1.8 Shock and Vibration Test Report 31
Chapter 2 Installation Requirements 33
2.1 SitePlanning 33
2.2 HeatDissipation 33
2.3 Rack and Clearance Requirements 33
2.4 PowerRequirements 34
2.4.1 sPSU-S Units - Power Usage/Requirements 34
2.4.2 Power Redundancy (N:1 and N:N) 35
2.5 Required 10GbE Cables and Port Adapters 36
2.6 Required Control and Management Cables 37
2.7 RequiredInstallationTool 38
Chapter 3 Unpacking 39
3.1 Packed Unit Dimensions and Weight 39
3.2 PackageContents 39

Rev A04
Mellanox Technologies
4
3.3 UnpackingtheSwitch 40
3.4 Center of Gravity Data 43
3.5 PackingforRe-Deployment 44
Chapter 4 Chassis Installation 45
4.1 MountingProcessSummary 45
4.1.1 Assembling and Mounting the Rails 45
4.1.2 Mounting the Chassis in the Rack 53
4.1.3 Installing Cabling Guide Brackets 56
4.2 Shipping Guidelines and Instructions 57
Chapter 5 Installing Modules 59
5.1 ChassisSlotNumbering 59
5.2 ModuleInsertionRules 59
5.3 Extracting/InsertingBoards 61
5.3.1 Installing a Fabric Board (sFB-8500) 63
5.3.2 Extracting a Fabric Board (sFB-8500) 63
5.3.3 Installing a Line Board (sLB-1024) 64
5.3.4 Extracting a Line Board (sLB-1024) 64
5.3.5 Installing a Management Board (sMBE) 65
5.3.6 Extracting a Management Board (sMBE) 66
5.4 Installing a Power Supply Unit (sPSU-S) 67
5.5 Extracting a Power Supply Unit (sPSU-S) 67
5.6 Replacing a Control Board (sCTRL) 68
5.7 Replacing Fan Units (sFUs) 69
5.8 Replacing the sFU-4 Unit 70
5.9 Replacing the sFU-8E Unit 70
Chapter 6 Connector and Cable Specifications 72
6.1 Cabling-Do'sand Don'ts 72
6.2 GeneralGuidelines 72
6.3 Connecting10GbECables 72
6.4 Connecting the Unit to a Local Management Terminal 73
Chapter 7 Final Checks, Operation and Troubleshooting 75
7.1 Required Checks Before Initial Operation 75
7.2 Powering-upthe Switch 75
7.3 Verifying Proper Switch Operation 75
7.4 Checking the LED Indicators 75
7.4.1 Line Board (sLB-1024) LED Indicators 76
7.4.2 Fabric Board (sFB-8500) LED Indicators 76
7.5 Power Supply Unit (sPSU-S) LED Indicators 77
7.5.1 Fan Unit LED Indicators 78
7.5.2 Management Board (sMBE) LED Indicators 78
7.5.3 Control Board (sCTRL) LED Indicators 79
Appendix A Port Pinouts 81
RJ-45 Management Port 82
DB-9 Serial Port 83

Rev A04
Mellanox Technologies 5
List of Figures
Figure 1: Vantage 8500 Switches in a Scale-Out Ethernet Configuration 14
Figure 2: Vantage 8500 Switch -Front View 15
Figure 3: Vantage 8500 Switch - 3D Rear View 16
Figure 4: Line Board (sLB-1024) 17
Figure 5: Fabric Board (sFB-8500) 18
Figure 6: Management Board (sMBE) 19
Figure 7: Power Supply Unit (sPSU-S) 19
Figure 8: Vertical Fan Unit (sFU-4) 20
Figure 9: Horizontal Fan Unit (sFU-8E) Front Panel 20
Figure 10: Control Board (sCTRL) Front Panel 21
Figure 11: Vantage 8500 Block Diagram 21
Figure 12: Vantage 8500 Switch -Front View 23
Figure 13: Vantage 8500 Rear View in Full Hardware Configuration 24
Figure 14: Fully Configured Switch at N:1 Power Redundancy 36
Figure 15: Unpacking Procedure for the Vantage 8500 Switch 40
Figure 16: Unpacking the Chassis 41
Figure 17: Chassis Partially Unpacked 42
Figure 18: Center of Gravity Data 43
Figure 19: sPSU-S Side Label on the Foam sPSU-S 44
Figure 20: Rail Kit Components - Left Bracket, Right Bracket, Screws, Washers and Nuts 46
Figure 21: Assembling the Rail Bracket and Insert 47
Figure 22: Positioning the Insert In Front of the Right Bracket 48
Figure 23: Assembling an Insert to a Rail Bracket (Inside View) 48
Figure 24: U Alignment 49
Figure 25: Above U Alignment 50
Figure 26: Mounting the Rail - Left Side Front and Rear View 50
Figure 27: Chassis Grounding Holes 52
Figure 28: Top Grounding Connection (Grounding Stud Installed) 52
Figure 29: Grounding Stud Installation 53
Figure 30: Mounting a Basic Vantage 8500 Switch Chassis in a Rack 55
Figure 31: Chassis Secured onto the Front Rack Rail 56
Figure 32: Cabling Guide Brackets 56
Figure 33: Screws, 3/8" Tooth Washer, and Flat Washers 57
Figure 34: Screws on Chassis Side Wall 58
Figure 35: Chassis Slot Numbering for sLB-1024 Modules 59

Rev A04
Mellanox Technologies
6
Figure 36: Fabric Board Slot Numbering 60
Figure 37: Using Ejectors to Extract a Board 62
Figure 38: Extracting an sFB-8500 Board 64
Figure 39: Extracting an sLB-1024 Board 65
Figure 40: Securing the sMBE Board 66
Figure 41: Extracting an sMBE Board 67
Figure 42: Extracting a sPSU-S Unit 68
Figure 43: Removing the sCTRL Board 69
Figure 44: Securing the sCTRL Board 69
Figure 45: Securing the sFU-4 Board 70
Figure 46: Removing the sFU-8E Unit 71
Figure 47: Securing the sFU-8E Unit 71
Figure 48: Line Board (sLB-1024) Front Panel 76
Figure 49: Fabric Board (sFB-8500) Front Panel 76
Figure 50: Power Supply Unit (sPSU-S) Front Panel 77
Figure 51: Horizontal Fan Unit (sFU-8E) Front Panel 78
Figure 52: Management Board (sMBE) Front Panel 79
Figure 53: Control Board (sCTRL) Front Panel 80

Rev A04
Mellanox Technologies 7
List of Tables
Table 1: Conventions 12
Table 2: Vantage 8500 Switch - Part Naming Conventions 16
Table 3: Vantage 8500 Switch Line Board Components 17
Table 4: Vantage 8500 Basic Hardware Configuration 22
Table 5: Vantage 8500 Full Hardware Configuration 22
Table 6: Power Calculation Table 27
Table 7: MTBF Information per Module 30
Table 8: Axes (X,Y,Z) Vibration Test for Transportation without a Rack 31
Table 9: Drop Tests 32
Table 10: Rack Clearance Requirements 34
Table 11: sPSU-S - Detailed Power Usage Data 35
Table 12: SFP+ Copper 10GbE Cables 36
Table 13: SFP+ Fiber Optic 10GbE Transceivers (Required for Fiber Optic Cables) 37
Table 14: Fiber Optic Cables (Require SFP+ Transceivers) 37
Table 15: Tools Required for Hardware Installation 38
Table 16: Factory-Packed Unit Weight and Dimensions 39
Table 17: Center of Gravity Data 43
Table 18: Fabric Board Insertion 59
Table 19: Line Board (sLB-1024) LED Indicators 76
Table 20: Fabric Board (sFB-8500) LED Indicators 77
Table 21: Power Supply Unit (sPSU-S) LED Indicators 78
Table 22: Fan Unit LED Indicators 78
Table 23: Management Board Indicators 79
Table 24: Control Board LED Indicators 80
Table 25: RJ-45 Management Port Pinout 82
Table 26: DB-9 RS-232 CLI Connector Pinout 83

Rev A04
Mellanox Technologies
8
About this Manual
This manual describes how to install the Vantage 8500.
Please refer to the official and latest product release notes for last-minute updates.
Technical support may be obtained directly from:
• Your regional distributor from whom this product was ordered
• Your OEM customer representative
For further information and assistance, go to http://www.Mellanox.com/support.htm.
We sometimes update the printed and electronic documentation after original publica-
tion. Therefore, review the documentation on http://www.Mellanox.com for any
updates.

Rev A04
Mellanox Technologies 9
Contact Us
Please send your documentation-related comments and feedback or report mistakes to docs@Mel-
lanox.com.
We are committed to improving our documentation and your feedback is important to us.

Rev A04
Mellanox Technologies
10
Intended Audience
This manual is for system administrators responsible for installing the Mellanox Vantage 8500
10GbE Layer-2 core switch, hereafter referred to as Vantage 8500. We assume you are familiar
with the concepts and terminology of Ethernet and local area networking.

Rev A04
Mellanox Technologies 11
Document Conventions
The following lists conventions used in this document.
Identifies important information that contains helpful suggestions.
Alerts you to the risk of system damage, or loss of data.
Warns you that failure to take or avoid a specific action might result in personal
injury or a malfunction of the hardware or software. Be aware of the hazards
involved with electrical circuitry and be familiar with standard practices for pre-
venting accidents before you work on any equipment.

Rev A04
Mellanox Technologies
12
Textual Conventions
The following table describes typographical conventions in Mellanox documentation. All terms
refer to isolated terms within body text or regular table text unless otherwise mentioned in the
Notes column.
Table 1 - Conventions
Term, Construct, Text
Block Example Notes
File name, pathname /opt/ufm/conf/gv.cfg
Console session (code) -> flashClear <CR> Complete sample line or block. Comprises both input and
output. The code can also be shaded.
Linux shell prompt #The “#”character stands for the Linux shell prompt.
Mellanox CLI Guest
Mode Switch > Mellanox CLI Guest Mode
Mellanox CLI admin
mode Switch # Mellanox CLI admin mode
String < > or [] Strings in < > or [ ] are descriptions of what will actually
be shown on the screen, for example, the contents of
<your ip> could be 192.168.1.1
Management GUI
label, item name New Network,
New Environment Management GUI
labels and item names
appear in bold,
whether or not the
name is explicitly dis-
played (for example,
buttons and icons).
User text entered into
Manager, e.g., to assign
as the name of a logical
object
Env1”, “Network1 Note the quotes. The text entered does not include the
quotes.

Rev A04
Mellanox Technologies 13
Related Documentation
For additional information, see the following documents:
• Mellanox VantageTM 8500 10GbE Layer-2 Core Switch User Manual
• Release Notes for the Mellanox Vantage 8500 Switch
• Regulatory Compliance and Safety Reference Guide

Product OverviewRev A04
Mellanox Technologies
14
1 Product Overview
1.1 The Vantage 8500 Switch
The Vantage 8500 switch is a completely modular, high-performance, high density, non-blocking
Layer-2 core switch, optimized for enterprise data center and cloud computing environments. The
Vantage 8500 switch is one of the most power-efficient and lowest latency Ethernet switches avail-
able, enabling new levels of efficiency, scalability, and real-time application performance, while at
the same time consolidating multiple or redundant network tiers and helping reduce infrastructure
expenses. The Vantage 8500 switch features an impressive 11.52 Tb/s non-blocking backplane,
allowing applications to perform at maximum bandwidth and efficiency.
The Vantage 8500 uses Converged Enhanced Ethernet (CEE) technology to provide InfiniBand-
like capabilities, such as a lossless switching fabric, and supports Fibre Channel over Ethernet
(FCoE) traffic, Ethernet multi-pathing, I/O virtualization, fabric-wide congestion management and
QoS. The switch is interoperable with standards-based switches, 10 GbE network adapter cards,
and third-party management and virtualization solutions enabling a best-of-breed, cost-effective
data center network.
In contrast to traditional Ethernet infrastructure configurations, the Vantage 8500 switch enables
the creation of a scale-out switch fabric, which is considerably easier to maintain and manage, as
shown in the following figure.
Figure 1: Vantage 8500 Switches in a Scale-Out Ethernet Configuration
The Mellanox switch operating system (VT-OS) comprises Layer-2 protocol
stack and advanced chassis management capabilities that automate regular
administrative tasks, via a user-friendly CLI, GUI, and SNMP-based interface.
For more information about the switch management and fabric management
options, please refer to the Mellanox Vantage™ 8500 Software User Manual
and to the Mellanox Unified Fabric Manager™ (UFM™) User Manual.

Rev A04
Mellanox Technologies 15
The Vantage 8500 switch is interoperable with existing top-of- rack switches, as well as with third-
party management and virtualization solutions.
1.2 Physical Description
This section provides a description of the Vantage 8500 components.
1.2.1 Chassis
The Vantage 8500 chassis is 15U high plus; mounting hardware requires 1/3 of 1 U either above or
below the chassis, 17.5 inches wide (19 inches with the fixed side brackets), and 22.75 inches
deep. The Vantage 8500 contains different plug-in modules and has mounting holes on each of its
vertical mounting panels (left hand and right-hand panels). The front panel provides access to the
plug-in latches of the fabric and management boards, and includes the airflow holes of the cooling
fans, as shown.
Figure 2: Vantage 8500 Switch -Front View

Product OverviewRev A04
Mellanox Technologies
16
The rear panel, as shown, provides access to the 10 GbE ports (up to 288 depending on the number
of line boards installed), line board latches, power supply units, and control board.
Figure 3: Vantage 8500 Switch - 3D Rear View
For details about the different front/rear LED indicators, see Section 7.4, Checking the LED Indi-
cators.
1.2.2 Plug-in Modules - Overview
As a modular system, the Vantage 8500 switch comprises plug-in modules that are installed into its
chassis. The following table summarizes the part types and their respective Mellanox names.
Table 2 - Vantage 8500 Switch - Part Naming Conventions
Part Mellanox Name Max # of Modules
Line Board sLB-1024 12

Rev A04
Mellanox Technologies 17
1.2.2.1 Line Board (sLB-1024)
A single sLB-1024 line board module provides 24 10GbE ports for the Vantage 8500 switch, as
shown.
You can install up to 12 line boards in the Vantage 8500 switch (via the back panel), depending on
the required hardware configuration.
In each sLB-1024 board, 12 10GbE ports numbered from 1 to 12 (left to right) are located in the
lower port row. The upper row includes ports 13 to 24 (from left to right).
Figure 4: Line Board (sLB-1024)
1.2.2.2 Line Board (sLB-1024) Architecture
The following table summarizes the Vantage 8500 Line Board components. There are two assem-
blies per line board.
Fabric Board sFB-8500 4
Power Supply Unit sPSU-S 5
Vertical Fan Unit sFU-4 1
Horizontal Fan Unit sFU-8E 1
Management Board sMBE 2
Control Board sCTRL 1
Table 3 - Vantage 8500 Switch Line Board Components
Component Function
CPU • Analyze and export statistics
• Detect & report fabric and I/O changes
• Configure Switch and FPGA policies
FPGA • Bridge between CPU bus and switch chip
•DMA
Table 2 - Vantage 8500 Switch - Part Naming Conventions
Part Mellanox Name Max # of Modules

Product OverviewRev A04
Mellanox Technologies
18
1.2.2.3 Fabric Board (sFB-8500)
The sFB-8500 fabric board is a main CPU and traffic processing unit for the Vantage 8500 switch.
You can install up to 4 fabric boards (1, 2, 3 or 4) in the Vantage 8500 switch (via the front panel),
depending on the required hardware configuration.
Figure 5: Fabric Board (sFB-8500)
1.2.2.4 Management Board (sMBE)
The Vantage 8500 switch includes two management boards, as shown. Each management board
includes two serial ports (RS-232; DB-9) for out-of-band management, as well as general info
LED indicators. For more information, see “Management Board (sMBE) LED Indicators” on
page 78.
Switch • 24 port 10GbE ASIC (Fulcrum Bali)
• CEE compliant
• Low latency (300ns)
Table 3 - Vantage 8500 Switch Line Board Components
Component Function

Rev A04
Mellanox Technologies 19
Figure 6: Management Board (sMBE)
1.2.2.5 Power Supply Unit (sPSU-S)
A sPSU-S module is a hot-swappable power supply unit for the Vantage 8500 switch. You can
install up to 5 power supplies in the chassis, depending on the number of installed modules (line
boards, fabric boards...) and the power they require, without power supply redundancy. For more
information, See “Product Options” on page 21. and See “Power Redundancy (N:1 and N:N)” on
page 35..
Figure 7: Power Supply Unit (sPSU-S)
1.2.2.6 Vertical Fan Unit (sFU-4)
The sFU-4 vertical fan unit, is part of the switch air cooling system that provides cooling for the
installed line boards that are located on the opposite side of the chassis. The sFU-4 unit includes
four fixed fans and supports fan speed control. The fan speed is dynamically controlled as a func-
tion of temperature: the higher the temperature - the faster the speed of the fans. The sFU-4 unit
does not have LED indicators, but the sFU-8E fan unit has an sFU Optional LED that lights up if
there is a problem with the sFU-4 module.

Product OverviewRev A04
Mellanox Technologies
20
Figure 8: Vertical Fan Unit (sFU-4)
1.2.2.7 Horizontal Fan Unit (sFU-8E)
The sFU-8 horizontal fan unit, as shown, is part of the air cooling system that cools the fabric
boards and the sMBE boards. The sFU-8 includes eight fans with fan speed control. Fan speed is
dynamically controlled as a function of temperature. The sFU-8E unit has LED indicators. The
LED indicators provide an on-site indication of the operating conditions of the air cooling system
modules, including an alarm indicator that lights up when the temperature rises to a high level. For
more information, see See “Replacing the sFU-8E Unit” on page 70..
Figure 9: Horizontal Fan Unit (sFU-8E) Front Panel
1.2.2.8 Control Board (sCTRL)
The control board (sCTRL), as shown, allows connection to a local out-of-band terminal control
station, via an Ethernet (up to 100 Mbps) or a serial interface (RS-232) connection. The board
includes two Ethernet interfaces (RJ-45) and two serial interfaces (DB-9).