Voltaire ISR 9024 User manual

Voltaire Switch User Manual
ISR 9024, ISR 9096, and ISR 9288 Switches
Business Headquarters
Voltaire Inc.
6 Fortune Drive, Suite 301
Billerica, MA USA 01821
Tel: [phone]
Fax: [phone]
Israel Office
Voltaire Ltd.
9 Hamenofim St.
Bldg. A Herzeliya
46725, Israel
Tel: [phone]
Fax: [phone]
Release AAA-BAA
May 2006
P/N 399Z00038

THIS DOCUMENT IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY
WARRANTY OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR
PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION
OR EXAMPLES.
Voltaire disclaims all liability, including liability for infringement of any proprietary rights, relating to use of
information in this specification.
No license, expressed or implied, by written or otherwise, to any intellectual property rights is granted herein.
This document as well as the software described in it is furnished under license and may only be used or copied
in accordance with the terms of the license. The information in this manual is furnished for informational use
only, is subject to change without notice, and should not be construed as a commitment by Voltaire. Voltaire
Ltd. assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any
software that may be provided in association with this document. Except as permitted by such license, no part of
this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means
without the express written consent of Voltaire Ltd.
Voltaire is a trademark or registered trademark of Voltaire Ltd. or its subsidiaries in the United States and other
countries.
Other names and brands may be claimed as the property of others and have been designated with an asterisk (*)
throughout this document.
Copyright © 2006, Voltaire Ltd.
Voltaire Part Number: 399Z00038

About this Manual
This preface describes the audience, organization and command syntax conventions
of the Voltaire Switch User Manual. It also provides information on how to obtain
related documentation and technical assistance. Information about hardware
installation of Voltaire switches can be found in ISR 9024 Installation Manual and
in the ISR 9XXX Installation Manual, which includes the ISR 9096 and the
ISR 9288.
NOTE
Consult Voltaire release notes for last minute updates and restrictions.
The Voltaire Technical Support Center (TSC) is at your service. You may access
Warranty Service through our Web Request Form by using the following link:
http://www.voltaire.com/support.html
Contact Us:
Please send your documentation-related comments and feedback or report mistakes
to [email protected].
We are committed to constant and never-ending improvement. Your input will
greatly help us in our endeavor.
Audience
This manual is primarily intended for system administrators who are familiar with
the fundamentals of router-based internetworking and network storage devices, but
who might not be familiar with the specifics of Voltaire products or the routing
protocols supported by Voltaire products
It is assumed that readers are familiar with InfiniBand technology and terminology.
Related Documentation
ISR 9XXX Installation Manual, Voltaire
ISR 9024 Installation Manual, Voltaire

About this Manual
ii
P/N 399Z00038 © 2006 Voltaire, Inc.
Document Conventions
NOTE
Text set off in this manner presents clarifying information, specific instructions,
commentary, sidelights, or interesting points of information.
IMPORTANT
Text set off in this manner indicates important information regarding a specific
feature.
CAUTION
Text set off in this manner indicates that failure to follow directions could result in
damage to equipment or loss of information.

About this Manual
iii
P/N 399Z00038 © 2006 Voltaire, Inc.
Document Organization
This guide contains the following chapters:
Chapter 1 – Introduction: Describes the InfiniBand technology and provides an
introduction to the ISR 9024, ISR 9096 and ISR 9288 switches.
Chapter 2 – CLI Overview: Provides a brief introduction on the Voltaire
Command Line Interface (CLI).
Chapter 3 – Initial System Setup: Provides instructions on how to perform an
initial setup of your system.
Chapter 4 – Basic CLI Maintenance Commands: This section explains how to
configure switch parameters and how to perform a software and firmware
upgrade.
Chapter 5 – Voltaire Device Manager (VDM): Describes the Voltaire Device
Manager, a Graphic User Interface (GUI) application for device management
and switch configuration and provides operation instructions.
Chapter 6 – Voltaire Fabric Manager (VFM): Describes the Voltaire Fabric
Manager (VFM), a Graphic User Interface (GUI) application that provides a
wide range of provisioning, configuration and maintenance functions for the
Switch, and provides operation instructions.
Chapter 7 – Names Settings: This section provides instructions on how to set
names to Switches and unmanaged ISR 9024.
Chapter 8 – Rack Grouping: Introduces the Rack grouping feature and
provides instructions on how to create, set, edit, view, and delete rack-groups.
Chapter 9 – Configuring Quality of Service (QoS): Provides a QoS overview,
instruction on how to perform SL to VL mapping, QoS arbitration, and an
example of SM parameter configuration.
Chapter 10 – Configuring Routing Algorithms: Lists the supported algorithms,
details the Balanced Routing, Rearrangeable Routing (Multi-Path with sFB
Consistency), and Up/Down Routing schemes, as well as providing
instructions on how to configure Routing via the VFM and the CLI.
Chapter 11 – Configuring Partitioning: Describes the Partitioning feature used
to enforce isolation among systems using an InfiniBand Fabric and provides
instructions on how to assign partition keys (Pkeys). It also describes how
Partitioning is used to associate hosts or devices that share common attributes.
Chapter 12 – Replacing Modules: Provides instructions on how to replace
sFBs, sLBs, sMBs and unmanaged ISR 9024.

About this Manual
iv
P/N 399Z00038 © 2006 Voltaire, Inc.
Chapter 13 – Configuring Cluster Monitor: Allow you to view your entire
cluster in a browser using the Ganglia tool.
Chapter 14 – Fabric Diagnostic: Provides information on logging and
monitoring functions for advanced fabric debugging and failure detection
capabilities.
Appendix A – CLI Commands Reference: Explains the CLI basic concept and
details the CLI command menus. The CLI provides configuration and
management functions for Voltaire switches, as well as a wide range of
provisioning, configuration and maintenance functions for the InfiniBand
fabric.
Appendix B – Port Counters : For Advanced Diagnostics - Describes and
details the port counter fields of the CSV file, details the CSV counters
summary reports and lists the ports counters currently available.
Appendix C – Supported Traps and Events: Defines the structure of traps and
lists the traps, events and alarms supported in this version.
Appendix D – Glossary
Appendix E – Index

About this Manual
v
P/N 399Z00038 © 2006 Voltaire, Inc.
What's New in this Manual?
This switch software version includes the following new features:
Subnet Management Enhancements: Enhanced “Bad-Ports” feature, Up-Down
Routing Scheme, Performance Improvement: Improve Fabric Configuration
Time, Subnet Manager Priority, Partitioning, Non-Intrusive Routing
Reconfiguration, and Subnet Prefix.
Rack Grouping (Creating, renaming, deleting rack groups, and viewing rack
information via the GUI and the CLI; groups.csv file)
The Voltaire switch software supports a new hardware module: sLB-8-12
Diagnostics Enhancements: Enhanced Event Notification - extended
notification mode, New alarm category in current alarm mechanism: Fabric
Config Error, Enhanced Performance Manager (PM) [PortCounters files; Port-
verify Utility; Port-Manage Utility]
Unique Switch Name
CLI Enhancements (exportLogs, new madstat and vendstat options)
New default firmware settings
Router Version Automatic Upgrade

Contents
Audience........................................................................................................................................i
Related Documentation................................................................................................................i
Document Conventions ..............................................................................................................ii
Document Organization.............................................................................................................iii
What's New in this Manual? .......................................................................................................v
Chapter 1. Introduction..........................................................................................................1-1
1.1 About InfiniBand...........................................................................................................1-2
1.2 ISR 9024 System Overview ..........................................................................................1-3
1.3 ISR 9288/9096 System Overview.................................................................................1-4
1.4 Voltaire Switch Management......................................................................................1-10
Chapter 2. CLI Overview........................................................................................................2-1
2.1 Interfaces for CLI Connection.......................................................................................2-2
2.2 CLI Command Modes...................................................................................................2-2
2.3 CLI Usage Tips .............................................................................................................2-3
2.4 Keyboard Usage in CLI Operation................................................................................2-3
2.5 CLI Command Structure ...............................................................................................2-5
2.6 CLI Case Sensitivity......................................................................................................2-5
2.7 CLI Command Prompts.................................................................................................2-6
Chapter 3. Initial System Setup.............................................................................................3-1
3.1 Prerequisite Tasks..........................................................................................................3-2
3.2 First-Time Configuration...............................................................................................3-2
3.3 Collecting Required Information...................................................................................3-2
3.4 Connecting a Console....................................................................................................3-3
3.5 Starting a CLI Session...................................................................................................3-3
3.6 Configuring the Management Interface IP Address......................................................3-4
3.7 Configuring the Management IP Address in ISR 9288/9096/9024 Switches ...............3-5
3.8 Configuring the ISR 9288/9096 with I/O Modules.......................................................3-7
3.9 Configuring Time and Date...........................................................................................3-9

Contents
ii
P/N 399Z00038 © 2006 Voltaire, Inc.
3.10 Configuring Passwords..................................................................................................3-9
3.11 Verifying Configuration..............................................................................................3-10
Chapter 4. Basic CLI Maintenance Commands...................................................................4-1
4.1 Prerequisites ..................................................................................................................4-2
4.2 Updating Software.........................................................................................................4-2
4.3 Prerequisites for Updating Software .............................................................................4-3
4.4 Backing up the Switch Configuration ...........................................................................4-4
4.5 Software and Firmware Upgrade Process .....................................................................4-5
4.6 CLI Commands Reference ............................................................................................4-6
Chapter 5. Voltaire Device Manager (VDM)..........................................................................5-1
5.1 Device Manager Description.........................................................................................5-2
5.2 Getting Started with the Device Manager .....................................................................5-3
5.3 The Device Manager Main Window.............................................................................5-6
5.4 Voltaire Device Manager Operating Conventions ......................................................5-16
5.5 Viewing Switch Information.......................................................................................5-16
5.6 Viewing Module Information......................................................................................5-27
5.7 Using the Voltaire Device Manager for Switch Management.....................................5-40
5.8 Reset Operations..........................................................................................................5-45
Chapter 6. Voltaire Fabric Manager (VFM)...........................................................................6-1
6.1 Fabric Manager Description..........................................................................................6-2
6.2 Fabric Manager Installation...........................................................................................6-3
6.3 Getting Started with the Fabric Manager ......................................................................6-5
6.4 The Fabric Manager Main Window ..............................................................................6-8
6.5 Viewing Properties Data..............................................................................................6-28
6.6 Information Windows..................................................................................................6-31
6.7 Configuring the Fabric ................................................................................................6-41
6.8 Configuring Nodes ......................................................................................................6-61
Chapter 7. Names Settings....................................................................................................7-1
7.1 CLI Switch Name Settings............................................................................................7-2
7.2 GUI Node Name Settings..............................................................................................7-3
7.3 Naming an Unmanaged ISR 9024.................................................................................7-4
Chapter 8. Rack Grouping.....................................................................................................8-1
8.1 Rack Grouping Overview..............................................................................................8-2
8.2 Rack Grouping Settings via the VFM ...........................................................................8-3
8.3 Elements of the Group.csv File...................................................................................8-10
8.4 Backing up and Editing Groups Settings via the CLI .................................................8-12

Contents
iii
P/N 399Z00038 © 2006 Voltaire, Inc.
8.5 Rack Grouping Installation Suggestion.......................................................................8-14
Chapter 9. Configuring Quality of Service (QoS)................................................................9-1
9.1 QoS Overview...............................................................................................................9-2
9.2 SL to VL Mapping ........................................................................................................9-2
9.3 QoS Arbitration.............................................................................................................9-4
9.4 Example of SM Parameter Configuration.....................................................................9-7
Chapter 10. Configuring Routing Algorithms......................................................................10-1
10.1 Supported Algorithms .................................................................................................10-2
10.2 Balanced Routing Scheme...........................................................................................10-2
10.3 Rearrangeable Routing (Multi-Path with sFB Consistency).......................................10-2
10.4 Up/Down Routing Scheme..........................................................................................10-3
10.5 Configuring Routing via the VFM ..............................................................................10-4
10.6 Configuring Routing via the CLI ................................................................................10-7
Chapter 11. Configuring Partitioning....................................................................................11-1
11.1 Partitioning Overview .................................................................................................11-2
11.2 Mechanisms Used to Create and Enforce Partitions ...................................................11-3
11.3 Setting Partitioning......................................................................................................11-3
Chapter 12. Replacing Modules............................................................................................12-1
12.1 Replacing sFBs (Spines)..............................................................................................12-2
12.2 Replacing sLBs............................................................................................................12-6
12.3 Replacing sMBs ..........................................................................................................12-7
12.4 Replacing an Unmanaged ISR 9024............................................................................12-8
Chapter 13. Configuring Cluster Monitor.............................................................................13-1
13.1 What is Ganglia? .........................................................................................................13-2
13.2 Cluster Configuration..................................................................................................13-2
Chapter 14. Fabric Diagnostic...............................................................................................14-1
14.1 Fabric Diagnostic Overview........................................................................................14-2
14.2 Fabric Inspect Settings ................................................................................................14-7
14.3 Diagnostic Tools........................................................................................................14-10
14.4 Additional Tools and Examples ................................................................................14-20