Cisco InfiniBand 4x User manual

Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Cisco 4x InfiniBand Switch Module for IBM
BladeCenter User Guide
Text Part Number: 78-17838-01

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the
document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide
© 2006 Cisco Systems, Inc. All rights reserved.
CCSP, the Cisco Square Bridge logo, Follow Me Browsing, and StackWise are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, and iQuick
Study are service marks of Cisco Systems, Inc.; and Access Registrar, Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, Cisco, the Cisco Certified
Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Empowering the Internet Generation,
Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, FormShare, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ
Net Readiness Scorecard, LightStream, Linksys, MeetingPlace, MGX, the Networkers logo, Networking Academy, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing,
ProConnect, RateMUX, ScriptShare, SlideCast, SMARTnet, StrataView Plus, SwitchProbe, TeleRouter, The Fastest Way to Increase Your Internet Quotient, TransPath, and VCO
are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship
between Cisco and any other company. (0501R)

i
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide
78-17838-01
CONTENTS
Preface v
Audience v
Organization v
Conventions v
Related Documentation vi
Contacting Technical Support vii
IBM Support Contact Information vii
Before You Call viii
Determining the Software Version viii
Downloading the Latest Software Release viii
Cisco Systems Support Contact Information viii
Cisco Technical Support & Documentation Website ix
Submitting a Service Request ix
Definitions of Service Request Severity ix
IBM Statement of Limited Warranty Z125-4753-08 04/2004 xi
Part 1 - General Terms xi
What this Warranty Covers xi
What This Warranty Does Not Cover xii
How to Obtain Warranty Service xii
What IBM Will Do to Correct Problems xii
Exchange of a Machine or Part xiii
Your Additional Responsibilities xiii
Limitation of Liability xiii
Governing Law xiv
Jurisdiction xiv
Part 2 - Country-unique Terms xiv
AMERICAS xiv
ARGENTINA xiv
BOLIVIA xiv
BRAZIL xv
CHILE xv
COLOMBIA xv

Contents
ii
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide
78-17838-01
EQUADOR xv
MEXICO xv
PARAGUAY xv
PERU xv
URUGUAY xvi
VENEZUELA xvi
NORTH AMERICA xvi
CANADA xvi
UNITED STATES xvi
ASIA PACIFIC xvi
AUSTRALIA xvi
CAMBODIA AND LAOS xvii
CAMBODIA, INDONESIA, AND LAOS xvii
HONG KONG S.A.R. OF CHINA AND MACAU S.A.R. OF CHINA xvii
INDIA xviii
JAPAN xviii
MALAYSIA xviii
NEW ZEALAND xviii
PEOPLE'S REPUBLIC OF CHINA (PRC) xix
PHILIPPINES xix
SINGAPORE xix
EUROPE, MIDDLE EAST, AFRICA (EMEA) xix
EUROPEAN UNION (EU) xxi
CONSUMERS xxii
FRANCE AND BELGIUM xxii
THE FOLLOWING TERMS APPLY TO THE COUNTRY SPECIFIED: xxiii
AUSTRIA xxiii
EGYPT xxiii
FRANCE xxiii
GERMANY xxiv
HUNGARY xxiv
IRELAND xxiv
SLOVAKIA xxv
SOUTH AFRICA, NAMIBIA, BOTSWANA, LESOTHO AND SWAZILAND xxv
UNITED KINGDOM xxv
Part 3 - Warranty Information xxvi
Warranty Period xxvi

Contents
iii
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide
78-17838-01
Types of Warranty Service xxvii
EU Country Telephone List xxviii
Notices xxix
Edition Notice xxx
Trademarks xxx
Important Notes xxxi
Product Recycling and Disposal xxxi
Battery Return Program xxxi
Electronic Emission Notices xxxii
Federal Communications Commission (FCC) Statement xxxii
Industry Canada Class A emission compliance statement xxxiii
Australia and New Zealand Class A statement xxxiii
United Kingdom telecommunications safety requirement xxxiii
European Union EMC Directive conformance statement xxxiii
Taiwanese Class A warning statement xxxiii
Chinese Class A warning statement xxxiv
Japanese Voluntary Control Council for Interference (VCCI) statement xxxiv
WEEE Notice xxxv
Safety xxxvii
CHAPTER 1About the Server Switch Module 1-1
Introduction 1-1
Server Switch Module and HCA Expansion Cards 1-2
Topologies 1-3
Single-Switch Topology 1-3
Dual-Switch Topology 1-3
Module Management 1-3
LEDs 1-4
Server Switch Module Status LEDs 1-4
Port Status LEDs 1-5
External InfiniBand Ports 1-5
CHAPTER 2Installing and Configuring the Server Switch 2-1
Installing the Server Switch Module 2-1
Downloading Operating System Software from Cisco.com 2-1
Configuring the Server Switch Module 2-2

Contents
iv
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide
78-17838-01
Changing the Default Password 2-3
Configuring the System Clock 2-3
Connecting Beyond BladeCenter 2-5
Connecting InfiniBand Cables 2-5
CHAPTER 3Specifications 3-1
Electrical Requirements 3-1
Environmental Requirements 3-1
I NDEX

v
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide
78-17838-01
Preface
Audience
The intended audience is the administrator responsible for installing, configuring, and managing Server
Switch equipment. This administrator should have experience administering similar networking or
storage equipment.
Organization
This publication is organized as follows:
Conventions
This document uses the following conventions:
Chapter Title Description
Chapter 1 About the Server Switch Module Describes the server switch module and
HCA expansion cards, including topology,
management, the LEDs, and external
InfiniBand ports.
Chapter 2 Installing and Configuring the
Server Switch
Installation, configuration, and external
connection of the server switch module are
described.
Chapter 3 Specifications Electrical and environmental requirements
are described in this chapter.
Convention Description
boldface font Commands, command options, and keywords are in
boldface. Bold text indicates Chassis Manager elements or
text that you must enter as-is.
italic font Arguments in commands for which you supply values are in
italics. Italics not used in commands indicate emphasis.

vi
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide
78-17838-01
Preface
Related Documentation
Notes use the following conventions:
Note Means reader take note. Notes contain helpful suggestions or references to material not covered in
the publication.
Cautions use the following conventions:
Caution Means reader be careful. In this situation, you might do something that could result in equipment
damage or loss of data.
Related Documentation
• Cisco 4x InfiniBand HCA Expansion Card for IBM BladeCenter User Guide
• Cisco Boot over IB User Guide for Linux
• Cisco Boot over IB User Guide for Windows
Menu1 > Menu2 >
Item…
Series indicate a pop-up menu sequence to open a form or
execute a desired function.
[ ] Elements in square brackets are optional.
{ x | y | z } Alternative keywords are grouped in braces and separated by
vertical bars. Braces can also be used to group keywords
and/or arguments; for example, {interface interface type}.
[ x | y | z ] Optional alternative keywords are grouped in brackets and
separated by vertical bars.
string A nonquoted set of characters. Do not use quotation marks
around the string or the string will include the quotation
marks.
screen font Terminal sessions and information the system displays are in
screen font.
boldface screen
font
Information you must enter is in boldface screen font.
italic screen font Arguments for which you supply values are in italic screen
font.
^The symbol ^ represents the key labeled Control—for
example, the key combination ^D in a screen display means
hold down the Control key while you press the D key.
< > Nonprinting characters, such as passwords are in angle
brackets.
[ ] Default responses to system prompts are in square brackets.
!, # An exclamation point (!) or a pound sign (#) at the beginning
of a line of code indicates a comment line.
Convention Description

vii
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide
78-17838-01
Preface
Contacting Technical Support
• Cisco Command Line Interface Reference Guide
• Cisco Chassis Manager User Guide (Web GUI)
• Cisco Element Manager User Guide
Contacting Technical Support
Please read this section before contacting IBM or Cisco technical support.
IBM Support Contact Information
For technical support, see the following Web site:
http://www.ibm.com/support
To contact technical support, send E-mail to [e-mail] or see one of the telephone numbers
below.
To obtain warranty service contact IBM or your IBM reseller. In Canada or the United States, call
1-800-IBM-SERV (426-7378). In the EU countries, see the telephone numbers in Table 1.
Ta b l e 1 EU Country Telephone List
Phone numbers are subject to change without notice.
Note IBM and Cisco are both available for more information. For support, contact IBM before Cisco.
Austria—43-1-24592-5901 Italy—9-02-482-9202
Belgium—02-718-4339 Luxembourg—352-360385-1
Denmark—4520-8200 Netherlands—020-514-5770
Finland—358-9-4591 Portugal—351-21-7915-147
France—0238-557-450 Spain—34-91-662-4916
Germany—07032-15-4920 Sweden—46-8-477-4420
Greece—30-210-688-1220 United Kingdom—01475-555-055
Ireland—353-1-815-4000

viii
Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide
78-17838-01
Preface
Contacting Technical Support
Before You Call
Be prepared to provide the following information to support personnel.
General Information
• Technical Support registration number, if applicable
• Error messages received
• Detailed description of the problem and specific questions
• Description of any troubleshooting steps already performed and results
Server configuration
• Type of server, chip set, CPU, amount of RAM, # of nodes
• Attached storage devices (output from cat /proc/scsi/scsi)
• InfiniBand configuration (output from /usr/local/topspin/sbin/hca_self_test)
Topspin Chassis configuration
• Chassis model
• Output from the show running-status all command
Topspin Chassis Serial Number
The chassis serial number and corresponding bar code are provided on the serial number label. This can
be found on the bottom of the chassis or the outside of the Topspin chassis box packaging. It can also
be found in the output of the show backplane command.
Determining the Software Version
If InfiniBand drivers are already installed on the host, to determine the version of the Topspin Linux host
drivers you are running, enter the following command at the prompt.
$ rpm -qa | grep topspin
topspin-ib-mod-rhel3-2.4.21-32.ELsmp-3.2.0-82
topspin-ib-mpi-rhel3-3.2.0-82
topspin-ib-rhel3-3.2.0-82
Downloading the Latest Software Release
To verify that you are running the latest available release, compare your version against the latest version
on the Cisco support website at http://www.cisco.com/cgi-bin/tablebuild.pl/sfs-linux. After registering
your product, you should have received a username and password to give you access to this site.
Cisco Systems Support Contact Information
Cisco Technical Support provides 24-hour-a-day award-winning technical assistance. The Cisco
Technical Support & Documentation website on Cisco.com features extensive online support resources.
In addition, if you have a valid Cisco service contract, Cisco Technical Assistance Center (TAC)
engineers provide telephone support. If you do not have a valid Cisco service contract, contact your
reseller.