Cisco SFS 7008P Operating and maintenance instructions

Corporate Headquarters:
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Systems, Inc., 170 West Tasman Drive, San Jose, CA 95134-1706 USA
Cisco SFS 7008P InfiniBand Server Switch
Installation and Configuration Note
Product number: PID=78-17434-01 Rev. A0
This installation and configuration note describes how to install and configure the Cisco SFS 7008P
switch. This publication contains these sections:
•Safety Overview, page 2
•Parts List, page 2
•Required Tools, Equipment, and Personnel, page 2
•Before You Install, page 3
•Installing the Cisco SFS 7008P Switch into a Rack, page 6
•Troubleshooting, page 20
•Translated Safety Warnings, page 21
•Obtaining Documentation, page 27
•Documentation Feedback, page 27
•Cisco Product Security Overview, page 28
•Obtaining Technical Assistance, page 29
•Obtaining Additional Publications and Information, page 31

2
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Safety Overview
Safety Overview
Safety warnings appear throughout this publication in procedures that may harm you if performed
incorrectly. A warning symbol precedes each warning statement.
Parts List
The following items are included in your SFS 7008 package:
•Chassis with TopSpin OS v2.5.0 preloaded
•2 Serial Cable Kits
– Serial Cable
– Serial cable document
– DB-9 connector male to RJ-45
– DB-9 connector female to RJ-45
– MX-9-1 roll-over cable
•Power cord for each power supply
•Bezel
•Parts bag
•Regulation and Compliance Statement
•1 year warranty package (Cisco PN 83-1190-01)
Required Tools, Equipment, and Personnel
In addition to the accessories provided with the switch, you should have the following:
•A #2 Phillips screw driver
•Screws (not provided) to attach the mounting brackets to the rack
•Any associated mounting clips to secure the brackets to your rack (two for each rail of the rack)
•A rack with enough clearance for the Cisco SFS 7008P switch and cables.
Warning IMPORTANT SAFETY INSTRUCTIONS
This warning symbol means danger. You are in a situation that could cause bodily injury. Before you
work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar
with standard practices for preventing accidents. Use the statement number provided at the end of
each warning to locate its translation in the translated safety warnings that accompanied this
device.
Statement 1071
SAVE THESE INSTRUCTIONS

3
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Before You Install
•Three people to complete the installation (highly recommended). The Cisco SFS 7008P switch
chassis weighs up to 106 pounds fully loaded.
Before You Install
Before you begin the installation, do the following:
•Make sure that you have the right cables and sufficient ventilation.
•Open the Cisco SFS 7008P package and remove the chassis, rack brackets, parts bag, and
documentation from the box. Remove the packaging from around the chassis. After you have
removed the straps, lift the box up and away from the switch and pallet. Do not cut the packaging,
as you could damage the switch. Save the box and packaging.
•Prepare a management workstation, such as a PC or laptop running a terminal program with a
rollover M/F DB-9 serial cable (included).
Verifying Basic Hardware
If you have not purchased a fully loaded chassis with high-availability components installed, verify that
you have the minimum components necessary to run the chassis. Specific components in the Cisco SFS
7008P switch (see Figure 1 and Figure 2) have interdependencies on other components.
Check for basic hardware by following these steps:
Step 1 Verify that you have at least one management interface module installed in slot 15 at the back of the unit.
(See Figure 1.) Slot 15 is paired with the Fabric Controller in slot 11. (See Figure 2.)

4
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Verifying Basic Hardware
Figure 1 Back of Unit
Step 2 Verify that you have a Chassis ID module installed. (See Figure 1).
Step 3 Verify that you have a Fabric Controller installed in the Core slot (see slot 11 in Figure 2). A second
Fabric Controller module (installed in slot 12) is not required for the chassis to function, but is required
to have a non-blocking fabric.
144988
Management I.O
modules
Power
plug
2
Power
plug
1
Chassis ID
Main air
exhaust
4x copper
line interface
modules

5
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Verifying Basic Hardware
Figure 2 Cisco SFS 7008P Switch Front ViewWithout Bezel
Step 4 Verify that you have at least one power supply installed. (See Figure 2.)
Step 5 Verify that you have at least one fan module installed. (See Figure 2.)
Step 6 Verify that you have at least one fabric module installed in one of the node slots. The node slots are 9,
10, 13 and 14. (See Figure 2.)
Step 7 Verify that you have two LIMs installed for each fabric module in the node slots. See Figure 3 for an
example of the connections between the LIMs and the fabric modules in the node slots:
154497
System
Status LED
pp y
Status LED
Power Supply
identity LED
Power supplies
Fan Fan
Node slot 1 Slot 9
Slot 10
Slot 11
Slot 12
Slot 13
Slot 14
Node slot 2
Node slot 3
Node slot 4
Core slot 1
Core slot 2
Fan tray LED

6
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Figure 3 LIM and Fabric Module Connections
Installing the Cisco SFS 7008P Switch into a Rack
This section describes how to install the Cisco SFS 7008P switch in an equipment rack. If you purchased
the custom Cisco SFS 7008P switch shelf, follow the directions in the Cisco SFS 7008 InfiniBand Server
Switch Hardware Installation Guide. To mount the Cisco SFS 7008P switch chassis in an
industry-standard rack, follow these steps:
Step 1 Install a shelf for the chassis as shown in Figure 4, if you have not already done so.
The shelf must support the weight of the chassis, which can be up to 106 pounds if all possible
components are installed.
Fabric card in slot 9
LIM card in slot 1
LIM card in slot 2
LIM card in slot 3
LIM card in slot 4
LIM card in slot 5
LIM card in slot 6
LIM card in slot 7
LIM card in slot 8
Fabric card in slot 10
Fabric card in slot 13
Fabric card in slot 14
154550

7
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Figure 4 Install Appropriate Shelf
Step 2 Ground yourself using an approved ground wrist-strap.
Step 3 Determine the direction that the switch will be installedand justified in the rack.
The direction that the switch will be justified determines which set of mounting brackets need to be
removed in the next step.
Step 4 Remove one set of mounting brackets, as shown in Figure 5:
•If the switch will be justified toward the bezel-side in the rack, then unscrew the mounting brackets
from the port-side of the switch.
•If the switch will be justified toward the port-side in the rack, then unscrew the mounting brackets
from the bezel-side of the switch.
144964

8
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Figure 5 Unscrew Extra Mounting Brackets from the Chassis
Step 5 Separate the mounting brackets from the chassis when all screws have been loosened.
Step 6 With 2 or 3 people working together, lift the switch into the rack as shown in Figure 6.
Figure 6 Chassis Set onto Shelf
Step 7 Push the chassis into the rack until the mounting brackets are flush with the rails.
Step 8 Use screws that fit your rack to attach both mounting brackets to the rack posts.
144965
144966

9
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Installing Optional High-Availability Components
If you purchased a High-Availability (HA) package, install the following components:
•Redundant Power Supply; see the power supply locations in Figure 7. Install the second power
supply according to the directions in the Power Supply Module Installation Guide in the package.
Figure 7 Power Supplies and Fans Located in the Front of the Unit with the Bezel Removed
•Redundant Fan Tray; see the fan locations in Figure 7. Install the second fan according to the Fan
Module Installation Guide in the package.
•Redundant Management Interface Module; see the location of the Management I/O Modules in
Figure 8. Install the second I/O Module according to the directions in the Management Interface
Module Installation Guide in the package.
154497
System
Status LED
Power Supply
Status LED
Power Supply
identity LED
Power supplies
Fan Fan
Node slot 1 Slot 9
Slot 10
Slot 11
Slot 12
Slot 13
Slot 14
Node slot 2
Node slot 3
Node slot 4
Core slot 1
Core slot 2
Fan tray LED

10
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Figure 8 Management I/O Modules Located in the Back of the Unit
Configuring and Connecting Host Channel Expansion Card for IBM BladeCenter
This section briefly describes configuring your InfiniBand host. Download the configuration software
and an integrated suite of drivers including IPoIB, SRP, SDP, uDAPL and MPI from the Cisco Software
Center at this URL:
http://www.cisco.com/public/sw-center
Install the Host Channel Expansion Card for IBM BladeCenter by doing the following:
Step 1 Power down the host system following proper grounding procedures.
Step 2 Unpack the HCA module and insert it into an open 64-bit PCI or PCI-X slot.
Step 3 Power on the host system.
Installing Host Channel Expansion Card Linux Drivers
Make sure your Linux kernel is supported. See www.cisco.com for a list of supported versions.
144988
Management I.O
modules
Power
plug
2
Power
plug
1
Chassis ID
Main air
exhaust
4x copper
line interface
modules

11
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
To install the Host Channel Expansion Card for IBM BladeCenter drivers after you have installed the
HCA Expansion Card, follow these steps:
Step 1 Locate your downloaded software direcory.
Step 2 Open a terminal or graphic window and access the top-level directory.
Step 3 Execute the command ./tsinstall in a terminal window.
# cdmount ./tsinstall
This program automatically detects the available kernel and installs the appropriate drivers as RPM
packages. By default, the files are saved in the /usr/local/cisco directory.
Step 4 Configure the IPoIB driver by assigning an IP address to interface ib0 and/or ib1, which corresponds to
the two HCA ports. IPoIB drivers load when the interfaces are used for the first time. To assign ib
interfaces, enter ifconfig ib# ip address netmask netmask.
This example configures one HCA:
# ifconfig ib0 192.168.0.1 netmask 255.255.255.0
#
# ifconfig ib1 192.168.1.1 netmask 255.255.255.0
#
This example configures two HCAs:
# ifconfig ib0 192.168.0.1 netmask 255.255.255.0
#
# ifconfig ib1 192.168.1.1 netmask 255.255.255.0
#
# ifconfig ib2 192.168.2.1 netmask 255.255.255.0
#
# ifconfig ib3 192.168.3.1 netmask 255.255.255.0
#
Step 5 Merge ib0 and ib1 for port-to-port redundancy, if desired.
Step 6 (Optional) Configure MPI by following these steps:
•Make sure that your connections to the HCA uses port 1 of the first HCA. MPI requires the use of
HCA 1, port 1.
•Set up rsh or ssh on two nodes so that you can run commands between a remote and local node
without entering a login or password. This is required to use the MPI protocol.
•Either add paths to your environment PATH:
/usr/local/cisco/mpi/mpich/bin
/usr/local/cisco/bin
or
Add directories to your PATH variable. For example:
export PATH=$PATH:/usr/local/cisco
Step 7 Configure all other drivers, including SDP.

12
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Installing Non-WHQL Host Channel Expansion Card Drivers on a Windows 2000 Server
(BoIB and non-BoIB)
The SFS 7008 chassis has Topspin operating system version 2.5.0 preloaded. DRIVERS TOO? OR
DOWNLOAD?? IN STEP 4, THE SDK COMES UP - WHERE DID THAT COME FROM?
To install Windows drivers after you have installed the HCA Expansion Card, follow these steps:
Step 1 Log on to your Windows 2000 host.
A Found New Hardware alert appears (see Figure 9).
Figure 9 Found New Hardware Alert
Step 2 Ignore the alert.
The Found New Hardware Wizard window opens (see Figure 10).
Figure 10 Found New Hardware Wizard WIZARD.TIF
Step 3 Click Cancel.

13
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Note This wizard may not appear immediately. When the wizard does appear, click Cancel.
Step 4 Run Topspin-ib-W2k-releaseNumber.exe.
The InfiniBand SDK Setup window opens (see Figure 11).
Figure 11 InfiniBand SDK Setup Window
Step 5 Click Next.
The End User License Agreement window opens.
Step 6 Click the I Agree to the terms listed above radio button, and then click Next.
The Product Information window appears.
Step 7 Click Next.
The Ready to Install window opens.
Step 8 Click Next.
The installation executes. The Installation Complete window appears when the installation completes.
Step 9 Click Finish.
The system prompts you to reboot. If you have not previously installed host drivers on this system, you
do not need to reboot.
Step 10 Click Yes.
Your host reboots.

14
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Installing Host Channel Expansion Card Windows 2003 Drivers
If you have not already done so, download an integrated suite of HCA drivers including IPoIB, SRP, SDP,
uDAPL and MPI from the Cisco Software Center at this URL:
http://www.cisco.com/public/sw-center
To install the Windows 2003 drivers, follow these steps:
Step 1 Log on to your Windows 2003 host.
Step 2 HOW DID SW GET ON MY COMPUTER?HOW DO I START THIS????????????.
The Product Install window appears (see Figure 12).
Figure 12 Product Install Window
Step 3 Click Next.
The End User License Agreement appears.
Step 4 Click the I Agree to the terms listed above radio button, and then click Next to continue.
The Product Information window appears.
Step 5 Click Next.
The Installation in Progress window appears with the status of the installation (see Figure 13).

15
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Figure 13 Installation Status
A Windows driver dialog box appears. This dialog box explains that the new drivers do not contain a
Windows signature. Although the drivers are not digitally signed, they are tested using the Microsoft
HCT suite. WHQL submission materials are available per request (CAB files, etc.).
Step 6 Click Continue Anyway (Figure 14) each time the Windows Logo testing dialog box appears.
Figure 14 Testing Dialog Box
The driver installation notice shown in Figure 15 appears for each host channel adapter you have
installed.
Step 7 Click Yes in the Driver Installation Notice dialog window. (See Figure 15.)

16
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Figure 15 Driver Installation Notice
Step 8 Important: Ignore the Found New Hardware Wizard (see Figure 16) when it appears because it will
continue without user intervention. This window appears three times (twice for the IPoIB and once for
SRP).
Figure 16 Ignore the Found New Hardware Wizard Windows
Step 9 Click Finish.
Step 10 Reboot the server.
a. In Windows, click Start > Shutdown.
You see the screen in Figure 17.
b. Select Restart from the drop-down menu(see Figure 17).
c. Optionally, enter a reason for the system reboot in the Comment field (see Figure 17), if your server
requires it.
d. Click OK.
The system reboots and new driver installation is complete.

17
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Figure 17 Restarting Windows Server
Installing WHQL Drivers on a Windows 2000 or 2003 Server (BoIB and non-BoIB)
WHQL drivers are packaged in a .ZIP file. To install WHQL drivers, see the documentation at
http://www.cisco.com. THERE IS NOTHING HERE??!!
Launching the SFS Chassis Manager (Optional)
SFS Chassis Manager is a web-based GUI that can be used to manage a single InfiniBand chassis from
a Web browser. SFS Chassis Manager supports the following browsers:
•Microsoft Internet Explorer version 6
•Netscape Navigator version 6
•Mozilla version 1.4
To launch SFS Chassis Manager, follow these steps:
Step 1 Type the Management IP address of your switch in the address field of your browser and press Enter.
A login window opens. (See Figure 18.)

18
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Figure 18 SFS Chassis Manager Log-In Window
Step 2 Enter your username and password in the log-in window and click Login .
SFS Chassis Manager loads in your browser window as shown in Figures 19 and 20.
Figure 19 SFS Chassis Manager Opening (Rear View)

19
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Installing the Cisco SFS 7008P Switch into a Rack
Figure 20 SFS Chassis Manager Opening (Front View)
Installing the SFS Element Manager GUI (Optional)
For more information regarding the SFS Element Manager, see the Cisco SFS 7000 Series Product
Family Element Manager User Guide.
To install Element Manager, follow these steps:
Step 1 Ensure that you have sufficient system resources. You need the following:
•32 MB free RAM
•50 MB disk space and 50MB additional temporary space during installation
•300 MHz processor
•800 x 600 screen resolution, 16-bit color depth
Step 2 Download the Element Manager software. To download the latest software, access the Cisco Software
Center at this URL:
http://www.cisco.com/public/sw-center
Step 3 Locate the appropriate install file.
•Linux/install_linux_ia64.bin
•Linux/install_linux_x86.bin
•Windows/install_x86.exe
•Solaris/install_solaris_sparc.bin

20
Cisco SFS 7008P InfiniBand Server Switch Installation and Configuration Note
78-17434-01 Rev. A0
Troubleshooting
Step 4 Run the installation wizard:
a. On Windows, double-click install.exe.
On Solaris and Linux systems, enter sh ./install_yourOS.bin.
b. Follow the onscreen instructions. Enter location and preference information as requested by the
installation program.
Step 5 Start the SFS Element Manager by following these steps:
a. On Windows platforms, click Programs -> Cisco SFS Element Manager -> Cisco SFS EM
On Linux platforms, change to the installation directory and run SFS Element Manager with
./CiscoEM.
b. When prompted to enter a device name, enter the DNS host name or IP address of the Cisco SFS
7008P switch, and community string for the unrestricted user. The default community string for the
unrestricted user is secret.
Troubleshooting
For all customers, partners, resellers, and distributors who hold valid Cisco service contracts, Cisco
Technical Support provides 24-hour-a-day, award-winning technical assistance. The Cisco Technical
Support Website on Cisco.com features extensive online support resources. In addition, Cisco Technical
Assistance Center (TAC) engineers provide telephone support. If you do not hold a valid Cisco service
contract, contact your reseller.
Cisco Technical Support Website
The Cisco Technical Support Website provides online documents and tools for troubleshooting and
resolving technical issues with Cisco products and technologies. The website is available 24 hours a day,
365 days a year, at this URL:
http://www.cisco.com/techsupport
Access to all tools on the Cisco Technical Support Website requires a Cisco.com user ID and password.
If you have a valid service contract but do not have a user ID or password, you can register at this URL:
http://tools.cisco.com/RPF/register/register.do
Note Use the Cisco Product Identification (CPI) tool to locate your product serial number before submitting
a web or phone request for service. You can access the CPI tool from the Cisco Technical Support
Website by clicking the Tools & Resources link under Documentation & Tools. Choose Cisco Product
Identification Tool from the Alphabetical Index drop-down list, or click the Cisco Product
Identification Tool link under Alerts & RMAs. The CPI tool offers three search options: by product ID
or model name; by tree view; or for certain products, by copying and pasting show command output.
Search results show an illustration of your product with the serial number label location highlighted.
Locate the serial number label on your product and record the information before placing a service call.