Qlogic InfiniPath User manual

IB0056101-00 C Page i
Q
Simplify
InfiniPath Install Guide
Version 2.0

InfiniPath Install Guide
Version 2.0 Q
Page ii IB0056101-00 C
Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no
responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its
use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications
described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no
representation nor warranty that such applications are suitable for the specified use without further testing or modification.
QLogic Corporation assumes no responsibility for any errors that may appear in this document.
No part of this document may be copied nor reproduced by any means, nor translated nor transmitted to any magnetic
medium without the express written consent of QLogic Corporation. In accordance with the terms of their valid PathScale
agreements, customers are permitted to make electronic and paper copies of this document for their own exclusive use.
Linux is a registered trademark of Linus Torvalds.
QLA, QLogic, SANsurfer, the QLogic logo, PathScale, the PathScale logo, InfiniPath and EKOPath are registered
trademarks of QLogic Corporation.
Red Hat and all Red Hat-based trademarks are trademarks or registered trademarks of Red Hat, Inc.
SuSE is a registered trademark of SuSE Linux AG.
All other brand and product names are trademarks or registered trademarks of their respective owners.
Document Revision History
Rev. 1.0, 8/20/2005
Rev. 1.1, 11/15/05
Rev. 1.2,02/15/06
Rev. 1.3 Beta 1, 4/15/06
Rev. 1.3, 6/15/06
Rev. 2.0 Beta, 8/15/06
QLogic Version Number IB0056101-00 A
Rev. 2.0 Beta2 10/15/06
QLogic Version Number IB0056101-00 B
Rev. 2.0 11/30/06
QLogic Version Number IB0056101-00 C
Changes Document Sections Affected
Updated Introduction so that most information is contained in modules.
The same info is contained in the User Guide Introduction.
section 1
Added cable management advisory section 2.6
Updated some sections for clarity. Fixed spacing errors. All
Added SLES 9 to list of distros that use openssh package. section 3.2.2
Added SLES 9 to list of RPM identifiers section 3.3,section 3.4
Updated tarball name section 3.5.1
Added new install directory name for kernel modules section 3.5.2
Updated install directory name for kernel modules section 3.13
Added section on MPI over uDAPL section 3.15

InfiniPath Install Guide
Version 2.0
Q
IB0056101-00 C Page iii
© 2006 QLogic Corporation. All rights reserved worldwide.
© PathScale 2004, 2005, 2006. All rights reserved.
First Published: August 2005
Printed in U.S.A.

InfiniPath Install Guide
Version 2.0 Q
Page iv IB0056101-00 C

InfiniPath Install Guide
IB0056101-00 C Page v
QInfiniPath Install Guide
Version 2.0
Table of Contents
Section 1 Introduction
1.1 Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
1.2 How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
1.3 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
1.4 Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
1.5 Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
1.6 What’s New in this Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
1.7 Supported Distributions and Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4
1.8 Software Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4
1.9 Conventions Used in this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-6
1.10 Documentation and Technical Support. . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7
Section 2 Hardware Installation
2.1 Hardware Installation Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
2.1.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
2.1.1.1 Form Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
2.1.2 Cabling and Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-3
2.1.3 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-3
2.1.4 Configuring the BIOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-3
2.2 Safety with Electricity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4
2.3 Unpacking Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5
2.3.1 Verify Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5
2.3.2 List of the Package Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5
2.3.3 Unpacking the InfiniPath Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-7
2.4 Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8
2.4.1 Hardware Installation for QMI7140 . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8
2.4.2 Hardware Installation for QLE7140 with PCIe Riser . . . . . . . . . . . . . . 2-8
2.4.3 Hardware Installation for QHT7140 with HTX Riser . . . . . . . . . . . . . . 2-11
2.5 Hardware Installation for QHT7140 without HTX Riser . . . . . . . . . . . . . . 2-14
2.6 Cabling the Adapter to the InfiniBand Switch . . . . . . . . . . . . . . . . . . . . . . 2-16
2.6.1 Standard InfiniBand Copper Cabling . . . . . . . . . . . . . . . . . . . . . . . . . . 2-16
2.6.1.1 Cabling for the QMI7140 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-17
2.6.2 Optical Fibre Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-17
2.7 Completing the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-17

InfiniPath Install Guide
Version 2.0
Page vi IB0056101-00 C
Q
Section 3 Software Installation
3.1 Software Installation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
3.2 Software Installation Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
3.2.1 Linux Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3
3.2.2 Environment for InfiniPath and OpenFabrics . . . . . . . . . . . . . . . . . . . . 3-3
3.2.3 Kernel Patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
3.2.4 Types of Nodes in a Cluster Environment . . . . . . . . . . . . . . . . . . . . . . 3-4
3.3 InfiniPath Software RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5
3.4 OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5
3.5 Downloading and Installing InfiniPath and OpenFabrics Software . . . . . . 3-6
3.5.1 Downloading the tar File Containing the RPMs . . . . . . . . . . . . . . . . . 3-6
3.5.2 Installing the Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-7
3.5.3 Installed Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8
3.5.4 Installing in an Alternate Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-9
3.5.4.1 Installing for Your Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-9
3.5.4.2 Check for Missing Files or RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-10
3.6 Managing and Installing Software Using Rocks . . . . . . . . . . . . . . . . . . . . 3-10
3.6.1 Installing Rocks and InfiniPath RPMs . . . . . . . . . . . . . . . . . . . . . . . . . 3-11
3.6.2 Installing Rocks and InfiniPath RPMs-Details . . . . . . . . . . . . . . . . . . . 3-11
3.6.3 Further Information on Rocks and kickstart . . . . . . . . . . . . . . . . . . . . . 3-14
3.7 InfiniPath and OpenFabrics Driver Overview . . . . . . . . . . . . . . . . . . . . . . 3-15
3.8 Configuring the InfiniPath Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-16
3.8.1 Configuring the ib_ipath Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-16
3.8.2 Configuring the ipath_ether Network Interface . . . . . . . . . . . . . . . . 3-16
3.8.2.1 ipath_ether Configuration on Fedora and RHEL4 . . . . . . . . . . . 3-17
3.8.2.2 ipath_ether Configuration on SUSE 9.3, SLES 9, and SLES 10 3-18
3.9 OpenFabrics Configuration and Startup . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20
3.9.1 Configuring the IPoIB Network Interface . . . . . . . . . . . . . . . . . . . . . . . 3-21
3.10 OpenSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-22
3.11 SRP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-22
3.12 Starting and Stopping the InfiniPath Software . . . . . . . . . . . . . . . . . . . . . 3-23
3.13 Rebuilding or Reinstalling the Drivers. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-24
3.14 Further Information on Configuring and Loading Drivers . . . . . . . . . . . . . 3-24
3.15 MPI Over uDAPL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-25
3.16 Compiler Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-25
3.17 Switch Configuration and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-25
3.18 LED Link and Data Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-26
3.19 Customer Acceptance Utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-26
3.20 Removing Software Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-28

InfiniPath Install Guide
IB0056101-00 C Page vii
QInfiniPath Install Guide
Version 2.0
3.20.1 Downgrading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-28
3.21 Installing Newer Drivers from Other Distributions. . . . . . . . . . . . . . . . . . . 3-28
3.22 Installation Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-29
Appendix A RPM Descriptions
A.1 InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1
A.2 Different Nodes May Use Different RPMs . . . . . . . . . . . . . . . . . . . . . . . . A-1
A.3 Documentation and InfiniPath RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2
A.4 OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-4
Figures
Figure Page
2-1 InfiniPath QLE7140 Card with Riser, Top View. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-6
2-2 InfiniPath QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View . . . 2-7
2-3 PCIe Slot in a Typical Motherboard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-9
2-4 InfiniPath QLE7140 Assembly with Riser Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-10
2-5 Assembled QLE7140 with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-11
2-6 HTX Slot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-12
2-7 InfiniPath Adapter with Riser Card. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-13
2-8 Assembled QHT7140 with Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-14
2-9 QHT7140 without Riser Installed in a 3U Chassis. . . . . . . . . . . . . . . . . . . . . . . . . . . 2-15
3-1 Relationship between InfiniPath and OpenFabrics Software . . . . . . . . . . . . . . . . . . 3-15
Tables
Table Page
1-1 PathScale-QLogic Adapter Model Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
1-2 InfiniPath/OpenFabrics Supported Distributions and Kernels . . . . . . . . . . . . . . . . . . 1-4
1-3 Typographical Conventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-6
2-1 Adapter Models and Related Platforms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
3-1 InfiniPath/OpenFabrics Supported Distributions and Kernels . . . . . . . . . . . . . . . . . . 3-3
3-2 InfiniPath RPMs to Use for Each Node in a Cluster . . . . . . . . . . . . . . . . . . . . . . . . . 3-5
3-3 OpenFabrics RPMs to Use for Each Node in a Cluster. . . . . . . . . . . . . . . . . . . . . . . 3-5
3-4 LED Link and Data Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-26
A-1 Documentation/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2
A-2 InfiniPath-Devel/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2
A-3 InfiniPath/RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3
A-4 InfiniPath/RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3
A-5 InfiniPath-MPI/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3

InfiniPath Install Guide
Version 2.0
Page viii IB0056101-00 C
Q
A-6 OpenFabrics/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-4
A-7 OpenFabrics-Devel/RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-5
A-8 OpenSM/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-6
A-9 OtherHCAs/RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-7

IB0056101-00 C 1-1
Section 1
Introduction
This chapter describes the contents, intended audience, and organization of the
InfiniPath Install Guide.
The InfiniPath Install Guide contains instructions for installing the QLogic InfiniPath
Interconnect hardware adapters and the InfiniPath and OpenFabrics software. The
adapters covered in this guide are the QLE7140, the QLogic InfiniPath PCI
Express™ adapter, the the QMI7140, which runs on Power PC systems, particularly
the IBM® BladeCenter H processor blades, and the QHT7040/QHT7140 HTX™
adapters.
For more information on system administration, using InfiniPath MPI, and
troubleshooting adapter hardware and software, see the InfiniPath Interconnect
User Guide.
1.1
Who Should Read this Guide
This installation guide is intended for cluster administrators responsible for installing
the InfiniPath QMI7140, the QLE7140, or the InfiniPath QHT7040/QHT7140 product
on your Linux cluster. More detailed installation information and instructions for
administration of the InfiniPath cluster are provided in the InfiniPath Interconnect
User Guide.
This guide assumes you are familiar with cluster networking and with the specific
hardware to which your processors are connected. Prior to installing the InfiniPath
Adapter, you should have basic knowledge of your host and target operating
systems, and working knowledge of message passing concepts.
This document does not contain all the information you need in order to use basic
Linux commands or to perform all system administration tasks. For this information,
see the software documentation that you received with your system.
1.2
How this Guide is Organized
The InfiniPath Install Guide is organized into these sections:
■Section 1, "Introduction"
■Section 2, "Hardware Installation", instructions for installing the InfiniPath
QLE7140, QMI7140, and QHT7040 or QHT7140 hardware
■Section 3, "Software Installation", instructions for installing the QLogic InfiniPath
and OpenFabrics software

1 – Introduction
Interoperability
1-2 IB0056101-00 C
Q
■Appendix A, "RPM Descriptions"
1.3
Overview
The material in this documentation pertains to an InfiniPath cluster. This is defined
as a collection of nodes, each attached to an InfiniBand™-based fabric through the
InfiniPath Interconnect. The nodes are Linux-based computers, each having up to
eight processors.
The InfiniPath interconnect is InfiniBand 4X, with a raw data rate of 10 Gb/s (data
rate of 8Gb/s).
InfiniPath utilizes standard, off-the-shelf InfiniBand 4X switches and cabling.
InfiniPath OpenFabrics software is interoperable with other vendors’ InfiniBand
HCAs running compatible OpenFabrics releases. There are two options for Subnet
Management in your cluster:
■Use the Subnet Manager on one or more managed switches supplied with your
Infiniband switches.
■Use the OpenSM component of OpenFabrics.
1.4
Switches
The InfiniPath interconnect is designed to work with all InfiniBand-compliant
switches. Use of OpenSM as a subnet manager is now supported. OpenSM is part
of the OpenFabrics component of this release.
1.5
Interoperability
InfiniPath participates in the standard InfiniBand Subnet Management protocols for
configuration and monitoring. InfiniPath OpenFabrics (including IPoIB) is
interoperable with other vendors’ InfiniBand HCAs running compatible OpenFabrics
releases. The InfiniPath MPI and Ethernet emulation stacks (ipath_ether) are not
interoperable with other InfiniBand Host Channel Adapters (HCA) and Target
Channel Adapters (TCA). Instead, InfiniPath uses an InfiniBand-compliant
vendor-specific protocol that is highly optimized for MPI and TCP between
InfiniPath-equipped hosts.
NOTE: OpenFabrics was known as OpenIB until March 2006. All relevant
references to OpenIB in this documentation have been updated to reflect
this change. See the OpenFabrics website at http://www.openfabrics.org
for more information on the OpenFabrics Alliance.

1 – Introduction
What’s New in this Release
IB0056101-00 C 1-3
Q
1.6
What’s New in this Release
QLogic Corp. acquired PathScale in April 2006. In this 2.0 release, product names,
internal program and output message names now refer to QLogic rather than
PathScale.
The new QLogic and former PathScale adapter model numbers are shown in the
table below.
This version of InfiniPath provides support for all QLogic’s HCAs, including:
■InfiniPath QLE7140, which is supported on systems with PCIe x8 or x16 slots
■InfiniPath QMI7140, which runs on Power PC systems, particularly on the IBM®
BladeCenter H processor blades
■InfiniPath QHT7040 and QHT7140, which leverage HTX™. The InfiniPath
QHT7040 and QHT7140 are exclusively for motherboards that support
HTXcards. The QHT7140 has a smaller form factor than the QHT7040, but is
otherwise the same. Unless otherwise stated, QHT7140 will refer to both the
QHT7040 and QHT7140 in this documentation.
Expanded MPI scalability enhancements for PCI Express have been added. The
QHT7040 and QHT7140 can support 2 processes per context for a total of 16. The
QLE7140 and QMI7140 also support 2 processes per context, for a total of 8.
Support for multiple versions of MPI has been added. You can use a different version
of MPI and achieve the high-bandwidth and low-latency performance that is
standard with InfiniPath MPI.
Table 1-1. PathScale-QLogic Adapter Model Numbers
Former
PathScale
Model Number
New QLogic Model
Number Description
HT-400 IBA6110 Single Port 10GBS InfiniBand to HTX ASIC
ROHS
PE-800 IBA6120 Single Port 10GBS InfiniBand to x8 PCI Express
ASIC ROHS
HT-460 QHT7040 Single Port 10GBS InfiniBand to HTX Adapter
HT-465 QHT7140 Single Port 10GBS InfiniBand to HTX Adapter
PE-880 QLE7140 Single Port 10GBS InfiniBand to x8 PCI Express
Adapter
PE-850 QMI7140 Single Port 10GBS InfiniBand IBM Blade Center
Adapter

1 – Introduction
Software Components
1-4 IB0056101-00 C
Q
Also included is expanded operating system support, and support for the latest
OpenFabrics software stack.
Multiple InfiniPath cards per node are supported. A single software installation
works for all the cards.
Additional up-to-date information can be found on the QLogic web site:
http://www.qlogic.com
1.7
Supported Distributions and Kernels
The InfiniPath interconnect runs on AMD Opteron, Intel EM64T, and IBM Power
Blade Center H) systems running Linux. The currently supported distributions and
associated Linux kernel versions for InfiniPath and OpenFabrics are listed in the
following table. The kernels are the ones that shipped with the distributions, unless
otherwise noted.
NOTE: IBM Power systems run only with the SLES 10 distribution.
The SUSE10 release series is no longer supported as of this InfiniPath 2.0 release.
Fedora Core 4 kernels prior to 2.6.16 are also no longer supported.
1.8
Software Components
The software provided with the InfiniPath Interconnect product consists of:
■InfiniPath driver (including OpenFabrics)
■InfiniPath ethernet emulation
Table 1-2. InfiniPath/OpenFabrics Supported Distributions and Kernels
Distribution
InfiniPath/OpenFabrics supported
kernels
Fedora Core 3 (FC3) 2.6.12 (x86_64)
Fedora Core 4 (FC4) 2.6.16, 2.6.17 (x86_64)
Red Hat Enterprise Linux 4 (RHEL4) 2.6.9-22, 2.6.9-34, 2.6.9-42(U2/U3/U4)
(x86_64)
CentOS 4.2-4.4 (Rocks 4.2-4.4) 2.6.9 (x86_64)
SUSE Linux 9.3 (SUSE 9.3) 2.6.11 (x86_64)
SUSE LInux Enterprise Server (SLES 9) 2.6.5 (x86_64)
SUSE LInux Enterprise Server (SLES 10) 2.6.16 (x86_64 and ppc64)

1 – Introduction
Software Components
IB0056101-00 C 1-5
Q
■InfiniPath libraries
■InfiniPath utilities, configuration, and support tools
■InfiniPath MPI
■InfiniPath MPI benchmarks
■OpenFabrics protocols, including Subnet Management Agent
■OpenFabrics libraries and utilities
OpenFabrics kernel module support is now built and installed as part of the InfiniPath
RPM install. The InfiniPath release 2.0 runs on the same code base as OpenFabrics
Enterprise Distribution (OFED) version 1.1. It also includes the OpenFabrics
1.1-based library and utility RPMs. InfiniBand protocols are interoperable between
InfiniPath 2.0 and OFED 1.1.
This release provides support for the following protocols:
■IPoIB (TCP/IP networking)
■SDP (Sockets Direct Protocol)
■OpenSM
■UD (Unreliable Datagram)
■RC (Reliable Connection)
■UC (Unreliable Connection)
■SRQ (Shared Receive Queue)
■uDAPL (user Direct Access Provider Library)
This release includes a technology preview of:
■SRP (SCSI RDMA Protocol)
Future releases will provide support for:
■iSER (iSCSI Extensions for RDMA)
No support is provided for RD.
NOTE: 32 bit OpenFabrics programs using the verb interfaces are not supported
in this InfiniPath release, but will be supported in a future release.
More details about the hardware and software are found in Section 2 and Section 3.

1 – Introduction
Conventions Used in this Guide
1-6 IB0056101-00 C
Q
1.9
Conventions Used in this Guide
This Guide uses these typographical conventions:
Table 1-3. Typographical Conventions
Convention Meaning
command Fixed-space font is used for literal items such as commands,
functions, programs, files and pathnames, and program
output;
variable Italic fixed-space font is used for variable names in programs
and command lines.
concept Italic font is used for emphasis, concepts.
user input Bold fixed-space font is used for literal items in commands or
constructs that you type in.
$Indicates a command line prompt.
#Indicates a command line prompt as root when using bash or
sh.
[ ] Brackets enclose optional elements of a command or
program construct.
... Ellipses indicate that a preceding element can be repeated.
> Right caret identifies the cascading path of menu commands
used in a procedure.
2.0 The current version number of the software is included in the
RPM names and within this documentation.
NOTE: Indicates important information.

1 – Introduction
Documentation and Technical Support
IB0056101-00 C 1-7
Q
1.10
Documentation and Technical Support
The InfiniPath product documentation includes:
■The InfiniPath Install Guide
■The InfiniPath Interconnect User Guide
■Release Notes
■Quick Start Guide
■Readme file
The Troubleshooting Appendix for installation, InfiniPath and OpenFabrics
administration, and MPI issues is located in the InfiniPath Interconnect User Guide.
Visit the QLogic support Web site for documentation and the latest software
updates.
http://www.qlogic.com

1 – Introduction
Documentation and Technical Support
1-8 IB0056101-00 C
Q

IB0056101-00 C 2-1
Section 2
Hardware Installation
This chapter lists the requirements and provides instructions for installing the
QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic
InfiniPath PCI Express Adapter and PCIe riser card, QLE7140, the QMI7140 for
IBM Blade Center H processor blades, and the InfiniPath QHT7040 or QHT7140
adapter hardware and HTX riser card. These components will also be referred to
as the adapter and the riser card in the remainder of this Guide.
The adapter is a low-latency, high-bandwidth, high message rate cluster
interconnect for InfiniBand. The InfiniPath interconnect is InfiniBand 4X, with a raw
data rate of 10Gb/s (data rate of 8Gb/s).
OpenFabrics is interoperable with other vendors’ InfiniBand HCAs running
compatible OpenFabrics releases.
2.1
Hardware Installation Requirements
This section lists hardware and software environment requirements for installing
the InfiniPath QLE7140, InfiniPath QMI7140, and the InfiniPath QHT7040 or
QHT7140.
2.1.1
Hardware
QLogic InfiniPath interconnect adapters are for use with UL Listed Computers. The
following statement is true for all the adapters:
This device complies with part 15 of the FCC Rules. Operation is subject to the
following two conditions: (1) This device may not cause harmful interference,
and (2) this device must accept any interference received, including interference
that may cause undesired operations.
Different adapter cards work on different platforms. The following table shows the
relationship between the adapter model and different types of motherboards.

2 – Hardware Installation
Hardware Installation Requirements
2-2 IB0056101-00 C
Q
Installation of the QLE7140, QHT7040 or QHT7140 in a 1U or 2U chassis requires
the use of a riser card. See figure 2-3 for an illustration of a PCI Express slot in a
typical motherboard. See figure 2-6 for an illustration of an HTX slot for a typical
Opteron motherboard.
Note that the motherboard vendor is the optimal source for information on the layout
and use of HyperTransport and PCIe-enabled expansion slots on supported
motherboards. For the most up-to-date listing of InfiniPath PCIe and HTX Adapter
model numbers and the motherboards in which they are supported, please go to
our web site:
http://www.qlogic.com
2.1.1.1
Form Factors
The QMI7140 is the model number for the adapter card that ships in the IBM CFFE
form factor. It does not use a faceplate.
The QLE7140 is the model number for the adapter card that ships in the PCIe low
profile short form factor. This card can be used with either a full height or low profile
faceplate.
The QHT7040 is the model number for the adapter card that ships in the HTX full
height short form factor. The HTX low profile form factor is referred to as the
QHT7140. It is the same as the QHT7040, except for its more compact size. In
either case, the adapter is backward and forward compatible for the motherboards
in which it is supported. Unless otherwise stated, the HTX adapter is referred to as
the QHT7140 in this documentation.
Table 2-1. Adapter Models and Related Platforms
QLogic Model
Number Platform Plugs Into
QMI7140 Blade Center H
processor blades
CFFE Connectora
QLE7140 PCI Express
systems
Standard PCI Express x8 or x16 slot
QHT7040 Motherboards with
HTX connectors
HyperTransport HTX slot
QHT7140 Motherboards with
HTX connectors
HyperTransport HTX slot
Table Notes
aSee the IBM documentation for information on the location of the CFFE connector. The
IBM web site is: http://www.ibm.com

2 – Hardware Installation
Hardware Installation Requirements
IB0056101-00 C 2-3
Q
When the QHT7040 or QHT7140 adapter is installed with the riser card, it may
prevent some or all of the other PCI expansion slots from being used. This depends
upon the form factor of the adapter and motherboard.
Run ipath_control -i to see information on which form factor adapter is installed.
The file /sys/bus/pci/drivers/ib_ipath/00/boardversion contains the same
information. For more information, see section C.9 in the Troubleshooting Appendix
in the InfiniPath Interconnect User Guide.
2.1.2
Cabling and Switches
InfiniPath utilizes standard, off-the-shelf InfiniBand cabling and 4X switches. The
InfiniPath interconnect is designed to work with all InfiniBand-compliant switches.
A standard InfiniBand copper cable up to a length of twenty meters is required. See
section 2.6.1 for more information. Optical fibre with optical adapters such as the
Emcore is also supported. See section 2.6.2.
There are two options for Subnet Management in your cluster:
■Use the Subnet Manager on one or more managed switches supplied with your
Infiniband switches.
■Use OpenSM component of OpenFabrics.
Cabling instructions are found in this Guide in section 2.6.
2.1.3
Software
Before installing the InfiniPath software, complete the hardware installation as
described in this chapter.
Verify that each machine in the cluster is running Linux kernel software from a
supported distribution.
See section 3.2 for information on supported distributions, associated kernels and
other software requirements.
2.1.4
Configuring the BIOS
A properly configured BIOS is required. The BIOS settings, which are stored in
non-volatile memory, contain certain parameters characterizing the system,. These
parameters may include date and time, configuration settings, and information
about the installed hardware.
There are currently two issues concerning BIOS settings that you need to be aware
of:

2 – Hardware Installation
Safety with Electricity
2-4 IB0056101-00 C
Q
■ACPI needs to be enabled
■MTRR mapping needs to be set to “Discrete”
MTRR (Memory Type Range Registers) is used by the InfiniPath driver to enable
write combining to the InfiniPath on-chip transmit buffers. This improves write
bandwidth to the InfiniPath chip by writing multiple words in a single bus transaction
(typically 64). This applies only to x86_64 systems.
However, some BIOSes don’t have the MTRR mapping option. It may be referred
to in a different way, dependent upon chipset, vendor, BIOS, or other factors. For
example, it is sometimes referred to as "32 bit memory hole", which should be
enabled.
If there is no setting for MTRR mapping or 32 bit memory hole, please contact your
system or motherboard vendor and inquire as to how write combining may be
enabled.
ACPI and MTRR mapping issues are discussed in greater detail in the
Troubleshooting section of the InfiniPath Interconnect User Guide.
NOTE: BIOS settings on IBM Blade Center H (Power) systems do not need
adjustment.
You can check and adjust these BIOS settings using the BIOS Setup Utility. For
specific instructions on how to do this, follow the hardware documentation that came
with your system.
2.2
Safety with Electricity
Observe these guidelines and safety precautions when working around computer
hardware and electrical equipment:
1. Locate the power source shutoff for the computer room or lab where you are
working. This is where you will turn OFF the power in the event of an emergency
or accident. Never assume that power has been disconnected for a circuit.
Always check first.
2. Don’t wear loose clothing. Fasten your tie or scarf, remove jewelry, and roll up
your sleeves. Wear safety glasses when working under any conditions that
might be hazardous to your eyes.
3. Shut down and disconnect the system’s power supply from AC service before
you begin work, to insure that standby power is not active. Power off all attached
devices such as monitors, printers, and external components. Note that many
motherboards and power supplies maintain standby power at all times.
Inserting or removing components while standby is active can damage them.
4. Use normal precautions to prevent electrostatic discharge, which can damage
integrated circuits.