HP 4X - DDR InfiniBand Mezzanine HCA Guide

Abstract.............................................................................................................................................. 2
Introduction......................................................................................................................................... 2
InfiniBand technology........................................................................................................................... 4
InfiniBand components...................................................................................................................... 5
InfiniBand software architecture ......................................................................................................... 5
MPI............................................................................................................................................. 7
IPoIB ........................................................................................................................................... 7
RDMA-based protocols.................................................................................................................. 7
RDS ............................................................................................................................................ 8
InfiniBand hardware architecture ....................................................................................................... 8
Link operation.................................................................................................................................. 9
Scale-out clusters built on InfiniBand and HP technology......................................................................... 11
Conclusion........................................................................................................................................ 13
Appendix A: Glossary........................................................................................................................ 14
For more information.......................................................................................................................... 15
Call to action .................................................................................................................................... 15
Using InfiniBand for a
scalable compute infrastructure
technology brief, 3rd edition

2
Abstract
With business models constantly changing to keep pace with today’s Internet-based, global economy,
IT organizations are continually challenged to provide customers with high performance platforms
while controlling their cost. An increasing number of enterprise businesses are implementing scale-out
architectures as a cost-effective approach for scalable platforms, not just for high performance
computing (HPC) but also for financial services and Oracle-based database applications.
InfiniBand (IB) is one of the most important technologies that enable the adoption of cluster computing.
This technology brief describes InfiniBand as an interconnect technology used in cluster computing,
provides basic technical information, and explains the advantages of implementing InfiniBand-based
scale-out architectures.
Introduction
The overall performance of enterprise servers is determined by the synergetic relationship between
three main subsystems: processing, memory, and input/output. The multiprocessor architecture used
in the latest single-server systems (Figure 1) provides a high degree of parallel processing capability.
However, multiprocessor server architecture cannot scale cost effectively to a large number of
processing cores. Scale-out cluster computing that builds an entire system by connecting stand-alone
systems with interconnect technology has become widely implemented for HPC and enterprise data
centers around the world.
Figure 1. Architecture of a dual-processor single server (node)
CPU(s)
Memory
Controller
Processor 1
Memory
PCI-X
Bridge
PCIe
Port
Controller
Ultra-3 SCSI
Fibre Channel
Gigabit Ethernet
320 MBps
400 MBps
> 1 Gbps
Memory
Controller
Processor 2
CPU(s)
Processor
Input/Output
InfiniBand
> 10 Gbps*
* 4x link, single data rate

3
Figure 2 shows an example of cluster architecture that integrates computing, storage, and
visualization functions into a single system. Applications are usually distributed to compute nodes
through job scheduling tools.
Figure 2. Sample clustering architecture
Scale-out systems allow infrastructure architects to meet performance and cost goals, but interconnect
performance, scalability, and reliability are key areas that must be carefully considered. A cluster
infrastructure works best when built with an interconnect technology that scales easily, reliably, and
economically with system expansion.
Ethernet is a pervasive, mature interconnect technology that can be cost-effective for some application
workloads. The emergence of 10-Gigabit Ethernet (10GbE) offers a cluster interconnect that meets
higher bandwidth requirements than 1GbE can provide. However, 10GbE still lags the latest
InfiniBand technology in latency and bandwidth performance, and lacks native support for the fat-tree
and mesh topologies used in scale-out clusters. InfiniBand remains the interconnect of choice for
highly parallel environments where applications require low latency and high bandwidth across the
entire fabric.

4
InfiniBand technology
InfiniBand is an industry-standard, channel-based architecture in which congestion-management
capabilities and zero-copy data transfers using remote direct memory access (RDMA) are core
capabilities—resulting in high-speed, low-latency interconnects for scale-out compute infrastructures.
InfiniBand uses a multi-layer architecture to transfer data from one node to another. In the InfiniBand
layer model (Figure 3), separate layers perform different tasks in the message passing process.
The upper layer protocols (ULPs) work closest to the operating system and application; they define the
services and affect how much software overhead the data transfer will require. The InfiniBand
transport layer is responsible for communication between applications. The transport layer splits the
messages into data payloads and encapsulates each data payload and an identifier of the
destination node into one or more packets. Packets can contain data payloads of up to four kilobytes.
The packets are passed to the network layer, which selects a route to the destination node and, if
necessary, attaches the route information to the packets. The data link layer attaches a local identifier
(LID) to the packet for communication at the subnet level. The physical layer transforms the packet into
an electromagnetic signal based on the type of network media—copper or fiber.
Figure 3. Distributed computing using InfiniBand architecture
InfiniBand has these important characteristics:
Very high bandwidth—up to 40Gbps Quad Data Rate (QDR)
Low latency end-to-end communication—MPI ping-pong latency approaching 1 microsecond
Hardware-based protocol handling, resulting in faster throughput and low CPU overhead due to
efficient OS bypass and RDMA
Native support for fat-tree and other common mesh topologies in fabric design
InfiniBand Link

5
InfiniBand components
InfiniBand architecture involves four key components:
Host channel adapter
Subnet manager
Target channel adapter
InfiniBand switch
A host node or server requires a host channel adapter (HCA) to connect to an InfiniBand
infrastructure. An HCA can be a card installed in an expansion slot or integrated onto the host’s
system board. An HCA can communicate directly with another HCA, with a target channel adapter,
or with an InfiniBand switch.
InfiniBand uses subnet manager (SM) software to manage the InfiniBand fabric and to monitor
interconnect performance and health at the fabric level. A fabric can be as simple as a point-to-point
connection or multiple connections through one or more switches. The SM software resides on a node
or switch within the fabric, and provides switching and configuration information to all of the switches
in the fabric. Additional backup SMs may be located within the fabric for failover should the primary
SM fail. All other nodes in the fabric will contain an SM agent that processes management data.
Managers and agents communicate using management datagrams (MADs).
A target channel adapter (TCA) is used to connect an external storage unit or I/O interface to an
InfiniBand infrastructure. The TCA includes an I/O controller specific to the device’s protocol (SCSI,
Fibre Channel, Ethernet, etc.) and can communicate with an HCA or an InfiniBand switch.
An InfiniBand switch provides scalability by allowing a number of HCAs, TCAs, and other IB switches
to connect to an InfiniBand infrastructure. The switch handles network traffic by checking the local link
header of each data packet received and forwarding the packet to the proper destination.
The most basic InfiniBand infrastructure will consist of host nodes or servers equipped with HCAs, an
InfiniBand switch, and subnet manager software. More expansive networks will include multiple
switches.
InfiniBand software architecture
InfiniBand, like Ethernet, uses a multi-layer processing stack to transfer data between nodes.
InfiniBand architecture, however, provides OS-bypass features such as the communication processing
duties and RDMA operations as core capabilities and offers greater adaptability through a variety of
services and protocols.
While the majority of existing InfiniBand clusters operate on the Linux platform, drivers and HCA
stacks are also available for Microsoft® Windows®, HP-UX, Solaris, and other operating systems
from various InfiniBand hardware and software vendors.
The layered software architecture of the HCA allows writing code without specific hardware in mind.
The functionality of an HCA is defined by its verb set, which is a table of commands used by the
application programming interface (API) of the operating system being run. A number of services and
software protocols are available (Figure 4) and, depending on type, can be implemented from user
space or from the kernel.

6
Figure 4. InfiniBand software layers
As indicated in Figure 4, InfiniBand supports a variety of upper level protocols (ULPs) and libraries
that have evolved since the introduction of InfiniBand. The key protocols are discussed in the following
pages.
User APIs
uDAPL
SDP Library
MAD API
Open Fabrics Verbs, CMA, and API
IPoIB
Upper Level Protocols
Provider
User
space
Kernel
space
Hardware-Specific Driver
B
Hardware
MPIs
Open SM
Application Level
Mid-Layer Modules
Connection Manager
SA Client
MAD Services
Diag Tools
RDS
RDMA-based Protocols
(kDAPL, SDP, SRP, iSER, NFS)
InfiniBand HCA
Open Fabrics Verbs and API
SMA
Clustered DB
VNIC
IP-Based Access
Sockets-Based Access
Block Storage Access
File System Access

7
MPI
The message passing interface (MPI) protocol is a library of calls used by applications in a parallel
computing environment to communicate between nodes. MPI calls are optimized for performance in a
compute cluster that takes advantage of high-bandwidth and low-latency interconnects. In parallel
computing environments, code is executed across multiple nodes simultaneously. MPI facilitates the
communication and synchronization among these jobs across the entire cluster.
To take advantage of the features of MPI, an application must be written and compiled to include the
libraries from the particular MPI implementation used. Several implementations of MPI are on the
market:
HP-MPI
Intel MPI
Publicly available versions such as MVAPICH2 and Open MPI
MPI has become the de-facto IB ULP standard. In particular, HP-MPI has been accepted by more
independent software vendors (ISVs) than any other commercial MPI. By using shared libraries,
applications built on HP-MPI can transparently select interconnects that significantly reduce the effort
for applications to support various popular interconnect technologies. HP-MPI is supported on HP-UX,
Linux, True64 UNIX, and Microsoft Windows Compute Cluster Server 2003.
IPoIB
Internet Protocol over InfiniBand (IPoIB) allows the use of TCP/IP or UDP/IP-based applications
between nodes connected to an InfiniBand fabric. IPoIB supports IPv4 or IPv6 protocols and
addressing schemes. An InfiniBand HCA is configured through the operating system as a traditional
network adapter that can use all of the standard IP-based applications such as PING, FTP, and
TELNET. IPoIB does not support the RDMA features of InfiniBand. Communication between IB nodes
using IPoIB and Ethernet nodes using IP will require a gateway/router interface.
RDMA-based protocols
DAPL - The Direct Access Programming Library (DAPL) allows low-latency RDMA communications
between nodes. The uDAPL provides user-level access to RDMA functionality on InfiniBand while the
kDAPL provides the kernel-level API. To use RDMA for the data transfers between nodes, applications
must be written with a specific DAPL implementation.
SDP –Sockets Direct Protocol (SDP) is an RDMA protocol that that operates from the kernel.
Applications must be written to take advantage of the SDP interface. SDP is based on the WinSock
Direct Protocol used by Microsoft server operating systems and is suited for connecting databases to
application servers.
SRP –SCSI RDMA Protocol (SRP) is a data movement protocol that encapsulates SCSI commands over
InfiniBand for SAN networking. Operating from the kernel level, SRP allows copying SCSI commands
between systems using RDMA for low-latency communications with storage systems.
iSER –iSCSI Enhanced RDMA (iSER) is a storage standard originally specified on the iWARP RDMA
technology and now officially supported on InfiniBand. The iSER protocol provides iSCSI
manageability to RDMA storage operations.
NFS –The Network File System (NFS) is a storage protocol that has evolved since its inception in the
1980s, undergoing several generations of development while remaining network-independent. With
the development of high-performance I/O such as PCIe and the significant advances in memory
subsystems, NFS over RDMA on InfiniBand offers low-latency performance for transparent file sharing
across different platforms.

8
RDS
The Reliable Datagram Sockets (RDS) is a low-overhead, low-latency, high-bandwidth transport
protocol that yields high performance for cluster interconnect-intensive environments such as Oracle
Database or Real Application Cluster (RAC). The RDS protocol offloads error-checking operations to
the InfiniBand fabric, freeing more CPU time for application processing.
InfiniBand hardware architecture
InfiniBand fabrics use high-speed, bi-directional serial interconnects between devices. Interconnect
bandwidth and distances are characteristics determined by the type of cabling and connections
employed. The bi-directional links contain dedicated send and receive lanes for full duplex operation.
For quad data rate (QDR) operation, each lane has a signaling rate of 10 Gbps. Bandwidth is
increased by adding more lanes per link. InfiniBand interconnect types include 1x, 4x, or 12x wide
full-duplex links (Figure 5)., The 4x is the most popular configuration and provides a theoretical full-
duplex QDR bandwidth of 80 (2 x 40) gigabits per second.
Figure 5. InfiniBand link types
Encoding overhead in the data transmission process limits the maximum data bandwidth per link to
approximately 80 percent of the signal rate. However, the switched fabric design of InfiniBand allows
bandwidth to grow or aggregate as links and nodes are added. Double data rate (DDR) and
especially quad data rate (QDR) operation increase bandwidth significantly (Table 1).
Table 1. InfiniBand interconnect bandwidth
Link
SDR
Signal rate
DDR
Signal rate
QDR
Signal rate
1x
2.5 Gbps
5 Gbps
10 Gbps
4x
10 Gbps
20 Gbps
40 Gbps
12x
30 Gbps
60 Gbps
120 Gbps
1x Link
(1 send channel,
1 receive channel,
each using differential voltage or
fiber optic signaling)
4x Link
12x Link

9
The operating distances of InfiniBand interconnects are contingent upon the type of cabling (copper or
fiber optic), the connector, and the signal rate. The common connectors used in InfiniBand
interconnects today are CX4 and quad small-form-factor pluggable (QSFP) as shown in Figure 6.
Figure 6. InfiniBand connectors
Fiber optic cable with CX4 connectors generally offers the greatest distance capability. The adoption
of 4X DDR products is widespread, and deployment of QDR systems is expected to increase.
Link operation
Each link can be divided (multiplexed) into a set of virtual lanes, similar to highway lanes (Figure 7).
Each virtual lane provides flow control and allows a pair of devices to communicate autonomously.
Typical implementations have each link accommodating eight lanes1; one lane is reserved for fabric
management and the other lanes for packet transport. The virtual lane design allows an InfiniBand
link to share bandwidth between various sources and targets simultaneously. For example, if a
10Gb/s link were divided into five virtual lanes, each lane would have a bandwidth of 2Gb/s. The
InfiniBand architecture defines a virtual lane mapping algorithm to ensure inter-operability between
end nodes that support different numbers of virtual lanes.
Figure 7. InfiniBand virtual lane operation
1The IBTA specification defines a minimum of two and a maximum of 16 virtual lanes per link.
Sources
Targets
QSFP
CX4

10
When a connection between two channel adapters is established, one of the following transport layer
communication protocols is selected:
Reliable connection (RC) –data transfer between two entities using receive acknowledgment
Unreliable connection (UC) –same as RC but without acknowledgement (rarely used)
Reliable datagram (RD) –data transfer using RD channel between RD domains
Unreliable datagram (UD) –data transfer without acknowledgement
Raw packets (RP) –transfer of datagram messages that are not interpreted
These protocols can be implemented in hardware; some protocols more efficient than others. The UD
and Raw protocols, for instance, are basic datagram movers and may require system processor
support depending on the ULP used.
When the reliable connection protocol is operating (Figure 8), hardware at the source generates
packet sequence numbers for every packet sent, and the hardware at the destination checks the
sequence numbers and generates acknowledgments for every packet sequence number received. The
hardware also detects missing packets, rejects duplicate packets, and provides recovery services for
failures in the fabric.
Figure 8. Link operation using reliable connection protocol