IBM Flex System EN4132 User manual

IBM Flex System EN4132 2-
port 10Gb Ethernet Adapter
User’s Guide

ii IBM Flex System EN4132 2-port 10Gb Ethernet Adapter User’s Guide

IBM Flex System EN4132 2-
port 10Gb Ethernet Adapter
User’s Guide

Note: Before using this information and the product it supports, read the general information in Appendix B, “Notices” on page 23, the Safety In-
formation and Environmental Notices and User's Guide documents on the IBM Notices for Network Devices CD, and the Warranty Information
document that comes with the product.
First Edition February 2012
© Copyright IBM Corporation 2012.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.

v
© Copyright IBM Corp. 2012
Contents
Safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Chapter 1. IBM Flex System EN4132 2-port 10Gb Ethernet adapter . . . . . . . . . . 1
Related documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Notices and statements in this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Features and specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Finding the MAC and serial number on the adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Chapter 2. Installing the IBM Flex System adapter. . . . . . . . . . . . . . . . . . . . . . . . 5
Installing device drivers and verification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Chapter 3. Solving problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Parts listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Appendix A. Getting help and technical assistance . . . . . . . . . . . . . . . . . . . . . 19
Before you call . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Using the documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Getting help and information from the World Wide Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Software service and support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Hardware service and support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
IBM Taiwan product service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Appendix B. Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Important notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Document format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Electronic emission notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

vi IBM Flex System EN4132 2-port 10Gb Ethernet Adapter User’s Guide

vii
© Copyright IBM Corp. 2012
Safety

viii

ix IBM Flex System EN4132 2-port 10Gb Ethernet Adapter User’s Guide

1
© Copyright IBM Corp. 2012
Chapter 1. IBM Flex System EN4132 2-port 10Gb
Ethernet adapter
The IBM Flex System EN4132 2-port 10Gb Ethernet adapter provides the highest
performing and most flexible interconnect solution for servers used in Enterprise
Data Centers, High-Performance Computing, and Embedded environments.
This User's Guide contains setup and installation instructions for the IBM Flex
System EN4132 2-port 10Gb Ethernet adapter and general information about the
adapter, including how to install drivers and how to get help.
For information about the types of compatible devices available for IBM products,
contact your IBM marketing representative or authorized reseller. For a list of
supported optional devices, see
http://www.ibm.com/servers/eserver/serverproven/compat/us/.
The warranty document for this product comes with the network device, and it
contains information about the terms of the warranty.
For the most up-to-date product documentation for all of your IBM Flex System
products, go to the IBM Flex System Information Center at
http://publib.boulder.ibm.com/infocenter/flexsys/information/index.jsp.
Related documentation
This User's Guide contains setup and installation instructions for the IBM Flex
System adapter and general information about the adapter and how to get help.
The most recent version of this User's Guide and all other related documents are
at http://publib.boulder.ibm.com/infocenter/flexsys/information/index.jsp.
In addition to this User's Guide, the following related documentation is available
for your adapter:
•IBM Flex System Installation and Service Guide
The Installation and Service Guide that comes with the IBM compute node contains
the instructions for installing the adapter in the compute node.
•Mellanox OFED for Linux User Manual
• This document is available at www.mellanox.com, and it provides information about
the Mellanox OFED software. Mellanox OFED is a single Virtual Protocol Interconnect
(VPI) software stack that is based on the OpenFabrics (OFED) Linux stack.
• MLNX_VPI for Windows User Manual
This README document is available at www.mellanox.com, and it provides informa-
tion about the Mellanox WinOF VPI driver Rev 2.1.3 package, which is distributed for
Windows Server 2008 (x86 and x64), Windows Server R2 (x64), and Windows 7 (x86
and x64).
•MLNX_VPI for Windows Installation Guide
This document is available at www.mellanox.com, and it provides information about
installing and testing MLNX_VPI for Windows on a single host machine that has an
IBM IB6132 FDR InfiniBand adapter installed.

2IBM Flex System EN4132 2-port 10Gb Ethernet Adapter User’s Guide
IBM Flex System Notices for Network Devices CD that comes with the adapter
contains the following documentation in portable document format (PDF):
•IBM Flex System Network Devices Basic Troubleshooting Information
This multilingual document contains basic troubleshooting information and a replace-
ment parts list for the switches, adapters,and pass-thru modules. The product docu-
mentation that comes with your IBM chassis, compute node, or network device might
contain more detailed troubleshooting information.
•IBM Safety Information
This multilingual document contains the translated versions of the safety messages
that appear in the IBM documentation.
•IBM Environmental Notices and User's Guide
This multilingual document contains the world wide environmental and recycling
information for your product.
•IBM License Agreement for Machine Code
This multilingual document contains the IBM license information for the machine
code for your product.
•IBM Warranty Information
This multilingual document is provided on the CD and in printed format. It contains
the IBM warranty information for your product.
For the most up-to-date documentation for all of the IBM Flex System products,
go to the IBM Flex System Information Center at http://publib.boulder.ibm.com/
infocenter/flexsys/information/index.jsp.
Notices and statements in this document
Notices for Network Device

3
IBM Flex System EN4132 2-port 10Gb Ethernet adapter
Features and specifications
The IBM Flex System EN4132 2-port 10Gb Ethernet adapter has the following fea-
tures and specifications:
• PCI Express 3.0 (1.1 and 2.0 compatible) through an x8 edge connector up to 8GT/s
• 10 Gb/s Ethernet
• CPU off-load of transport operations
• CORE-Direct® application off-load
• GPUDirect application off-load
• RDMA over Converged Ethernet (RoCE)
• End-to-end QoS and congestion control
• Hardware-based I/O virtualization
• TCP/UDP/IP stateless off-load
• Ethernet encapsulation (EoIB)
• RoHS-6 compliant
Table 1 - IBM Flex System
Physical Size: 3.3in. x 3.8 in. (84.89mm x 96.7 mm)
Protocol Support
Ethernet: 10GBASE-CX4, 10GBASE-R, and 1000BASE-R
Data Rate: 1/10Gb/s – Ethernet
PCI Express Gen3: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1 compatible)
Regulatory
EMC: Refer to the following link:
www.mellanox.com/related-docs/user_manuals/
Regulatory_and_Compliance_Guide.pdf
Safety: IEC/EN 60950-1:2006
ETSI EN 300 019-2-2
IEC 60068-2- 64, 29, 32
RoHS: RoHS-R6
EN4132 2-port 10Gb Ethernet adapter specifications table

4IBM Flex System EN4132 2-port 10Gb Ethernet Adapter User’s Guide
Finding the MAC and serial number on the adapter
Each adapter has a label on the print side that shows the adapter serial number
and the MAC for the Ethernet protocol.
Figure 1: Board Label 1
Figure 2: Board Label 2

5
Installing the IBM Flex System adapter
Chapter 2. Installing the IBM Flex System adapter
This chapter provides information and instructions for installing the adapter. See
the documentation that came with the IBM Flex System compute node for the
instructions needed to install the adapter in the node; then, return to this User's
Guide for the information and instructions needed to complete the installation.
Installation of this adapter should only be done by a properly qualified technician
or engineer. Read and follow all safety precautions specified in this document.
Use the documentation supplied with the IBM Flex System chassis to remove and
replace the compute node from the chassis.
1. Remove the cover from the compute node
a. With the compute node sitting on the work table slide the release button,
located on the right-side of the compute node, forward while pressing it
down as well. This action releases the cover of the compute node.
b. Lift the cover from the compute node.
Figure 3: Lift Off Cover
2. The adapter can be placed in either of the two available slots.

6
© Copyright IBM Corp. 2012
Figure 4: Compute Node
B
Switch Bay
3-4
A
Switch Bay
1-2
3. Line up the adapter so that the pins of the adapter are over the sockets in the
compute node.
Figure 5: IBM Flex System Adapter
4. Plug the adapter into the socket print-side up.

7
Installing the IBM Flex System adapter
Figure 6: Plug the Adapter
5. Close the retaining clips.
Figure 7: Retaining Clips
6. Replace the compute node cover.

8IBM Flex System EN4132 2-port 10Gb Ethernet Adapter User’s Guide
Installing device drivers and verification
The following provides brief device driver installation instructions. For more on
device driver installation instructions, refer to the following operating system-
specific documentation:
www.mellanox.com => Products => Software/Drivers => InfiniBand and VPI SW/
Drivers => Linux SW/Drivers => MLNX_OFED
AIX Version 7.1 with the 7100-01 Technology Level Service Pack 3
• AIX device driver software - this adapter device driver is included in the following
AIX releases or later levels. Refer to the installation instructions for the specific AIX
release to install or update driver software.
www.mellanox.com => Products => Software/Drivers => InfiniBand and VPI SW/
Drivers => Windows SW/Drivers => MLNX_WinOF VPI
www.mellanox.com => Products => Software/Drivers => InfiniBand and VPI SW/
Drivers => VMware Drivers
Linux drivers
For Linux, download and install the latest OpenFabrics Enterprise Distribution
(OFED) software package available via the Mellanox web site at: http://www.mel-
lanox.com => Products => Software/Drivers => InfiniBand & VPI Software/Drivers
=> Mellanox OFED => Download. Follow the installation instructions included in
the download package (also available from the download page). To ensure that
communication has been established follow the instructions below.
Get the device location
Get the device location on the PCI bus by running lspci and locating lines with the
string “Mellanox Technologies”:
Check the OFED version
To get the version of the running Mellanox OFED/BXOFED, run the following
command:
Load the Ethernet Driver
By default, the Mellanox OFED stack loads mlx4_en. Run ‘lsmod’ to verify that
the module is listed.
> lspci |grep -i Mellanox
07:00.0 Network controller: Mellanox Technologies MT27500 Fam-
ily [ConnectX-3]
> # ofed_info |head -1
MLNX_OFED_LINUX-1.5.3-2.1.0 (OFED-1.5.3-2.1.0)

9
Installing the IBM Flex System adapter
If you don’t see the mlx4_en driver, run: ’ modprobe mlx4_en’
Another option is to use the command below to see which modules are active.
Example:
The “Usage: openibd {start|stop|restart|status}” command to modify this file and
thereby control the drivers.
Check the network interface name:
run: ifconfig -a
run: ethtool <interface>
> Host# lsmod |grep mlx4_en
mlx4_en 96382 0
mlx4_core 1184104 2 mlx4_en
Edit ”/etc/infiniband/openib.conf” which modules needs to load
from a service.
Load MLX4_EN module
MLX4_EN_LOAD=yes
> ifconfig –a
eth0 Link encap:Ethernet HWaddr 5C:F3:FC:5F:40:EA
inet addr:10.7.5.170 Bcast:10.7.7.255 Mask:255.255.252.0
inet6 addr: fe80::5ef3:fcff:fe5f:40ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:20009 errors:0 dropped:0 overruns:0 frame:0
TX packets:4492 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7005902 (6.6 MiB) TX bytes:399581 (390.2 KiB)
> Host# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
10000baseT/Full
Advertised auto-negotiation: N
Speed: 1000Mb/s

10 IBM Flex System EN4132 2-port 10Gb Ethernet Adapter User’s Guide
Ethernet Driver Usage and Configuration
To check driver and device information run:
Example:
Windows drivers
For Windows, download and install the latest Mellanox WinOF VPI for Windows
software package available via the Mellanox web site at: http://www.mella-
nox.com => Products => Software/Drivers => InfiniBand & VPI Software/Drivers
=> Mellanox WinOF VPI => Download. Follow the installation instructions
included in the download package (also available from the download page). To
ensure that communication has been established follow the instructions below.
Get the device location on the PCI bus by performing the following steps:
1. Open Device Manager on the server. Click start=> Run, and then enter
"devmgmt.msc"
2. Expand System Devices and locate the adapter
3. Right click and select Properties to display the adapter properties window
4. Click the Details tab and select Device Instance ld (Windows 2003) or Hard-
ware Ids (Windows 2008/R2) from the Property pull-down menu
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
> ethtool -i eth<x>
> ethtool -i eth0
driver:mlx4_en (MT_1020110019_CX-3)
version: 1.5.6.33 (Oct 2011)
firmware-version:2.10.0000
bus-info: 0000:07:00.0

11
Installing the IBM Flex System adapter
Figure 8: PCI Device (Example)
5. In the Value display box, check the fields VEN and DEV (fields are separated
by ‘&’). In the display example above, notice the sub-string
“PCI\VEN_15B3&DEV_1003”: VEN is equal to 0x15B3 – this is the Vendor ID
of Mellanox Technologies; and DEV is equal to 1003 – this is a valid Mellanox
Technologies PCI Device ID.
Verify MLX driver
1. Start the “Device Manager” (open a command line window and enter:
devmgmt.msc)
2. Open “Network Adapters”
3. Right click the relevant adapter and select Properties
4. Click the Driver tab (example)