HP NC373m - PCI Express Dual Port Multifunction Gigabit Server... User manual

HP ProLiant Accelerated iSCSI for Linux
User Guide
Part Number 538529-001
March 2009 (First Edition)

© Copyright 2007, 2009 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express
warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP
shall not be liable for technical or editorial errors or omissions contained herein.
Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S.
Government under vendor’s standard commercial license.
Intended audience
This document is for the person who installs, administers, and troubleshoots servers and storage systems.
HP assumes you are qualified in the servicing of computer equipment and trained in recognizing hazards
in products with hazardous energy levels.

Contents 3
Contents
Overview..................................................................................................................................... 4
iSCSI protocol........................................................................................................................................... 4
Supported operating systems ...................................................................................................................... 4
Supported hardware.................................................................................................................................. 4
Installation and configuration ......................................................................................................... 6
Downloading files ..................................................................................................................................... 6
Installing the BRCM Multifunction drivers for Linux ......................................................................................... 6
Installing the Linux iSCSI offload bnx2i driver................................................................................................ 7
Linux iSCSI offload bnx2i driver limitations ......................................................................................... 8
Uninstalling the HP Linux iSCSI offload bnx2i driver RPM package ........................................................ 8
Linux iSCSI offload bnx2i driver messages.................................................................................................... 8
Installing the license for embedded adapters............................................................................................... 10
Red Hat........................................................................................................................................ 11
SLES ............................................................................................................................................ 11
Reporting iSCSI Offload status ........................................................................................................ 12
HP ProLiant Licensing Log and Event File........................................................................................... 12
Troubleshoot known issues for HP port-based Linux iSCSI offload......................................................... 13
Uninstalling the license for HP port-based Linux iSCSI offload........................................................................ 14
Removing the HP ProLiant Licensing RPM ..........................................................................................14
Removing the hp-pbl RPM package.................................................................................................. 14
HP NC-Series iSCSI offload initiator utilities ................................................................................................ 14
Red Hat........................................................................................................................................ 14
SLES ............................................................................................................................................ 15
Loading the Linux iSCSI offload bnx2i module............................................................................................. 16
Executing the bnx2id application .............................................................................................................. 16
Binding the iSCSI target............................................................................................................... 17
Changing the transport binding of the iSCSI node ....................................................................................... 17
Connecting to the iSCSI target...................................................................................................... 18
Sample commands to discover targets and create iSCSI connections.............................................................. 18
Acronyms and abbreviations........................................................................................................ 19
Index......................................................................................................................................... 20

Overview 4
Overview
iSCSI protocol
The HP iSCSI protocol transports block-level storage requests of TCP connections over IPv4 and IPv6.
Using the iSCSI protocol, systems can connect to remote storage and use it as a physical disk although
the remote storage provider or target may actually be providing virtual physical disks. HP Accelerated
iSCSI supports all targets that are certified and supported by Linux software.
HP Gigabit Ethernet Multifunction Server Adapters can be enabled as iSCSI devices or virtual iSCSI host
bus adapters (HBAs), a feature commonly referred to as accelerated iSCSI. iSCSI HBAs or accelerated
adapters, provide the capability to run iSCSI protocol over offloaded TCP connections. By offloading the
IPv4 or IPv6 TCP connection processing to an HP Gigabit Ethernet Multifunction Server Adapter, CPU
utilization is reduced allowing additional processing capacity for applications. The HP accelerated iSCSI
solution supports all targets that are certified and supported by Linux software.
Supported operating systems
•Red Hat Enterprise Linux 5 AS/ES for AMD64/EM64T
•Red Hat Enterprise Linux 5 AS/ES for x86
•SUSE LINUX Enterprise Server 10 (SP1, SP2) for x86
•SUSE LINUX Enterprise Server 10 (SP1, SP2) for AMD64
Supported hardware
Servers. Server support includes all ProLiant servers that include an iLO or RiLO management device with
any of the supported operating systems.
Adapters. HP adapter support includes the following:
•HP NC370T PCI-X Multifunction Gigabit Server Adapter
•HP NC370F PCI-X Multifunction Gigabit Server Adapter
•HP NC370i PCI-X Multifunction Gigabit Server Adapter *
•HP NC371i PCI-X Multifunction Gigabit Server Adapter*
•HP NC373F PCI Express Multifunction Gigabit Server Adapter
•HP NC373T PCI Express Multifunction Gigabit Server Adapter
•HP NC373m PCI Express Dual Port Multifunction Gigabit Server Adapter for c-Class BladeSystem
•HP NC373i Integrated Multifunction Gigabit Server Adapter*
•HP NC374m PCI Express Dual Port Multifunction Gigabit Server Adapter
•HP NC380T PCI Express Dual Port Multifunction Gigabit Server Adapter

Overview 5
•HP NC382T PCI Express Dual Port Multifunction Gigabit Server Adapter
•HP NC382m Dual Port 1GbE Multifunction BL-c Adapter
•HP NC382i Integrated Quad Port PCI Express Gigabit Server Adapter
•HP NC532m Dual Port Flex-10 10GbE Multifunction BL-c Adapter
•HP NC532i Dual Port 10GbE Multifunction BL-c Adapter*
* Requires an HP ProLiant Accelerated iSCSI license to enable acceleration
iSCSI targets. All iSCSI targets supported by the Linux software initiator are supported by HP.

Installation and configuration 6
Installation and configuration
Downloading files
1. Go to the HP website (http://www.hp.com/#Support).
2. Select Download drivers and software (and firmware).
3. Type the adapter name in the For product box, and then click Go. For example, NC382T.
4. Select an operating system.
5. Click HP NC-Series iSCSI Offload Driver for Linux.
6. Click download and save the HP SoftPaq (sp#####.exe) file to a directory on your hard drive. The
SoftPaq file is a self-extracting executable with a file name based on the SoftPaq number.
7. Click the SoftPaq file to extract the files.
Installing the BRCM Multifunction drivers for Linux
The HP NC-Series Broadcom Multifunction Drivers for Linux are released in a source RPM format and the
file name for the package is netxtreme2-<version>.sri.rpm. The package contains the source code and
build instructions for bnx2, bnx2x, and cnic drivers.
1. Build Environment Setup. The netxtreme2 package requires the presence of packages that provide
the right build environment. These packages provide kernel headers, makefiles, and symbol files
(suggested only on SLES distributions).
Red Hat installations:
For each kernel flavor you want to build the driver make sure you have the associated kernel-devel
package. For example, kernel-2.6.18.92.el5 requires kernel-devel-2.6.18-92.el5 and kernel-PAE-
2.6.18.92.el5 requires kernel-PAE-devel-2.6.18-92.el5.
SUSE SLES installations:
Make sure that kernel-source and kernel-syms packages are installed. The kernel-syms package
contains symbol versions for all of the iterations that the distribution supports.
2. Install the source RPM package.
# rpm -ivh netxtreme2-<version>.src.rpm
3. Build the binary RPM for the netxtreme2.
Red Hat installations:
# cd /usr/src/redhat
# rpmbuild -bb SPECS/netxtreme2.spec
SLES installations:
# cd /usr/src/packages
# rpmbuild -bb SPECS/netxtreme2.spec
If you get an error during the build process, go back to step one above.
4. Check for the existence of a current version of the netxtreme2 package as follows:
# rpm -q netxtreme2

Installation and configuration 7
If an old version of the package exists, the RPM package should be removed.
# rpm -e netxtreme2
5. Verify that the old netxtreme2 package has been removed as follows:
# rpm -q netxtreme2
6. Verify if a old version of bnx2 or bnx2x is loaded into the kernel and remove it, if needed, as
follows:
If you are installing the package over the network, it is not necessary to remove the modules. This
step is only necessary if you are doing the upgrade on the physical console and you want the new
drivers to take over immediately.
#lsmod | grep bnx2x
#lsmod | grep bnx2
7. Before removing the drivers, bring down the interfaces controlled by bnx2x and bnx2.
#ifconfig eth<n> down
8. Remove the drivers.
#rmmod bnx2x
#rmmod bnx2
9. Install the new RPM package. This installs bnx2/bnx2x and cnic drivers. Do not use the U (Upgrade)
flag in the RPM command below--previous netxtreme2 packages must be removed as described
above before installing this version.
#rpm -ivh RPMS/<arch>/netxtreme2-<version>.<arch>.rpm
The bnx2.ko, bnx2x.ko and cnic.ko modules are installed in the following directory:
Red Hat EL 4/5 installations:
/lib/modules/<kernel_version>/updates
SLES 9/10 installations:
/lib/modules/<kernel_version>/updates
10. Configure your network setting and address. See your Linux vendor documentation for more
information. Helpful network configuration tools such as "yast2" in SLES or linuxconf/redhat-config-
network/netconfig in Red Hat exist for easy configuration.
11. Ensure that the /etc/modules.conf file is configured similar to the example listed below. The example
below is presented as if more than one adapter is present. If so, one eth# instance should exist for
each ethernet port. See the modules.conf man page for more information.
If you have a 1Gb Network card use bnx2 below otherwise use bnx2x for 10Gb.
alias eth0 bnx2
alias eth1 bnx2
For SLES, the configuration file is /etc/modprobe.conf or /etc/modprobe.conf.local
12. You can now reboot your server or restart the network services. Upon reboot the network should start
with either bnx2 or bnx2x driver loaded.
To verify that the driver is loaded use the following command.
# lsmod
You should find bnx2 or bnx2x listed. You can also verify that the correct bnx2 or bnx2x driver is
loaded through any of the following methods. Note that version of the driver loaded should be same
as that of the package version.
Installing the Linux iSCSI offload bnx2i driver
The Linux iSCSI offload bnx2i driver is released in a source RPM format.

Installation and configuration 8
1. To install the source RPM package, change to the linux\iscsi-offload directory on your hard drive
and use the following command:
# rpm -ivh bnx2i-<version>.src.rpm
2. Build the binary RPM for the bnx2i driver.
For Red Hat EL 5 installations enter the following commands:
# cd /usr/src/redhat
# rpmbuild -bb SPECS/bnx2i.spec
For SLES installations enter the following commands:
# cd /usr/src/packages
# rpmbuild -bb SPECS/bnx2i.spec
3. Check for the existence of a current version of the bnx2i package using the following command:
# rpm -q bnx2i
If an older version of the package exists, remove the old RPM package using the following
command:
# rpm -e bnx2i
Remove the loaded bnx2i module using the following command:
# modprobe -r bnx2i
This command removes the bnx2i module along with its dependencies.
Verify that the old bnx2i package has been removed using the following command:
# rpm -q bnx2i
4. Install the bnx2i binary RPM package using the following command:
# rpm -ivh RPMS/<arch>/bnx2i-<version>.<arch>.rpm
The bnx2i.ko module is installed in the following directory:
/lib/modules/<kernel_version>/kernel/drivers/iscsi
Linux iSCSI offload bnx2i driver limitations
•The current version of the driver has been tested on 2.6.x kernels beginning with the 2.6.18 kernel.
The driver may not compile on kernels older than the 2.6.18 kernel. SLES10 SP1 is an exception. It
also runs on the 2.6.16.46 kernel because SUSE upgraded the iscsi_transport kernel module to
support offload initiators in SLES10 SP1.
•When installing SLES 10 SP2 select to enable IPv6 so that autoload is enabled for the IPv6 kernel
module. The cnic driver which is delivered in the NetXtreme2 RPM and is required for iSCSI offload
will fail to resolve symbols if IPv6 is disabled because it has a dependancy on the IPv6 module.
Uninstalling the HP Linux iSCSI offload bnx2i driver RPM
package
If you need to uninstall the HP bnx2 RPM, use the following command:
# rpm -e bnx2i-<version>
Linux iSCSI offload bnx2i driver messages
Linux iSCSI offload bnx2i driver messages are logged in the file /var/log/messages.
Use the following command to control the level that messages appear on the console:
dmesg -n <level>

Installation and configuration 9
Most systems are set to level 6 by default. To include debug-level messages, set the level to 7.
The following sample messages are the most common bnx2i driver messages that may be logged in the
file /var/log/messages.
•bnx2i driver signon
Broadcom NetXtreme II iSCSI Driver bnx2i v1.0.30 (Sep 29, 2007)
•Network port to iSCSI transport name binding
bnx2i: netif=eth2, iscsi=bcm5708-050000
bnx2i: netif=eth1, iscsi=bcm5706-030c00
•Driver completes handshake with iSCSI offload enabled CNIC device
bnx2i [05:00.00]: ISCSI_INIT passed
This message displays only when a user attempts to make an iSCSI connection.
•Driver detects iSCSI Offload is not enabled on the CNIC device
bnx2i: iSCSI not supported, dev=eth3
bnx2i: bnx2i: LOM is not enabled to offload iSCSI connections, dev=eth0
•Driver unable to allocate TCP port for iSCSI connection
bnx2i: run 'bnx2id' daemon to alloc TCP ports
•Exceeds maximum allowed iSCSI connection offload limit
bnx2i: unable to allocate iSCSI context resources
•Network route to target node and transport name binding are 2 different devices
bnx2i: conn bind, ep=0x... ($ROUTE_HBA) does not belong to hba
$USER_CHOSEN_HBA
where ROUTE_HBA --> net device on which connection was offloaded based
on route information
USER_CHOSEN_HBA --> HBA to which target node is bound (using iSCSI
transport name)
•Target cannot be reached on any of CNIC devices
bnx2i: check route, can't connect using cnic
•Network route is assigned to network interface that is down
bnx2i: check route, hba not found
•Attempting to offload iSCSI connection onto Jumbo frame enabled device
bnx2i: eth# network i/f mtu is set to #mtu
bnx2i: iSCSI HBA can support mtu of 1500
The user must change mtu to < 1500 using ethtool and restart the interface to offload iSCSI
connections.
•SCSI-ML initiated host reset (session recovery)
bnx2i: attempting to reset host, #3
•CNIC detects iSCSI protocol violation — FATAL errors
bnx2i: iscsi_error - wrong StatSN rcvd
bnx2i: iscsi_error - hdr digest err
bnx2i: iscsi_error - data digest err
bnx2i: iscsi_error - wrong opcode rcvd
bnx2i: iscsi_error - AHS len > 0 rcvd
bnx2i: iscsi_error - invalid ITT rcvd
bnx2i: iscsi_error - wrong StatSN rcvd
bnx2i: iscsi_error - wrong DataSN rcvd
bnx2i: iscsi_error - pend R2T violation
bnx2i: iscsi_error - ERL0, UO
bnx2i: iscsi_error - ERL0, U1
bnx2i: iscsi_error - ERL0, U2

Installation and configuration 10
bnx2i: iscsi_error - ERL0, U3
bnx2i: iscsi_error - ERL0, U4
bnx2i: iscsi_error - ERL0, U5
bnx2i: iscsi_error - ERL0, U
bnx2i: iscsi_error - invalid resi len
bnx2i: iscsi_error - MRDSL violation
bnx2i: iscsi_error - F-bit not set
bnx2i: iscsi_error - invalid TTT
bnx2i: iscsi_error - invalid DataSN
bnx2i: iscsi_error - burst len violation
bnx2i: iscsi_error - buf offset violation
bnx2i: iscsi_error - invalid LUN field
bnx2i: iscsi_error - invalid R2TSN field
bnx2i: iscsi_error - invalid cmd len1
bnx2i: iscsi_error - invalid cmd len2
bnx2i: iscsi_error - pend r2t exceeds MaxOutstandingR2T value
bnx2i: iscsi_error - TTT is rsvd
bnx2i: iscsi_error - MBL violation
bnx2i: iscsi_error - data seg len != 0
bnx2i: iscsi_error - reject pdu len error
bnx2i: iscsi_error - async pdu len error
bnx2i: iscsi_error - nopin pdu len error
bnx2i: iscsi_error - pend r2t in cleanup
bnx2i: iscsi_error - IP fragments rcvd
bnx2i: iscsi_error - IP options error
bnx2i: iscsi_error - urgent flag error
•CNIC detects iSCSI protocol violation — non-FATAL, warning
bnx2i: iscsi_warning - invalid TTT
bnx2i: iscsi_warning - invalid DataSN
bnx2i: iscsi_warning - invalid LUN field
The driver must be configured for some violations to be reported as a warning instead of as a critical
error.
•Driver puts a session through recovery
conn_err - hostno 3 conn 03fbcd00, iscsi_cid 2 cid a1800
•REJECT iSCSI PDU received from the target
bnx2i - printing rejected PDU contents
[0]: 1 ffffffa1 0 0 0 0 20 0
[8]: 0 7 0 0 0 0 0 0
[10]: 0 0 40 24 0 0 ffffff80 0
[18]: 0 0 3 ffffff88 0 0 3 4b
[20]: 2a 0 0 2 ffffffc8 14 0 0
[28]: 40 0 0 0 0 0 0 0
•Open-iSCSI daemon handing over session to driver
bnx2i: conn update - MBL 0x800 FBL 0x800MRDSL_I 0x800 MRDSL_T 0x2000
Installing the license for embedded adapters
The HP Accelerated iSCSI capability for Linux systems is available as a licensable feature for most
embedded HP Multifunction Gigabit Server Adapters through the purchase of an HP ProLiant Accelerated
iSCSI port-based license. License keys are installed on a per-port basis for each embedded adapter. See
the list of supported adapters to see if your adapter requires a license.
This license is not required for standup and mezzanine HP Multifunction Gigabit Server Adapters and HP
Multifunction Gigabit Server Adapters for HP BladeSystem servers.