Chelsio Communications Chelsio T5 User manual

Chelsio T5/T4 Unified Wire For XenServer ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [e-mail]
Copyright © 2014.Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
370 San Aleso Ave.
Suite 100
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Fax: [phone]
Chelsio KK (Japan)
SHIMA Akasaka Bldg.
Minato-ku, Tokyo
Japan 107-0052
Tel: [phone]

Chelsio T5/T4 Unified Wire For XenServer iii
Document History
Version
Revision Date
1.0.0
03/15/2013
1.0.1
04/02/2013
1.0.2
06/06/2014

Chelsio T5/T4 Unified Wire For XenServer iv
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 5
1. Introduction 6
1.1. Features 6
1.2. Hardware Requirements 6
1.3. Software Requirements 6
1.4. Package Contents 7
2. Hardware Installation 8
3. Software/Driver Installation 11
3.1. During OS installation 11
3.2. Runtime Installation 12
3.3. Firmware Update 13
4. Software/Driver Uninstallation 14
5. Software/Driver Update 15
II. NETWORK (NIC) 16
1. Introduction 17
1.1. Hardware Requirements 17
2. Software/Driver Loading 18
3. Software/Driver Configuration and Fine-tuning 19
3.1. Instantiate Virtual Functions (SR-IOV) 19
III. ISCSI PDU OFFLOAD INITIATOR 20
1. Introduction 21
1.1. Hardware Requirements 21
2. Software/Driver Loading 23
3. Software/Driver Unloading 24
4. Software/Driver Configuration and Fine-tuning 25
4.1. Accelerating open-iSCSI Initiator 25
4.2. Auto login from cxgb4i initiator at OS bootup 27
IV. APPENDIX 29
1. Chelsio End-User License Agreement (EULA) 30

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 5
I.Chelsio Unified Wire

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 6
1.Introduction
Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip,
single firmware cards provide enterprises and data centers with high performance solutions for
various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, ISCSI,
FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
The T4 adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified
Wire. The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and
Filtering.
Ideal for all data, storage and high performance clustering applications, the T5/T4 Adapters
enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets,
Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio T5/T4 adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
1.1. Features
Chelsio’s Unified Wire software is an ISO image which installs Network and iSCSI PDU Offload
Initiator drivers for Chelsio’s T5 and T4 adapters on XenServer.
1.2. Hardware Requirements
Chelsio’s Unified Wire software supports Chelsio T5 and T4 Series of Unified Wire Adapters. To
know more about the list of adapters supported by each driver, please refer to their respective
sections.
1.3. Software Requirements
1.3.1. XenServer Requirements
Chelsio’s Unified Wire has been developed to run on XenServer platforms. Currently the
software is available for the following version:

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 7
Citrix XenServer Host 6.2.0
Kernel version: 2.6.32.43-0.4.1.xs1.8.0.835.170778xen
Other versions have not been tested and are not guaranteed to work.
1.3.2. Resource Requirements
Control Domain (Dom0) memory: Based on the number of 40G, 10G and 1G ports, driver
allocates memory enough to run at the desired rate. It is suggested to increase the control
domain memory to 2GB using the procedure explained in
http://support.citrix.com/article/CTX134951
SW-IOMMU Size: Increasing SW-IOMMU size by appending swiotlb=128 to
/boot/extlinux.conf avoids PCI-DMA: Out of SW-IOMMU space for <size> bytes
at device <BDF> messages.
1.4. Package Contents
The software supplied is an ISO image. It consists of the following:
RPM packages containing drivers and associated tools, firmware, etc.
Shell script to install the driver package.

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 8
2.Hardware Installation
Follow the steps to install Chelsio Adapter in your system:
1. Shutdown and power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the 10Gb Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIE x8
slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Verify if the adapter was installed successfully by using the lspci command
a. For T5 adapters :
[root@host]# lspci |grep -i Chelsio
07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.3 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.4 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.5 SCSI storage controller: Chelsio Communications Inc T520-LL-CR
Unified Wire Storage Controller
07:00.6 Fibre Channel: Chelsio Communications Inc T520-LL-CR Unified Wire
Storage Controller

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 9
b. And for T4 adapters :
[root@host]# lspci | grep –i Chelsio
03:00.0 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.1 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.2 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.3 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.4 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.5 SCSI storage controller: Chelsio Communications Inc T420-CR Unified
Wire Storage Controller
03:00.6 Fibre Channel: Chelsio Communications Inc T420-CR Unified Wire
Storage Controller
03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000
For Chelsio T5/T4 adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
Once Unified Wire package is installed and loaded, examine the output of dmesg to see if the
card is discovered.
For T5 adapters:
eth4: Chelsio T520-LL rev 0 10GBASE-SFP RNIC PCIe 5 GT/s x8 MSI-X, Offload
capable
0000:07:00.4: S/N: PT03140177, P/N: 110116750C0
And, for T4 adapters:
eth0: Chelsio T420-CR rev 2 10GBASE-SFP RNIC PCIe x8 MSI-X
0000:04:00.4: S/N: PT18111226, P/N: 110112140D0

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 10
The above outputs indicate the hardware configuration of the adapters as well as the Serial
numbers. As observed by the x8, the card is properly installed in an x8 slot on the machine and
using MSI-X interrupts.
Network device names for Chelsio’s physical ports are assigned using the
following convention: the port farthest from the motherboard will appear as the
first network interface. However, for T5 40G and T420-BT adapters, the
association of physical Ethernet ports and their corresponding network device
names is opposite. For these adapters, the port nearest to the motherboard will
appear as the first network interface.
Note

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 11
3.Software/Driver Installation
Chelsio Unified Wire must be installed by the root user. Any attempt to install the software as a
regular user will fail.
There are two methods of installation: During operating system installation and Runtime.
3.1. During OS installation
1. If you haven’t done so already, download the Unified Wire software (ISO image) from
Chelsio Download Center, http://service.chelsio.com
2. Burn the image to a CD/DVD.
3. Start an interactive install of XenServer.
4. When the installer prompts, opt for installing drivers through Supplemental Packs. Choose
“Yes” and hit [Enter].
5. Insert the disc into disc drive. Choose “Ok” and hit [Enter].

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 12
6. The installer will now search and display the drivers. Choose “Use” and hit [Enter].
7. The selected drivers will now be installed.
8. To install any additional drivers follow the same procedure mentioned above. Or choose
“Skip” and hit [Enter] to complete the driver installation.
3.2. Runtime Installation
1. If you haven’t done so already, download the Unified Wire software (ISO image) from
Chelsio Download Center, http://service.chelsio.com
2. Copy the ISO image on to a host running XenServer.
3. Mount the image onto a directory by running the following commands:
[root@host]# mkdir /tmp/iso
[root@host]# mount -o loop <driver.iso> /tmp/iso
[root@host]# cd /tmp/iso
4. Run the install.sh script to install the drivers and tools:
[root@host]# ./install.sh

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 13
5. Finally, unmount the ISO image:
[root@host]# cd
[root@host]# umount /tmp/iso
3.3. Firmware Update
The T5 and T4 firmwares are installed on the system, typically in /lib/firmware/cxgb4, and
the driver will auto-load them if an update is required.
The firmware version can be verified using ethtool:
[root@host]# ethtool -i <iface>

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 14
4.Software/Driver Uninstallation
1. Use the following query command to determine the name of the driver RPM:
[root@host]# rpm -qa | grep -i <driver_name>
2. Now, with the result from the above query, execute the following command to uninstall all
related packages:
[root@host]# rpm -e <package_name>
E.g.:
[root@host~]# rpm -qa | grep -i cxgb4
cxgb4-modules-kdump-2.6.32.43-0.4.1.xs1.8.0.835.170778-1.2.6-2
cxgb4-t5fw-1.10.11.252-2
cxgb4-t4fw-1.10.11.252-2
cxgb4-modules-xen-2.6.32.43-0.4.1.xs1.8.0.835.170778-1.2.6-2
cxgb4-phyfw-1.2.6-2
cxgb4-tools-1.16-2
[root@host ~]# rpm -e cxgb4-modules-kdump-2.6.32.43-
0.4.1.xs1.8.0.835.170778-1.2.6-2
[root@host ~]# rpm -e cxgb4-phyfw-1.2.6-2
[root@host ~]# rpm -e cxgb4-t5fw-1.10.11.252-2
[root@host ~]# rpm -e cxgb4-modules-xen-2.6.32.43-0.4.1.xs1.8.0.835.170778-
1.2.6-2
[root@host ~]# rpm -e cxgb4-tools-1.16-2
[root@host ~]# rpm –e cxgb4-t4fw-1.10.11.252-2

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire For XenServer Page 15
5.Software/Driver Update
For any distribution specific problems, please check README and Release Notes included in
the release for possible workaround.
Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various
software/drivers. You can also subscribe to our newsletter for the latest software updates.

Chapter II. Network (NIC)
Chelsio T5/T4 Unified Wire For XenServer Page 16
II.Network (NIC)

Chapter II. Network (NIC)
Chelsio T5/T4 Unified Wire For XenServer Page 17
1.Introduction
Chelsio’s T5 and T4 series of Unified Wire Adapters provide extensive support for NIC
operation, including all stateless offload mechanisms for both IPv4 and IPv6 (IP, TCP and UDP
checksum offload, LSO - Large Send Offload aka TSO - TCP Segmentation Offload, and assist
mechanisms for accelerating LRO - Large Receive Offload).
A high performance fully offloaded and fully featured TCP/IP stack meets or exceeds software
implementations in RFC compliance. Chelsio’s T5/T4 engine provides unparalleled performance
through a specialized data flow processor implementation and a host of features designed for
high throughput and low latency in demanding conditions and networking environments, using
standard size Ethernet frames.
TCP offload is fully implemented in the hardware, thus freeing the CPU from TCP/IP overhead.
The freed CPU can be used for any computing needs. The TCP offload in turn removes network
bottlenecks and enables applications to take full advantage of the networking capabilities.
1.1. Hardware Requirements
1.1.1. Supported Adapters
The following are the currently shipping Chelsio adapters that are compatible with Chelsio
Network driver:
T502-BT
T580-CR
T580-SO-CR
T520-LL-CR
T520-SO-CR
T520-CR
T522-CR
T580-LP-CR
T540-CR
T420-CR
T420-LL-CR
T440-CR
T440-LP-CR
T420-BCH
T422-CR
T420-SO-CR
T420-CX
T420-BT
T404-BT

Chapter II. Network (NIC)
Chelsio T5/T4 Unified Wire For XenServer Page 18
2.Software/Driver Loading
The driver must be loaded by the root user. Any attempt to load the driver as a regular user will
fail.
To load the driver, execute the following:
[root@host]# modprobe cxgb4

Chapter II. Network (NIC)
Chelsio T5/T4 Unified Wire For XenServer Page 19
3.Software/Driver Configuration and Fine-tuning
3.1. Instantiate Virtual Functions (SR-IOV)
To instantiate the Virtual functions, load the cxgb4 driver with num_vf parameter with a non-zero
value. For example:
[root@host]# modprobe cxgb4 num_vf=1,0,0,0
The number(s) provided for num_vf parameter specifies the number of Virtual Functions to be
instantiated per Physical Function. The Virtual Functions can be assigned to Virtual Machines
(Guests). A maximum of 64 Virtual Functions can be instantiated with 16 Virtual Functions per
Physical Function.
To get familiar with physical and virtual function terminologies, please refer the
PCI Express specification.
Note

Chapter III. iSCSI PDU Offload Initiator
Chelsio T5/T4 Unified Wire For XenServer Page 20
III.iSCSI PDU Offload Initiator