Chelsio Communications T5 Series User manual

Chelsio T5/T4 Network Driver For XenServer ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [e-mail]
Copyright © 2015.Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
370 San Aleso Ave.
Suite 100
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Fax: [phone]
Chelsio KK (Japan)
SHIMA Akasaka Bldg.
Minato-ku, Tokyo
Japan 107-0052
Tel: [phone]

Chelsio T5/T4 Network Driver For XenServer iii
Document History
Version
Revision Date
1.0.0
05/18/2012
1.0.1
04/24/2015

Chelsio T5/T4 Network Driver For XenServer iv
TABLE OF CONTENTS
1. Introduction 5
1.1. Features 5
1.2. Hardware Requirements 5
1.3. Software Requirements 6
1.4. Package Contents 7
2. Hardware Installation 8
3. Software/Driver Installation 11
3.1. During OS installation 11
3.2. Runtime Installation 12
3.3. Firmware Update 13
4. Software/Driver Loading 14
5. Software/Driver Configuration and Fine-tuning 15
5.1. Instantiate Virtual Functions (SR-IOV) 15
6. Software/Driver Unloading 16
7. Software/Driver Uninstallation 17
8. Software/Driver Update 18
9. Appendix 19
9.1. Chelsio End-User License Agreement (EULA) 19

Chapter 1. Introduction
Chelsio T5/T4 Network Driver For XenServer Page 5
1.Introduction
Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip,
single firmware cards provide enterprises and data centers with high performance solutions for
various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, ISCSI,
FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
The T4 adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified
Wire. The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and
Filtering.
Ideal for all data, storage and high performance clustering applications, the T5/T4 Adapters
enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets,
Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio T5/T4 adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
This document describes the installation, use and maintenance of Network driver for
XenServer and its various components.
1.1. Features
Chelsio’s Network driver package for XenServer is an ISO image which installs drivers
for Chelsio’s T5/T4 Unified Wire adapters on XenServer.
1.2. Hardware Requirements
1.2.1. Supported Adapters
The following are the currently shipping Chelsio adapters that are compatible with Chelsio
Network driver:
T520-BT
T502-BT
T580-CR
T580-SO-CR

Chapter 1. Introduction
Chelsio T5/T4 Network Driver For XenServer Page 6
T520-LL-CR
T520-SO-CR
T520-CR
T522-CR
T580-LP-CR
T540-CR
T420-CR
T420-LL-CR
T440-CR
T440-LP-CR
T420-BCH
T422-CR
T420-SO-CR
T420-CX
T420-BT
T404-BT
1.3. Software Requirements
1.3.1. XenServer Requirements
Chelsio’s Network driver has been developed to run on XenServer platforms. Currently the
software is available for the following version:
Citrix XenServer Host 6.5.0
Kernel version: 3.10.0+2
Other versions have not been tested and are not guaranteed to work.
1.3.2. Resource Requirements
Control Domain (Dom0) memory: Based on the number of 40/10/1G ports, driver allocates
memory enough to run at the desired rate. It is suggested to increase the control domain
memory to 2GB using the procedure explained in
http://support.citrix.com/article/CTX134951
SW-IOMMU Size: Increasing SW-IOMMU size by appending swiotlb=128 to
/boot/extlinux.conf avoids PCI-DMA: Out of SW-IOMMU space for <size> bytes
at device <BDF> messages.

Chapter 1. Introduction
Chelsio T5/T4 Network Driver For XenServer Page 7
1.4. Package Contents
The software supplied is an ISO image. It consists of the following:
RPM packages containing drivers and associated tools, firmware, etc.
Shell script to install the driver package.

Chapter 2. Hardware Installation
Chelsio T5/T4 Network Driver For XenServer Page 8
2.Hardware Installation
Follow the steps to install Chelsio Adapter in your system:
1. Shutdown and power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIE
x8/x16 slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 40/10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Run update-pciids command to download the current version of PCI ID list
[root@host~]# update-pciids
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 198k 100 198k 0 0 491k 0 --:--:-- --:--:-- --:--:-- 626k
Done.
13. Verify if the adapter was installed successfully by using the lspci command
a. For T5 adapters :
[root@host~]# lspci |grep -i Chelsio
07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.3 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.4 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified

Chapter 2. Hardware Installation
Chelsio T5/T4 Network Driver For XenServer Page 9
Wire Ethernet Controller
07:00.5 SCSI storage controller: Chelsio Communications Inc T520-LL-CR
Unified Wire Storage Controller
07:00.6 Fibre Channel: Chelsio Communications Inc T520-LL-CR Unified Wire
Storage Controller
b. And for T4 adapters :
[root@host~]# lspci | grep –i Chelsio
03:00.0 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.1 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.2 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.3 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.4 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.5 SCSI storage controller: Chelsio Communications Inc T420-CR Unified
Wire Storage Controller
03:00.6 Fibre Channel: Chelsio Communications Inc T420-CR Unified Wire
Storage Controller
03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000
For Chelsio T5/T4 adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
Once driver is installed and loaded, examine the output of dmesg to see if the card is
discovered.
For T5 adapters:
eth4: Chelsio T520-LL rev 0 10GBASE-SFP RNIC PCIe 5 GT/s x8 MSI-X, Offload
capable
0000:07:00.4: S/N: PT03140177, P/N: 110116750C0

Chapter 2. Hardware Installation
Chelsio T5/T4 Network Driver For XenServer Page 10
And, for T4 adapters:
eth0: Chelsio T420-CR rev 2 10GBASE-SFP RNIC PCIe x8 MSI-X
0000:04:00.4: S/N: PT18111226, P/N: 110112140D0
The above outputs indicate the hardware configuration of the adapters as well as the Serial
numbers. As observed by the x8, the card is properly installed in an x8 slot on the machine and
using MSI-X interrupts.
Network device names for Chelsio’s physical ports are assigned using the following
convention: the port farthest from the motherboard will appear as the first network
interface. However, for T5 40G and T420-BT adapters, the association of physical
Ethernet ports and their corresponding network device names is opposite. For
these adapters, the port nearest to the motherboard will appear as the first network
interface.
Note

Chapter 3. Software/Driver Installation
Chelsio T5/T4 Network Driver For XenServer Page 11
3.Software/Driver Installation
Chelsio Network driver must be installed by the root user. Any attempt to install the software as
a regular user will fail.
There are two methods of installation: During operating system installation and Runtime.
3.1. During OS installation
1. If you haven’t done so already, download the Network driver(ISO image) from Chelsio
Download Center, http://service.chelsio.com
2. Burn the image to a CD/DVD.
3. Start an interactive install of XenServer.
4. When the installer prompts, opt for installing drivers through Supplemental Packs. Choose
“Yes” and hit [Enter].
5. Insert the disc into disc drive. Choose “Ok” and hit [Enter].

Chapter 3. Software/Driver Installation
Chelsio T5/T4 Network Driver For XenServer Page 12
6. The installer will now search and display the drivers. Choose “Use” and hit [Enter].
7. The selected drivers will now be installed.
8. To install any additional drivers follow the same procedure mentioned above. Or choose
“Skip” and hit [Enter] to complete the driver installation.
3.2. Runtime Installation
1. If you haven’t done so already, download the Network driver (ISO image) from Chelsio
Download Center, http://service.chelsio.com
2. Copy the ISO image on to a host running XenServer.
3. Mount the image onto a directory by running the following commands:
[root@host~]# mkdir /tmp/iso
[root@host~]# mount -o loop <driver.iso> /tmp/iso
[root@host~]# cd /tmp/iso
4. Run the install.sh script to install the drivers and tools:
[root@host~]# ./install.sh

Chapter 3. Software/Driver Installation
Chelsio T5/T4 Network Driver For XenServer Page 13
5. Finally, unmount the ISO image:
[root@host~]# cd
[root@host~]# umount /tmp/iso
3.3. Firmware Update
The T5 and T4 firmwares are installed on the system, typically in /lib/firmware/cxgb4, and
the driver will auto-load them if an update is required.
The firmware version can be verified using ethtool:
[root@host~]# ethtool -i <iface>

Chapter 4. Software/Driver Loading
Chelsio T5/T4 Network Driver For XenServer Page 14
4.Software/Driver Loading
The driver must be loaded by the root user. Any attempt to load the driver as a regular user will
fail.
To load the driver, execute the following:
[root@host~]# modprobe cxgb4

Chapter 5. Software/Driver Configuration and Fine-tuning
Chelsio T5/T4 Network Driver For XenServer Page 15
5.Software/Driver Configuration and Fine-tuning
5.1. Instantiate Virtual Functions (SR-IOV)
To instantiate the Virtual functions, load the cxgb4 driver with num_vf parameter with a non-zero
value. For example:
[root@host~]# modprobe cxgb4 num_vf=1,0,0,0
The number(s) provided for num_vf parameter specifies the number of Virtual Functions to be
instantiated per Physical Function. The Virtual Functions can be assigned to Virtual Machines
(Guests). A maximum of 64 Virtual Functions can be instantiated with 16 Virtual Functions per
Physical Function.
To get familiar with physical and virtual function terminologies, please refer the PCI
Express specification.
Note

Chapter 6. Software/Driver Unloading
Chelsio T5/T4 Network Driver For XenServer Page 16
6.Software/Driver Unloading
To unload the driver, execute the following:
[root@host~]# rmmod cxgb4

Chapter 7. Software/Driver Uninstallation
Chelsio T5/T4 Network Driver For XenServer Page 17
7.Software/Driver Uninstallation
1. Use the following query commands to determine the RPMs installed:
[root@host~]# rpm -qa | grep -i cxgb4
[root@host~]# rpm -qa | grep -i chelsio
2. Now, with the results from the above queries, execute the following command to uninstall all
related packages:
[root@host~]# rpm -e <package_name>

Chapter 8. Software/Driver Update
Chelsio T5/T4 Network Driver For XenServer Page 18
8.Software/Driver Update
For any distribution specific problems, please check README and Release Notes included in
the release for possible workaround.
Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various
software/drivers. You can also subscribe to our newsletter for the latest software updates.

Chapter 9. Appendix
Chelsio T5/T4 Network Driver For XenServer Page 19
9.Appendix
9.1. Chelsio End-User License Agreement (EULA)
Installation and use of the driver/software implies acceptance of the terms in the Chelsio End-
User License Agreement (EULA).
IMPORTANT: PLEASE READ THIS SOFTWARE LICENSE CAREFULLY BEFORE DOWNLOADING OR OTHERWISE
USING THE SOFTWARE OR ANY ASSOCIATED DOCUMENTATION OR OTHER MATERIALS
(COLLECTIVELY, THE "SOFTWARE"). BY CLICKING ON THE "OK" OR "ACCEPT" BUTTON YOU
AGREE TO BE BOUND BY THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO THE TERMS OF
THIS AGREEMENT, CLICK THE "DO NOT ACCEPT" BUTTON TO TERMINATE THE INSTALLATION
PROCESS.
1. License. Chelsio Communications, Inc. ("Chelsio") hereby grants you, the Licensee,
and you hereby accept, a limited, non-exclusive, non-transferable license to install
and use the Software with one or more Chelsio network adapters on a single server
computer for use in communicating with one or more other computers over a network.
You may also make one copy of the Software in machine readable form solely for back-up
purposes, provided you reproduce Chelsio's copyright notice and any proprietary
legends included with the Software or as otherwise required by Chelsio.
2. Restrictions. This license granted hereunder does not constitute a sale of the
Software or any copy thereof. Except as expressly permitted under this Agreement, you
may not:
(i) reproduce, modify, adapt, translate, rent, lease, loan, resell, distribute, or
create derivative works of or based upon, the Software or any part thereof; or
(ii) make available the Software, or any portion thereof, in any form, on the
Internet. The Software contains trade secrets and, in order to protect them, you may
not decompile, reverse engineer, disassemble, or otherwise reduce the Software to a
human-perceivable form. You assume full responsibility for the use of the Software and
agree to use the Software legally and responsibly.
3. Ownership of Software. As Licensee, you own only the media upon which the Software
is recorded or fixed, but Chelsio retains all right, title and interest in and to the
Software and all subsequent copies of the Software, regardless of the form or media in
or on which the Software may be embedded.
4. Confidentiality. You agree to maintain the Software in confidence and not to
disclose the Software, or any information or materials related thereto, to any third
party without the express written consent of Chelsio. You further agree to take all
reasonable precautions to limit access of the Software only to those of your employees
who reasonably require such access to perform their employment obligations and who are
bound by confidentiality agreements with you.
5. Term. This license is effective in perpetuity, unless terminated earlier. You may
terminate the license at any time by destroying the Software (including the related
documentation), together with all copies or modifications in any form. Chelsio may
terminate this license, and this license shall be deemed to have automatically
terminated, if you fail to comply with any term or condition of this Agreement. Upon

Chapter 9. Appendix
Chelsio T5/T4 Network Driver For XenServer Page 20
any termination, including termination by you, you must destroy the Software
(including the related documentation), together with all copies or modifications in
any form.
6. Limited Warranty. If Chelsio furnishes the Software to you on media, Chelsio
warrants only that the media upon which the Software is furnished will be free from
defects in material or workmanship under normal use and service for a period of thirty
(30) days from the date of delivery to you.
CHELSIO DOES NOT AND CANNOT WARRANT THE PERFORMANCE OR RESULTS YOU MAY OBTAIN BY USING
THE SOFTWARE OR ANY PART THEREOF. EXCEPT FOR THE FOREGOING LIMITED WARRANTY, CHELSIO
MAKES NO OTHER WARRANTIES, EXPRESS OR IMPLIED, AND HEREBY DISCLAIMS ALL OTHER
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, NON-INFRINGEMENT OF THIRD PARTY RIGHTS,
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow the
exclusion of implied warranties or limitations on how long an implied warranty may
last, so the above limitations may not apply to you. This warranty gives you specific
legal rights and you may also have other rights which vary from state to state.
7. Remedy for Breach of Warranty. The sole and exclusive liability of Chelsio and its
distributors, and your sole and exclusive remedy, for a breach of the above warranty,
shall be the replacement of any media furnished by Chelsio not meeting the above
limited warranty and which is returned to Chelsio. If Chelsio or its distributor is
unable to deliver replacement media which is free from defects in materials or
workmanship, you may terminate this Agreement by returning the Software.
8. Limitation of Liability. IN NO EVENT SHALL CHELSIO HAVE ANY LIABILITY TO YOU OR ANY
THIRD PARTY FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR PUNITIVE
DAMAGES, HOWEVER CAUSED, AND ON ANY THEORY OF LIABILITY, ARISING OUT OF OR RELATED TO
THE LICENSE OR USE OF THE SOFTWARE, INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR LOSS
OF ANTICIPATED PROFITS, EVEN IF CHELSIO HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES. IN NO EVENT SHALL CHELSIO'S LIABILITY ARISING OUT OF OR RELATED TO THE
LICENSE OR USE OF THE SOFTWARE EXCEED THE AMOUNTS PAID BY YOU FOR THE LICENSE GRANTED
HEREUNDER. THESE LIMITATIONS SHALL APPLY NOTWITHSTANDING ANY FAILURE OF ESSENTIAL
PURPOSE OF ANY LIMITED REMEDY.
9. High Risk Activities. The Software is not fault-tolerant and is not designed,
manufactured or intended for use or resale as online equipment control equipment in
hazardous environments requiring fail-safe performance, such as in the operation of
nuclear facilities, aircraft navigation or communication systems, air traffic control,
direct life support machines, or weapons systems, in which the failure of the Software
could lead directly to death, personal injury, or severe physical or environmental
damage. Chelsio specifically disclaims any express or implied warranty of fitness for
any high risk uses listed above.
10. Export. You acknowledge that the Software is of U.S. origin and subject to U.S.
export jurisdiction. You acknowledge that the laws and regulations of the United
States and other countries may restrict the export and re-export of the Software. You
agree that you will not export or re-export the Software or documentation in any form
in violation of applicable United States and foreign law. You agree to comply with all
applicable international and national laws that apply to the Software, including the
U.S.
Export Administration Regulations, as well as end-user, end-use, and destination
restrictions issued by U.S. and other governments.