Chelsio Communications T5 Series User manual

Chelsio T5/T4 Network Driver For XenServer ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [e-mail]
Copyright © 2015.Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
370 San Aleso Ave.
Suite 100
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Fax: [phone]
Chelsio KK (Japan)
SHIMA Akasaka Bldg.
Minato-ku, Tokyo
Japan 107-0052
Tel: [phone]

Chelsio T5/T4 Network Driver For XenServer iii
Document History
Version
Revision Date
1.0.0
05/18/2012
1.0.1
04/24/2015

Chelsio T5/T4 Network Driver For XenServer iv
TABLE OF CONTENTS
1. Introduction 5
1.1. Features 5
1.2. Hardware Requirements 5
1.3. Software Requirements 6
1.4. Package Contents 7
2. Hardware Installation 8
3. Software/Driver Installation 11
3.1. During OS installation 11
3.2. Runtime Installation 12
3.3. Firmware Update 13
4. Software/Driver Loading 14
5. Software/Driver Configuration and Fine-tuning 15
5.1. Instantiate Virtual Functions (SR-IOV) 15
6. Software/Driver Unloading 16
7. Software/Driver Uninstallation 17
8. Software/Driver Update 18
9. Appendix 19
9.1. Chelsio End-User License Agreement (EULA) 19

Chapter 1. Introduction
Chelsio T5/T4 Network Driver For XenServer Page 5
1.Introduction
Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip,
single firmware cards provide enterprises and data centers with high performance solutions for
various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, ISCSI,
FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
The T4 adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified
Wire. The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and
Filtering.
Ideal for all data, storage and high performance clustering applications, the T5/T4 Adapters
enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets,
Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio T5/T4 adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
This document describes the installation, use and maintenance of Network driver for
XenServer and its various components.
1.1. Features
Chelsio’s Network driver package for XenServer is an ISO image which installs drivers
for Chelsio’s T5/T4 Unified Wire adapters on XenServer.
1.2. Hardware Requirements
1.2.1. Supported Adapters
The following are the currently shipping Chelsio adapters that are compatible with Chelsio
Network driver:
T520-BT
T502-BT
T580-CR
T580-SO-CR

Chapter 1. Introduction
Chelsio T5/T4 Network Driver For XenServer Page 6
T520-LL-CR
T520-SO-CR
T520-CR
T522-CR
T580-LP-CR
T540-CR
T420-CR
T420-LL-CR
T440-CR
T440-LP-CR
T420-BCH
T422-CR
T420-SO-CR
T420-CX
T420-BT
T404-BT
1.3. Software Requirements
1.3.1. XenServer Requirements
Chelsio’s Network driver has been developed to run on XenServer platforms. Currently the
software is available for the following version:
Citrix XenServer Host 6.5.0
Kernel version: 3.10.0+2
Other versions have not been tested and are not guaranteed to work.
1.3.2. Resource Requirements
Control Domain (Dom0) memory: Based on the number of 40/10/1G ports, driver allocates
memory enough to run at the desired rate. It is suggested to increase the control domain
memory to 2GB using the procedure explained in
http://support.citrix.com/article/CTX134951
SW-IOMMU Size: Increasing SW-IOMMU size by appending swiotlb=128 to
/boot/extlinux.conf avoids PCI-DMA: Out of SW-IOMMU space for <size> bytes
at device <BDF> messages.

Chapter 1. Introduction
Chelsio T5/T4 Network Driver For XenServer Page 7
1.4. Package Contents
The software supplied is an ISO image. It consists of the following:
RPM packages containing drivers and associated tools, firmware, etc.
Shell script to install the driver package.

Chapter 2. Hardware Installation
Chelsio T5/T4 Network Driver For XenServer Page 8
2.Hardware Installation
Follow the steps to install Chelsio Adapter in your system:
1. Shutdown and power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIE
x8/x16 slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 40/10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Run update-pciids command to download the current version of PCI ID list
[root@host~]# update-pciids
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 198k 100 198k 0 0 491k 0 --:--:-- --:--:-- --:--:-- 626k
Done.
13. Verify if the adapter was installed successfully by using the lspci command
a. For T5 adapters :
[root@host~]# lspci |grep -i Chelsio
07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.3 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.4 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified

Chapter 2. Hardware Installation
Chelsio T5/T4 Network Driver For XenServer Page 9
Wire Ethernet Controller
07:00.5 SCSI storage controller: Chelsio Communications Inc T520-LL-CR
Unified Wire Storage Controller
07:00.6 Fibre Channel: Chelsio Communications Inc T520-LL-CR Unified Wire
Storage Controller
b. And for T4 adapters :
[root@host~]# lspci | grep –i Chelsio
03:00.0 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.1 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.2 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.3 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.4 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire
Ethernet Controller
03:00.5 SCSI storage controller: Chelsio Communications Inc T420-CR Unified
Wire Storage Controller
03:00.6 Fibre Channel: Chelsio Communications Inc T420-CR Unified Wire
Storage Controller
03:00.7 Ethernet controller: Chelsio Communications Inc Device 0000
For Chelsio T5/T4 adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
Once driver is installed and loaded, examine the output of dmesg to see if the card is
discovered.
For T5 adapters:
eth4: Chelsio T520-LL rev 0 10GBASE-SFP RNIC PCIe 5 GT/s x8 MSI-X, Offload
capable
0000:07:00.4: S/N: PT03140177, P/N: 110116750C0

Chapter 2. Hardware Installation
Chelsio T5/T4 Network Driver For XenServer Page 10
And, for T4 adapters:
eth0: Chelsio T420-CR rev 2 10GBASE-SFP RNIC PCIe x8 MSI-X
0000:04:00.4: S/N: PT18111226, P/N: 110112140D0
The above outputs indicate the hardware configuration of the adapters as well as the Serial
numbers. As observed by the x8, the card is properly installed in an x8 slot on the machine and
using MSI-X interrupts.
Network device names for Chelsio’s physical ports are assigned using the following
convention: the port farthest from the motherboard will appear as the first network
interface. However, for T5 40G and T420-BT adapters, the association of physical
Ethernet ports and their corresponding network device names is opposite. For
these adapters, the port nearest to the motherboard will appear as the first network
interface.
Note