Chelsio Communications Terminator Series User manual

Chelsio Unified Wire for FreeBSD ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [email protected]
Copyright © 2017. Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Chelsio KK (Japan)
Yamato Building 8F,
5-27-3 Sendagaya,
Shibuya-ku,
Tokyo 151-0051,
Japan

Chelsio Unified Wire for FreeBSD iii
Version History
Version
Revision Date
1.0.0
03/04/2013
1.0.1
08/29/2013
1.0.2
11/11/2016
1.0.3
12/05/2016
1.0.4
12/28/2016
1.0.5
01/27/2017
1.0.6
02/24/2017
1.0.7
05/10/2017
1.0.8
05/23/2017

Chelsio Unified Wire for FreeBSD iv
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 5
1. Introduction 6
1.1. Features 6
1.2. Hardware Requirements 6
1.3. Software Requirements 7
1.4. Package Contents 7
2. Hardware Installation 8
3. Software/Driver Installation 11
3.1. Compiling and Installing the kernel 11
3.2. Userland Tools 12
3.3. Firmware Update 13
4. Software/Driver Update 14
II. NETWORK (NIC-TOE) 15
1. Introduction 16
1.1. Hardware Requirements 16
1.2. Software Requirements 17
2. Software/Driver Loading 18
2.1. Loading NIC driver 18
2.2. Loading TOE driver 18
3. Software/Driver Configuration and Fine-tuning 19
3.1. Enabling network interface 19
3.2. Enabling TCP Offload 19
3.3. Enabling DDP 20
3.4. Connection Offload Policies (COP) 20
3.5. Network Device Configuration 21
4. Software/Driver Unloading 22
4.1. Unloading the NIC driver module 22
4.2. Unloading the TOE driver module 22
III. IWARP 23
1. Introduction 24
1.1. Hardware Requirements 24
1.2. Software Requirements 24
2. Software/Driver Loading 25
3. Software/Driver Configuration and Fine-tuning 26
3.1. Verifying iWARP Functionality 26
4. Software/Driver Unloading 27
IV. ISCSI PDU OFFLOAD TARGET 28
1. Introduction 29
1.1. Hardware Requirements 29
1.2. Software Requirements 30
2. Software/Driver Loading 31

Chelsio Unified Wire for FreeBSD v
2.1. Module Installation 31
2.2. Module Loading 31
3. Software/Driver Configuration and Fine-tuning 33
3.1. Starting iSCSI Target 33
4. Software/Driver Unloading 34
V. CLASSIFICATION AND FILTERING 35
1. Introduction 36
1.1. Hardware Requirements 36
1.2. Software Requirements 37
2. Usage 38
2.1. Configuration 38
2.2. Creating Filter Rules 38
2.3. Listing Filter Rules 40
2.4. Removing Filter Rules 40
VI. OFFLOAD IPV6 41
1. Introduction 42
1.1. Hardware Requirements 42
1.2. Software Requirements 42
2. Software/Driver Configuration and Fine-tuning 43
VII. APPENDIX 44
Chelsio End-User License Agreement (EULA) 45

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 5
I.Chelsio Unified Wire

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 6
1.Introduction
Thank you for choosing Chelsio Unified Wire adapters. These high speed, single chip, single
firmware cards provide enterprises and data centers with high performance solutions for various
Network and Storage related requirements.
The Terminator series is Chelsio’s next generation of highly integrated, hyper-virtualized
controllers. The adapters are built around a programmable protocol-processing engine, with full
offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, iSCSI, FCoE
and NAT support. It scales to true 100Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
Ideal for all data, storage and high performance clustering applications, the adapters enable
a unified fabric over a single wire by simultaneously running all unmodified IP sockets, Fibre
Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
1.1. Features
Chelsio Unified Wire driver package for FreeBSD contains Network, iSCSI and iWARP drivers
for Chelsio's Unified Wire adapters.
The Unified Wire Installer installs the following Chelsio drivers, features and software:
Network driver (NIC-TOE)
iWARP
iSCSI PDU Offload Target
Classification and Filtering
Offload IPv6
Please refer to their respective sections for more information.
1.2. Hardware Requirements
The Chelsio Unified Wire supports all x64 architectures supporting PCIE (x4, x8, x16) slots.
AMD CPUs, 64-bit (x86_64/amd64)
Intel CPUs, 64-bit (x86_64)
The Chelsio Unified Wire supports 3.3v PCI bus only. Running an adapter on a
PCI x4 slot is not recommended as performance will be significantly reduced by
the limitations of PCI.
Note

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 7
1.3. Software Requirements
The Chelsio Unified Wire software has been developed to run on FreeBSD based platforms. To
know more about the complete list of versions supported by each driver, please refer to their
respective sections.
1.4. Package Contents
The Chelsio Unified Wire Source Package consists of the following files/directories:
kernel-head: Source code for different drivers/software.
docs: The docs directory contains support documents - README, Release Notes and
User’s Guide (this document) for the driver.
firmware: Firmware binaries and firmware configuration files.
EULA: Chelsio’s End User License Agreement.

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 8
2.Hardware Installation
1. Shutdown/power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIE
x8/x16 slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the transceiver modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8/x16 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber/twinax cable, multi-mode for short range (SR) optics or single-mode for long
range (LR) optics, to the Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Verify if the adapter was installed successfully by using the following command:
a. For T6 adapters:

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 9
b. For T5 adapters:
For Chelsio adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the card
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
Once driver is installed and loaded, run the following commands and examine the output to see
if the card is discovered.
[root@host~]# sysctl dev.t5nex.0.%desc
dev.t5nex.0.%desc: Chelsio T520-LL RNIC (rev 0), S/N:RE16130313,
E/C:0000000000000000
[root@host~]# dmesg
t5nex0: <Chelsio T520-LL-CR> mem 0xe5300000-0xe537ffff,0xe4000000-
0xe4ffffff,0xe5884000-0xe5885fff irq 16 at device 0.4 on pci1
t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 10
The above outputs indicate the hardware configuration of the card as well as the Serial number
of the card. As observed by the x8, the card is properly installed in an x8 slot on the machine
and the card is using MSI-X interrupts.
Network device names for Chelsio’s physical ports are assigned using the
following convention: the port farthest from the motherboard will appear as the
first Ethernet network interface. However, for T5 40G and T420-BT adapters, the
association of physical Ethernet ports and their corresponding network device
names is opposite. For these adapters, the port nearest to the motherboard will
appear as the first network interface.
Note

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 11
3.Software/Driver Installation
3.1. Compiling and Installing the kernel
1. Install FreeBSD 10.X/11.X from
ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/
2. If you haven’t done so already, download the tar-ball ChelsioUwire-FBSD-x.x.x.x.tar.gz from
Chelsio Download Center, http://service.chelsio.com
3. Untar the tar-ball using the following command:
[root@host~]# tar xvf ChelsioUwire-FBSD-x.x.x.x.tar.gz
4. This package has the entire FreeBSD world (operating system). You need to build entire
FreeBSD world since infiniband core driver compiles only within FreeBSD kernel and not
separately as a module.
Build the FreeBSD world (operating system) from the package as follows:
a. Add the following entry in /etc/src.conf file:
WITH_OFED='yes'
b. Create a backup of existing kernel source.
[root@host~]# mv /usr/src /usr/src_bkp
c. Copy the contents of ChelsioUwire-FBSD-x.x.x.x/kernel-head to /usr/src/
d. Build and install the world as follows:
[root@host~]# cd usr/src/
[root@host~]# make buildworld
[root@host~]# make installworld

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 12
e. Build and install the kernel as follows:
[root@host~]# make buildkernel
[root@host~]# make installkernel
f. Add the following entries to /boot/loader.conf to enable capabilities:
hw.cxgbe.config_file="flash"
hw.cxgbe.iscsicaps_allowed=0xf
g. Reboot the machine for changes to take effect.
h. After reboot, check if you have booted to the right kernel by executing uname -a.
You should see a similar output:
5. Verify that the kernel supports TOE:
[root@host~]# config -x /boot/kernel/kernel | grep TCP_OFFLOAD
options TCP_OFFLOAD
3.2. Userland Tools
Build and install cxgbetool
[root@host~]# cd /usr/src/usr.sbin/cxgbetool
[root@host~]# make && make install

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 13
3.3. Firmware Update
Firmware will have to be loaded manually. Run the following commands for T6 adapters:
[root@host~]# kldload if_cxgbe
[root@host~]# cxgbetool t6nexX loadfw ChelsioUwire-FBSD-
x.x.x.x/firmware/t6fw-x.xx.xx.x.bin
[root@host~]# cxgbetool t6nexX loadcfg ChelsioUwire-FBSD-
x.x.x.x/firmware/t6-config.txt
Reboot system for changes to take effect.
Firmware version can be verified using:
[root@host~]# sysctl -n dev.t6nex.X.firmware_version
For T5 adapters, use t5nex as the parent device, t5fw-x.xx.xx.x.bin as the
firmware binary file and t5-config.txt as the firmware configuration file. Similarly,
for T4 adapters, use t4nex, t4fw-x.xx.xx.x.bin and t4-config.txt respectively.
Note

Chapter I. Chelsio Unified Wire
Chelsio Unified Wire for FreeBSD 14
4.Software/Driver Update
For any distribution specific problems, please check README and Release Notes included in
the release for possible workaround.
Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various
software/drivers. You can also subscribe to our newsletter for the latest software updates.

Chapter II. Network (NIC-TOE)
Chelsio Unified Wire for FreeBSD 15
II.Network (NIC-TOE)

Chapter II. Network (NIC-TOE)
Chelsio Unified Wire for FreeBSD 16
1.Introduction
Chelsio’s Unified Wire adapters provide extensive support for NIC operation, including all
stateless offload mechanisms for both IPv4 and IPv6 (IP, TCP and UDP checksum offload, LSO
aka TSO, and assist mechanisms for accelerating LRO. A high performance fully offloaded and
fully featured TCP/IP stack meets or exceeds software implementations in RFC compliance.
Chelsio’s Terminator engine provides unparalleled performance through a specialized data flow
processor implementation and a host of features designed for high throughput and low latency
using standard size Ethernet frames. TCP offload is fully implemented in the hardware, thus
freeing the CPU from TCP/IP overhead. The freed CPU can be used for any computing needs.
The Network (cxgbe) driver provides support for PCI Express Ethernet adapters based on
the Chelsio Terminator ASIC.
1.1. Hardware Requirements
1.1.1. Supported Adapters
The following are the Chelsio adapters that are compatible with Chelsio network driver:
T62100-CR
T62100-LP-CR
T62100-SO-CR
T6425-CR
T6225-CR
T6225-LL-CR
T6225-SO-CR
T580-CR
T580-LP-CR
T580-SO-CR*
T540-CR
T520-CR
T520-LL-CR
T520-SO-CR*
T420-CR
T420-LL-CR
T440-CR
T440-LP-CR
T422-CR
T420-SO-CR*
T420-CX
T420-BT
T404-BT
*Only NIC supported

Chapter II. Network (NIC-TOE)
Chelsio Unified Wire for FreeBSD 17
1.2. Software Requirements
Currently the Network driver is available for the following version(s):
FreeBSD 10.3-RELEASE
FreeBSD 11.0-RELEASE
Other versions have not been tested and are not guaranteed to work.

Chapter II. Network (NIC-TOE)
Chelsio Unified Wire for FreeBSD 18
2.Software/Driver Loading
The driver(s) must be loaded by the root user. Any attempt to load the driver(s) as a regular
user will fail.
2.1. Loading NIC driver
Load the NIC driver using the command given below:
[root@host~]# kldload if_cxgbe
2.2. Loading TOE driver
The TOE driver module (t4_tom.ko) is needed only if you plan to use the TOE and iWARP
capabilities, and must be loaded manually. Run the following command to do so:
[[root@host~]# kldload t4_tom
The NIC module (if_cxgbe) by itself can be loaded / unloaded at any time, but
once the TOE driver module (t4_tom) is loaded, it will prevent both the modules
from being unloaded.
Note

Chapter II. Network (NIC-TOE)
Chelsio Unified Wire for FreeBSD 19
3.Software/Driver Configuration and Fine-tuning
3.1. Enabling network interface
Enable any Chelsio T6 network interface using the following command:
[root@host~]# ifconfig ccX <ip_address> up
For T5 and T4, use cxlX and cxgbeX respectively.
3.2. Enabling TCP Offload
Enable TCP offload on a T6 interface using ifconfig. An error usually indicates that t4_tom is not
loaded.
[root@host~]# ifconfig ccX toe
For T5 and T4, use cxlX and cxgbeX respectively.
If successful, TOE4 and TOE6 will be displayed in the list of the enabled options as shown in
example below:
When a TOE capable interface is the outbound path to a destination to which a new connection
is being established, the connection should be offloaded automatically. Verify with netstat:
[root@host~]# netstat -np tcp | grep toe