Chelsio Communications DPDK User manual

Chelsio DPDK Driver for Linux ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [e-mail]
Copyright © 2017. Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Fax: [phone]
Chelsio KK (Japan)
Yamato Building 8F,
5-27-3 Sendagaya,
Shibuya-ku,
Tokyo 151-0051,
Japan

Chelsio DPDK Driver for Linux iii
Document History
Version
Revision Date
1.0.0
07/08/2015
1.0.1
10/26/2015
1.0.2
04/01/2016
1.0.3
07/12/2016
1.0.4
12/30/2016
1.0.5
01/30/2017
1.0.6
02/24/2017

Chelsio DPDK Driver for Linux iv
TABLE OF CONTENTS
1. Introduction 5
1.1. Features 5
1.2. Hardware Requirements 6
1.3. Software Requirements 6
1.4. Package Contents 7
2. Hardware Installation 8
3. Software/Driver Installation 10
3.1. Firmware Update 11
3.2. Flashing Firmware Configuration File 11
4. Software/Driver Loading 12
5. Software/Driver Configuration and Fine Tuning 14
5.1. Huge Pages 14
5.2. Binding network ports 15
5.3. Unbinding network ports 16
5.4. Performance Tuning 16
6. Running DPDK Test Applications 17
6.1. Testpmd application 17
6.2. Pktgen application 20
7. Classification and Filtering 24
7.1. Pre-requisites 24
7.2. Usage 24
7.3. Layer 2 example 33
7.4. Forwarding multi-cast traffic 35
7.5. Filtering using outer VLAN 37
8. Software/Driver Unloading 38
9. Software/Driver Uninstallation 39
10. Software/Driver Update 40
11. Appendix 41
11.1. Chelsio End-User License Agreement (EULA) 41

Chapter 1. Introduction
Chelsio DPDK Driver for Linux 5
1.Introduction
Thank you for choosing Chelsio Unified Wire adapters. These high speed, single chip, single
firmware cards provide enterprises and data centers with high performance solutions for various
Network and Storage related requirements.
The Terminator series is Chelsio’s next generation of highly integrated, hyper-virtualized
1/10/25/40/50/100GbE controllers. The adapters are built around a programmable protocol-
processing engine, with full offload of a complete Unified Wire solution comprising NIC, TOE,
iWARP RDMA, iSCSI, FCoE and NAT support. It scales to true 100Gb line rate operation from a
single TCP connection to thousands of connections, and allows simultaneous low latency and
high bandwidth operation thanks to multiple physical channels through the ASIC.
Ideal for all data, storage and high performance clustering applications, the Unified Wire
adapters enable a unified fabric over a single wire by simultaneously running all unmodified IP
sockets, Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
This document describes the installation, use and maintenance of the DPDK driver and its
various components.
1.1. Features
Chelsio Data Plane Development Kit (DPDK) driver package is a collection of data plane
libraries and NIC drivers optimized for running in the Linux user space to boost packet
processing.
The driver has support for:
Multiple queues for Tx and Rx
Receive Side Scaling (RSS)
VLAN filtering
Checksum offload
Promiscuous mode
All multicast mode
Port hardware statistics
Jumbo frames (only for UIO)
Classification and Filtering (only for UIO)

Chapter 1. Introduction
Chelsio DPDK Driver for Linux 6
1.2. Hardware Requirements
The following are the currently shipping Chelsio adapters that are compatible with DPDK driver:
T62100-CR
T62100-LP-CR
T62100-SO-CR
T6225-CR
T6225-SO-CR
T580-CR
T580-LP-CR
T580-SO-CR
T540-CR
T520-CR
T520-LL-CR
T520-SO-CR
T520-BT
1.3. Software Requirements
The Chelsio DPDK driver has been developed to run on 64-bit Linux platforms. Following is the
list of supported distributions:
Linux Distribution
Driver/Software
RHEL 7.2, 3.10.0-327.el7
UIO, VFIO, Filtering
RHEL 7.2, 3.10.0-327.el7.ppc64le (POWER8 LE)
VFIO*
RHEL 7.1, 3.10.0-229.el7
UIO, VFIO, Filtering
* Alpha release. Will be fully functional in the next release.
Other kernel versions have not been tested and are not guaranteed to work.
flex, bison, byacc, patch, patchutils, autoconf, automake and rpm-build packages
must be present in the machine (required for libpcap installation).
Note

Chapter 1. Introduction
Chelsio DPDK Driver for Linux 7
1.4. Package Contents
The Chelsio DPDK driver package consists of the following files/directories:
Makefile: Makefile for building and installing Chelsio DPDK driver and tools.
EULA: Chelsio’s End User License Agreement.
src: Source code for Chelsio DPDK driver, Kernel driver and Tools.
scripts: Contains res_hugepages.sh shell script used to reserve Huge Page memory.
docs: Support documents - README, Release Notes and User’s Guide (this document) for
the driver.

Chapter 2. Hardware Installation
Chelsio DPDK Driver for Linux 8
2.Hardware Installation
Follow these steps to install Chelsio Adapter in your system:
i. Shutdown/power off your system.
ii. Power off all remaining peripherals attached to your system.
iii. Unpack the Chelsio adapter and place it on an anti-static surface.
iv. Remove the system case cover according to the system manufacturer’s instructions.
v. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
vi. For maximum performance, it is highly recommended to install the adapter into a PCIE
x8/x16 slot.
vii. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
viii. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
ix. After securing the card, ensure that the card is still fully seated in the PCIE x8/x16 slot as
sometimes the process of securing the card causes the card to become unseated.
x. Connect a fiber/twinax cable, multi-mode for short range (SR) optics or single-mode for long
range (LR) optics, to the Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
xi. Power on your system.
xii. Run update-pciids command to download the current version of PCI ID list
[root@host~]# update-pciids
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 198k 100 198k 0 0 491k 0 --:--:-- --:--:-- --:--:-- 626k
Done.
xiii. Verify if the adapter was installed successfully by using the lspci command:

Chapter 2. Hardware Installation
Chelsio DPDK Driver for Linux 9
For Chelsio adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
xiv. Install and load the appropriate network driver
xv. Finally, verify if the card is discovered:
The above outputs indicate the hardware configuration of the adapters as well as the Serial
numbers.
Network device names for Chelsio’s physical ports are assigned using the following
convention: the port farthest from the motherboard will appear as the first Ethernet
network interface. However, for T5 40G adapters, the association of physical
Ethernet ports and their corresponding network device names is opposite. For this
adapter, the port nearest to the motherboard will appear as the first network
interface.
Note

Chapter 3. Software/Driver Installation
Chelsio DPDK Driver for Linux 10
3.Software/Driver Installation
i. If you haven't done already, download the driver package Chelsio-DPDK-x.x.x.x.tar.gz.
ii. Untar the tar ball:
[root@host~]# tar zxvf Chelsio-DPDK-x.x.x.x.tar.gz
iii. Change your working directory to Chelsio-DPDK -x.x.x.x directory:
[root@host~]# cd Chelsio-DPDK-x.x.x.x
iv. Compile and install the driver using one of the following options:
To install DPDK source, testpmd and Pktgen tool only:
[root@host~]# make dpdk_install
To install DPDK source and all the tools provided in the DPDK repository:
[root@host~]# make install
The above commands will
install the driver with the following build configuration:
arch: x86_64/ppc_64
machine: native/power8
execenv: linuxapp
toolchain: gcc
create a target environment directory "x86_64-native-linuxapp-gcc/ppc_64-
power8-linuxapp-gcc”in the driver package directory.
Note
Please ensure that any existing version of DPDK driver is uninstalled before
proceeding.
Important
The pktgen application has been modified to disable packet classification, to reflect
correct Rx rate as packet classification is an expensive operation. It will not be
installed on POWERPC64 architecture.
Note

Chapter 3. Software/Driver Installation
Chelsio DPDK Driver for Linux 11
3.1. Firmware Update
Firmware is installed on the system, typically in /lib/firmware/cxgb4, and the kernel mode
driver (cxgb4) will auto-load the firmware if an update is required:
Load the kernel mode NIC driver (cxgb4):
[root@host~]# modprobe cxgb4
Verify the firmware version using ethtool:
[root@host~]# ethtool -i <iface>
3.2. Flashing Firmware Configuration File
i. If not done already, load the kernel mode NIC driver (cxgb4):
[root@host~]# modprobe cxgb4
ii. Flash the firmware configuration file (for T6 adapters):
[root@host~]# cxgbtool <iface> loadcfg /lib/firmware/cxgb4/t6-config.txt
For T5 adapters, use t5-config.txt.
Note

Chapter 4. Software/Driver Loading
Chelsio DPDK Driver for Linux 12
4.Software/Driver Loading
The driver must be loaded by the root user. Any attempt to load as a regular user will fail.
UIO Support
Follow the steps mentioned below to load DPDK driver with UIO support:
i. Disable Intel VT-d in system BIOS.
ii. Turn off Intel iommu by adding below entry in Kernel grub/grub2 menu:
intel_iommu=off
iii. Reboot the system for changes to take effect.
iv. Load the UIO module:
[root@host~]# modprobe uio
[root@host~]# insmod Chelsio-DPDK-x.x.x.x/src/DPDK/x86_64-native-linuxapp-
gcc/kmod/igb_uio.ko
VFIO Support
Follow the steps mentioned below to load DPDK driver with VFIO support:
i. Enable Intel VT-d in system BIOS.
ii. Add the following entry to grub/grub2 menu:
intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
E.g.:
kernel /vmlinuz-3.13.0-32-generic ro root=UUID=5149fae1-c52b-42a9-a48c-
b0a70937e8fb intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
crashkernel=128M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Before proceeding, unload all Chelsio inbox and outbox kernel mode drivers using
the following commands:
[root@host~]# rmmod cxgb4
[root@host~]# rmmod csiostor
Note

Chapter 4. Software/Driver Loading
Chelsio DPDK Driver for Linux 13
iii. Reboot the system for changes to take effect.
iv. Load the VFIO module:
[root@host~]# modprobe vfio-pci

Chapter 5. Software/Driver Configuration and Fine Tuning
Chelsio DPDK Driver for Linux 14
5.Software/Driver Configuration and Fine Tuning
5.1. Huge Pages
5.1.1. Using script
Run the res_hugepages.sh shell script (copied to /sbin during installation):
[root@host~]# res_hugepages.sh
5.1.2. Manual
i. Mount hugetlbfs:
[root@host~]# mkdir -p /mnt/huge
[root@host~]# mount -t hugetlbfs nodev /mnt/huge
ii. Reserve Huge Page memory manually:
x86_64
[root@host~]# echo 1024 >
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
POWERPC64
[root@host~]# echo 512 > /sys/devices/system/node/node0/hugepages/hugepages-
16384kB/nr_hugepages
In case of dual socket machines, run the above command for other CPU nodes.

Chapter 5. Software/Driver Configuration and Fine Tuning
Chelsio DPDK Driver for Linux 15
5.2. Binding network ports
Run the following command to bind network ports to DPDK environment, with UIO support:
[root@host~]# dpdk_nic_bind.py --bind=igb_uio <PCI-ID-PF4>
Now, verify using:
[root@host~]# dpdk_nic_bind.py –status
Example:
Run the following command to bind network ports to DPDK environment, with VFIO
support:
[root@host~]# dpdk_nic_bind.py --bind=vfio-pci <PCI-ID-PF4>
Now, verify using:
[root@host~]# dpdk_nic_bind.py –status
Example:
Please make sure that adapter's physical function 4 is used, since it is assigned
for all NIC functions of the adapter.
Note

Chapter 5. Software/Driver Configuration and Fine Tuning
Chelsio DPDK Driver for Linux 16
5.3. Unbinding network ports
i. Run the following command to unbind network ports from DPDK environment:
[root@host~]# dpdk_nic_bind.py -u <PCI-ID-PF4>
ii. Run the following command to verify:
[root@host~]# dpdk_nic_bind.py –status
5.4. Performance Tuning
In order to auto tune the system for best performance, Chelsio recommends:
disabling virtualization, c-state technology, Intel I/O AT and SR-IOV in the BIOS settings.
setting the Power Profile to Maximum Performance in BIOS settings.
installing the adapter into a PCIe Gen3 x8/x16 slot.
using traffic with multiple tuples. The 'range' option of DPDK-Pktgen app can be used to
create traffic with different 4-tuple values, so as to take advantage of RSS and hence, allow
the traffic to be distributed across different queues/CPUs in Rx direction.

Chapter 6. Running DPDK Test Applications
Chelsio DPDK Driver for Linux 17
6.Running DPDK Test Applications
6.1. Testpmd application
The Testpmd application is provided as a part of the Chelsio DPDK driver package, and can be
used to test the DPDK in a packet forwarding mode and also to access NIC hardware features
such as Flow Director.
6.1.1. Syntax
Execute the following command to test DPDK using the Testpmd application:
[root@host~]# testpmd -c <coremask> -n <channels> -- -i --nb-cores=<cores> -
-txq=<TxQueue> --rxq=<RxQueue> --max-pkt-len=<PacketSize>
Where,
-c <coremask>
: A hexadecimal bit mask of the cores to run on. Note that core
numbering can change between platforms and should be
determined beforehand.
-n <channels>
: Number of memory channels per processor socket.
-i
: Enable interactive mode.
--nb-cores=<cores>
: Number of forwarding cores.
--txq=<TxQueue>
: Number of Tx queues.
--rxq=<RxQueue>
: Number of Rx queues.
--max-pkt-len=<PacketSize>
: Enable Jumbo mode and specify the packet size allowed for
forwarding.
6.1.2. Examples
i. To run traffic using single Tx and Rx queue:
[root@host~]# testpmd -c f -n 4 -- -i --nb-cores=2 --txq=1 --rxq=1
ii. To run traffic using 4 Tx and Rx queues:
[root@host~]# testpmd -c 3ff -n 4 -- -i --nb-cores=8 --txq=4 --rxq=4
In case of POWERPC64, if testpmd is not initializing, reallocate the huge pages
using the setup script “setup.sh”present in <driver-package>/src/DPDK/tools/
directory.
Note

Chapter 6. Running DPDK Test Applications
Chelsio DPDK Driver for Linux 18
6.1.3. Flow Control
Flow control pause TX/RX is disabled by default and can be enabled via Testpmd as follows:
testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg
off 0
testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg
off 1
To disable again, run:
testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg
on 0
testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg
on 1
6.1.4. RSS
RSS is enabled by default and can be disabled via Testpmd as follows:
testpmd> port config all rss none
To enable again, run:
testpmd> port config all rss [all|ip|tcp|udp|sctp|ether|none]
6.1.5. Jumbo Mode
There are multiple ways to enable sending and receiving of jumbo frames: the first method
involves using the command prompt and the other two via testpmd application.
Command Prompt
Run the following command at the prompt:
[root@host~]# testpmd -c ffff -n 3 -- -i --nb-cores=8 --txq=4 --rxq=4 --max-
pkt-len=9018

Chapter 6. Running DPDK Test Applications
Chelsio DPDK Driver for Linux 19
Testpmd
i. This method involves using the mtu command, which changes the MTU of an individual port
without having to stop the selected port. To configure each port individually, run the mtu
command as follows:
testpmd> port config mtu 0 9000
testpmd> port config mtu 1 9000
ii. This method involves stopping all the ports first and then running max-pkt-len command to
configure the MTU of all the ports with a single command.To configure all the ports at once,
stop all the ports first and run the max-pkt-len command as follows:
testpmd> port stop all
testpmd> port config all max-pkt-len 9000

Chapter 6. Running DPDK Test Applications
Chelsio DPDK Driver for Linux 20
6.2. Pktgen application
Pktgen is a traffic generator application capable of displaying real time metrics for ports and can
handle packets with UDP, TCP, ARP, ICMP, GRE, MPLS and Queue-in-Queue. The application
can run command scripts to set up repeatable test cases.
6.2.1. Syntax
Execute the following command to run Pktgen application. Observe unidirectional pkts/sec and
throughput performance under tool output:
[root@host~]# pktgen -c <coremask> -J -n <channels> -- -T -P -m
"<[cores]>.<port_id>" -N
Where,
-c <coremask>
: A hexadecimal bit mask of the cores to run on. Note that core
numbering can change between platforms and should be determined
beforehand.
-n <channels>
: Number of memory channels per processor socket.
-T
: Enable pktgen themes.
-P
: Enable PROMISCUOUS mode on all ports.
-m <[cores]>.<port_id>
: Map CPU logical cores to Chelsio ports.
-N
: Enable NUMA support.
-J
: Enable Jumbo mode.
6.2.2. Examples
Running single Tx and Rx queue traffic
i. Configure Pktgen tool to use single Tx and Rx queue, mapping core 1 to Rx and core 2 to
Tx queue on port 0:
[root@host~]# pktgen -c f -n 4 -- -T -P -m "[1:2].0" –N
.
.
.
Pktgen created by Keith Wiles -- >>> Powered by Intel® DPDK <<<
- Ports 0-3 of 4 ** Main Page ** Copyright (c) <2010-2015>, Wind River
Systems, Inc. All rights reserved. Powered by Intel® DPDK
Flags:Port : P-------------:0
The pktgen application has been modified to disable packet classification, to reflect
correct Rx rate as packet classification is an expensive operation. It will not be
installed on POWERPC64 architecture.
Note