Chelsio T5 User manual

Chelsio T5 Network Driver for OpenIndiana ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [e-mail]
Copyright © 2016. Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Chelsio KK (Japan)
Yamato Building 8F,
5-27-3 Sendagaya,
Shibuya-ku,
Tokyo 151-0051,
Japan

Chelsio T5 Network Driver for OpenIndiana iii
Document History
Version
Revision Date
1.0.0
01/08/2013
1.0.1
02/14/2014
1.0.2
02/24/2014
1.0.3
06/30/2014
1.0.4
07/25/2016

Chelsio T5 Network Driver for OpenIndiana iv
TABLE OF CONTENTS
1. Introduction 5
1.1. Features 5
1.2. Hardware Requirements 5
1.3. Software Requirements 6
1.4. Package Contents 6
2. Hardware Installation 7
3. Software/Driver Installation 9
4. Software/Driver Configuration and Fine-tuning 10
4.1. Kernel tuning 10
4.2. Firmware update 10
4.3. Troubleshooting 11
5. Software/Driver Uninstallation 13
6. Software/Driver Update 14
7. Appendix 15
7.1. Chelsio End-User License Agreement (EULA) 15

Chapter 1. Introduction
Chelsio T5 Network Driver for OpenIndiana 5
1.Introduction
Thank you for choosing Chelsio T5 Unified Wire adapters. These high speed, single chip,
single firmware cards provide enterprises and data centers with high performance solutions for
various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, ISCSI,
FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
Ideal for all data, storage and high performance clustering applications, the T5 adapters
enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets,
Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio T5 adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
1.1. Features
This driver package has the Chelsio T5 ASIC based 40Gb and 10Gb PCI Express network
adapter driver for OpenIndiana. The driver package installs two kernel modules - t4nex and
cxgbe. The t4nex is the bus nexus driver that attaches to the Chelsio adapter. The cxgbe is the
network driver for each port and attaches to the t4nex bus nexus. Chelsio network interfaces
appear as cxgbeX by default, where X is any number assigned to each interface by the kernel.
The driver supports Jumbo Frames, Transmit/Receive checksum offload, and TCP
segmentation offload (TSO).
1.2. Hardware Requirements
1.2.1. Supported Adapters
The following are the currently shipping Chelsio adapters that are compatible with Chelsio
OpenIndiana Network driver:
T580-CR
T580-LP-CR
T580-SO-CR
T540-CR
T520-LL-CR

Chapter 1. Introduction
Chelsio T5 Network Driver for OpenIndiana 6
T520-SO-CR
T520-CR
T520-BT
1.3. Software Requirements
The Chelsio network driver has been developed to run on Intel based OpenIndiana platforms.
Currently the driver is available for following version:
OpenIndiana Hipster 2016.04
Other versions have not been tested and are not guaranteed to work.
1.4. Package Contents
The Chelsio network driver package for OpenIndiana consists of the following:
tools:
cxgbetool: The cxgbetool queries or sets various aspects of Chelsio network
interface cards. It complements standard tools used to configure network settings
and provides functionality not available through such tools.
CHLScxgbe.i386.pkg: driver binary.
docs: The docs directory contains support documents - README, Release Notes and
User’s Guide (this document) for the software.

Chapter 2. Hardware Installation
Chelsio T5 Network Driver for OpenIndiana 7
2.Hardware Installation
Follow these steps to install Chelsio Adapter in your system:
1. Shutdown/power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the 10Gb Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIE
x8/x16 slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 40/10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Verify if the Chelsio adapter was installed successfully by using the following command:
root@host:~# scanpci –v
.
.
pci bus 0x0007 cardnum 0x00 function 0x04: vendor 0x1425 device 0x5410
Chelsio Communications Inc T580-LP-CR Unified Wire Ethernet Controller
CardVendor 0x1425 card 0x0000 (Chelsio Communications Inc, Card unknown)
STATUS 0x0010 COMMAND 0x0046
CLASS 0x02 0x00 0x00 REVISION 0x00
BIST 0x00 HEADER 0x80 LATENCY 0x00 CACHE 0x40
BASE0 0xf8f80000 SIZE 524288 MEM
BASE2 0xf9000000 SIZE 16777216 MEM
BASE4 0xf8f7e000 SIZE 8192 MEM
BASEROM 0x00000000 addr 0x00000000
MAX_LAT 0x00 MIN_GNT 0x00 INT_PIN 0x01 INT_LINE 0x0b
.
.

Chapter 2. Hardware Installation
Chelsio T5 Network Driver for OpenIndiana 8
Once driver is installed and loaded, examine the output of kstat t4nex to see if the card is
discovered:
module: t4nex instance: 0
name: config class: nexus
bus_speed 3
bus_type pci-express
bus_width x8 lanes
chip_ver 0
core_clock 500000
crtime 343330.811561770
driver_version 4.0.0.3c253+
ec_level 000000000000000
fw_vers 1.15.37.0
id T580-LP-CR
part_number 110118750C0
pci_device_id 0x5410
pci_vendor_id 0x1425
port_cnt 2
port_type 40G/40G/-/-
serial_number PT24151111
snaptime 343537.865818717
tp_vers 0.1.4.9
The above output indicates the hardware configuration of the card as well as the serial number.
Network device names for Chelsio’s physical ports are assigned using the following
convention: the port farthest from the motherboard will appear as the first Ethernet
interface. However, for T5 40G adapters, the association of physical Ethernet ports
and their corresponding network device names is opposite. For these adapters, the
port nearest to the motherboard will appear as the first network interface.
.
Note

Chapter 3. Software/Driver Installation
Chelsio T5 Network Driver for OpenIndiana 9
3.Software/Driver Installation
The driver must be installed by the root user. Any attempt in installing the driver as a regular
user will fail.
i. If you haven’t done already, download the tarball, cxgbe-openindiana-x.x.x.x.tar.gz from
Chelsio Download Center, http://service.chelsio.com/
ii. Remove any previous versions of the driver installed in the system. Driver uninstallation is
explained in the Software/Driver Uninstallation section.
iii. Install the driver package using the following commands:
root@host:~# tar zxfvm cxgbe-openindiana-x.x.x.x.tar.gz
root@host:~# cd cxgbe-openindiana-x.x.x.x
root@host:~# pkgadd -d CHLScxgbe.i386.pkg
iv. Reboot the system.
v. After reboot, run the following command before using the cxgbe devices. This command will
reconfigure new attached devices without returning any output.
root@host:~# devfsadm
root@host:~#
Please follow the above step on every reboot.
Note

Chapter 4. Software/Driver Configuration and Fine-tuning
Chelsio T5 Network Driver for OpenIndiana 10
4.Software/Driver Configuration and Fine-tuning
4.1. Kernel tuning
The driver's behavior and performance can be influenced by a variety of kernel tunables.
General information about TCP/IP tuning on OpenIndiana can be found at
http://www.solarisinternals.com/wiki/index.php/Networks
The use of MSI-X interrupts is recommended. This can be achieved by adding below line(s) to
the /etc/system (depending on the OpenIndiana version). A server restart is necessary for the
changes in /etc/system to be effective.
set ddi_msix_alloc_limit=32
OR
set ddi_msix_alloc_limit=32
set pcplusmp:apic_multi_msi_max=16
set pcplusmp:apic_msix_max=16
set pcplusmp:apic_intr_policy=1
4.2. Firmware update
The driver will auto-load T5 firmware if an update is required. The firmware version can be
verified using
root@host:~# kstat t4nex

Chapter 4. Software/Driver Configuration and Fine-tuning
Chelsio T5 Network Driver for OpenIndiana 11
4.3. Troubleshooting
Run the commands in this section in order, save their output, and contact [email protected]
if you have any problems installing or running the driver.
What system is this?
root@host:~# uname –a
root@host:~# cat /etc/release
Is the driver installed? Is it the right version?
root@host:~# pkginfo | grep CHLS
root@host:~# pkginfo -l CHLScxgbe
root@host:~# /usr/xpg4/bin/grep -e cxgbe -e t4nex /etc/driver_aliases
root@host:~# /usr/xpg4/bin/grep -e cxgbe -e t4nex /etc/name_to_major
Are there any Chelsio cards in the system? (Look for 1425 in the output of these
commands.)
root@host:~# scanpci (or /usr/X11/bin/scanpci)
root@host:~# prtconf -p
Is the driver loaded? Did it attach to the Chelsio cards in the system? (Look for 1425, cxgbe,
t4nex in the output of these commands.)
root@host:~# modinfo | /usr/xpg4/bin/grep -e cxgbe -e t4nex
root@host:~# dmesg
root@host:~# prtconf -v
Are the network ports registered with the kernel?
root@host:~# dladm show-link
root@host:~# dladm show-linkprop
root@host:~# ifconfig -a

Chapter 4. Software/Driver Configuration and Fine-tuning
Chelsio T5 Network Driver for OpenIndiana 12
Is a link established? Can packets be transmitted and received?
root@host:~# ifconfig cxgbeX plumb a.b.c.d/nm up
where a.b.c.d is the IP address and nm is the netmask for cxgbeX interface
root@host:~# kstat cxgbe:X:mac:link_state
(where X is the interface number. For example, 0 for cxgbe0)
root@host:~# netstat -I cxgbeX 1
Run the above command while attempting to send/receive traffic.
Detailed information about the driver state and statistics.
root@host:~# kstat t4nex
root@host:~# kstat cxgbe

Chapter 5. Software/Driver Uninstallation
Chelsio T5 Network Driver for OpenIndiana 13
5.Software/Driver Uninstallation
Make sure that there are no VLANs/VNICs and aggregations using any of the Chelsio
interfaces. If there are any, unplumb and destroy them before proceeding. The package
uninstaller unplumbs all of the plumbed cxgbeX interfaces.
Uninstall the package using package manager:
root@host:~# pkgrm CHLScxgbe

Chapter 6. Software/Driver Update
Chelsio T5 Network Driver for OpenIndiana 14
6.Software/Driver Update
For any distribution specific problems, please check README and Release Notes included in
the release for possible workaround.
Please visit Chelsio Download Center, http://service.chelsio.com/, for regular updates on
various software/drivers. You can also subscribe to our newsletter for the latest software
updates.

Chapter 7. Appendix
Chelsio T5 Network Driver for OpenIndiana 15
7.Appendix
7.1. Chelsio End-User License Agreement (EULA)
Installation and use of the driver/software implies acceptance of the terms in the Chelsio End-
User License Agreement (EULA).
IMPORTANT: PLEASE READ THIS SOFTWARE LICENSE CAREFULLY BEFORE DOWNLOADING OR OTHERWISE
USING THE SOFTWARE OR ANY ASSOCIATED DOCUMENTATION OR OTHER MATERIALS
(COLLECTIVELY, THE "SOFTWARE"). BY CLICKING ON THE "OK" OR "ACCEPT" BUTTON YOU AGREE
TO BE BOUND BY THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO THE TERMS OF THIS
AGREEMENT, CLICK THE "DO NOT ACCEPT" BUTTON TO TERMINATE THE INSTALLATION PROCESS.
1. License. Chelsio Communications, Inc. ("Chelsio") hereby grants you, the Licensee,
and you hereby accept, a limited, non-exclusive, non-transferable license to install
and use the Software with one or more Chelsio network adapters on a single server
computer for use in communicating with one or more other computers over a network. You
may also make one copy of the Software in machine readable form solely for back-up
purposes, provided you reproduce Chelsio's copyright notice and any proprietary
legends included with the Software or as otherwise required by Chelsio.
2. Restrictions. This license granted hereunder does not constitute a sale of the
Software or any copy thereof. Except as expressly permitted under this Agreement, you
may not:(i) reproduce, modify, adapt, translate, rent, lease, loan, resell,
distribute, or create derivative works of or based upon, the Software or any part
thereof; or (ii) make available the Software, or any portion thereof, in any form, on
the Internet. The Software contains trade secrets and, in order to protect them, you
may not decompile, reverse engineer, disassemble, or otherwise reduce the Software to
a human-perceivable form. You assume full responsibility for the use of the Software
and agree to use the Software legally and responsibly.
3. Ownership of Software. As Licensee, you own only the media upon which the Software
is recorded or fixed, but Chelsio retains all right, title and interest in and to the
Software and all subsequent copies of the Software, regardless of the form or media in
or on which the Software may be embedded.
4. Confidentiality. You agree to maintain the Software in confidence and not to
disclose the Software, or any information or materials related thereto, to any third
party without the express written consent of Chelsio. You further agree to take all
reasonable precautions to limit access of the Software only to those of your employees
who reasonably require such access to perform their employment obligations and who are
bound by confidentiality agreements with you.
5. Term. This license is effective in perpetuity, unless terminated earlier. You
may terminate the license at any time by destroying the Software (including the
related documentation), together with all copies or modifications in any form. Chelsio
may terminate this license, and this license shall be deemed to have automatically
terminated, if you fail to comply with any term or condition of this Agreement. Upon
any termination, including termination by you, you must destroy the Software
(including the related documentation), together with all copies or modifications in
any form.

Chapter 7. Appendix
Chelsio T5 Network Driver for OpenIndiana 16
6. Limited Warranty. If Chelsio furnishes the Software to you on media, Chelsio
warrants only that the media upon which the Software is furnished will be free from
defects in material or workmanship under normal use and service for a period of thirty
(30) days from the date of delivery to you. CHELSIO DOES NOT AND CANNOT WARRANT THE
PERFORMANCE OR RESULTS YOU MAY OBTAIN BY USING THE SOFTWARE OR ANY PART THEREOF.
EXCEPT FOR THE FOREGOING LIMITED WARRANTY, CHELSIO MAKES NO OTHER WARRANTIES, EXPRESS
OR IMPLIED, AND HEREBY DISCLAIMS ALL OTHER WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
NON-INFRINGEMENT OF THIRD PARTY RIGHTS, MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. Some states do not allow the exclusion of implied warranties or limitations
on how long an implied warranty may last, so the above limitations may not apply to
you. This warranty gives you specific legal rights and you may also have other rights
which vary from state to state.
7. Remedy for Breach of Warranty. The sole and exclusive liability of Chelsio and
its distributors, and your sole and exclusive remedy, for a breach of the above
warranty, shall be the replacement of any media furnished by Chelsio not meeting
the above limited warranty and which is returned to Chelsio. If Chelsio or its
distributor is unable to deliver replacement media which is free from defects in
materials or workmanship, you may terminate this Agreement by returning the Software.
8. Limitation of Liability. IN NO EVENT SHALL CHELSIO HAVE ANY LIABILITY TO YOU OR
ANY THIRD PARTY FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR PUNITIVE
DAMAGES, HOWEVER CAUSED, AND ON ANY THEORY OF LIABILITY, ARISING OUT OF OR RELATED
TO THE LICENSE OR USE OF THE SOFTWARE, INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR LOSS OF ANTICIPATED PROFITS, EVEN IF CHELSIO HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES. IN NO EVENT SHALL CHELSIO'S LIABILITY ARISING OUT OF
OR RELATED TO THE LICENSE OR USE OF THE SOFTWARE EXCEED THE AMOUNTS PAID BY YOU FOR
THE LICENSE GRANTED HEREUNDER. THESE LIMITATIONS SHALL APPLY
NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED REMEDY.
9. High Risk Activities. The Software is not fault- tolerant and is not
designed, manufactured or intended for use or resale as online equipment control
equipment in hazardous environments requiring fail-safe performance, such as in
the operation of nuclear facilities, aircraft navigation or communication
systems, air traffic control, direct life support machines, or weapons systems, in
which the failure of the Software could lead directly to death, personal injury,
or severe physical or environmental damage. Chelsio specifically disclaims any
express or implied warranty of fitness for any high risk uses listed above.
10. Export. You acknowledge that the Software is of U.S. origin and subject to
U.S. export jurisdiction. You acknowledge that the laws and regulations of
the United States and other countries may restrict the export and re- export of
the Software. You agree that you will not export or re-export the Software or
documentation in any form in violation of applicable United States and foreign law.
You agree to comply with all applicable international and national laws that
apply to the Software, including the U.S. Export Administration Regulations, as well
as end-user, end-use, and destination restrictions issued by U.S. and other
governments.
11. Government Restricted Rights. The Software is subject to restricted rights as
follows. If the Software is acquired under the terms of a GSA contract: use,
reproduction or disclosure is subject to the restrictions set forth in the
applicable ADP Schedule contract. If the Software is acquired under the
terms of a DoD or civilian agency contract, use, duplication or disclosure by
the Government is subject to the restrictions of this Agreement in accordance with

Chapter 7. Appendix
Chelsio T5 Network Driver for OpenIndiana 17
48 C.F.R. 12.212 of the Federal Acquisition Regulations and its successors and 49
C.F.R. 227.7202-1 of the DoD FAR Supplement and its successors.
12. General. You acknowledge that you have read this Agreement, understand it,
and that by using the Software you agree to be bound by its terms and conditions.
You further agree that it is the complete and exclusive statement of the agreement
between Chelsio and you, and supersedes any proposal or prior agreement, oral
or written, and any other communication between Chelsio and you relating to the
subject matter of this Agreement. No additional or any different terms will be
enforceable against Chelsio unless Chelsio gives its express consent, including an
express waiver of the terms of this Agreement, in writing signed by an officer of
Chelsio. This Agreement shall be governed by California law, except as to copyright
matters, which are covered by Federal law. You hereby irrevocably submit to
the personal jurisdiction of, and irrevocably waive objection to the laying of
venue (including a waiver of any argument of forum non conveniens or other
principles of like effect) in, the state and federal courts located in Santa
Clara County, California, for the purposes of any litigation undertaken in
connection with this Agreement. Should any provision of this Agreement be declared
unenforceable in any jurisdiction, then such provision shall be deemed severable from
this Agreement and shall not affect the remainder hereof. All rights in the Software
not specifically granted in this Agreement are reserved by Chelsio. You may not
assign or transfer this Agreement (by merger, operation of law or in any
other manner) without the prior written consent of Chelsio and any attempt to do so
without such consent shall be void and shall constitute a material breach of this
Agreement.
Should you have any questions concerning this Agreement, you may contact Chelsio by
writing to:
Chelsio Communications, Inc.
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A