Chelsio T5 User manual

Chelsio T5/T4 Unified Wire for Windows ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [e-mail]
Copyright © 2013 Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
370 San Aleso Ave.
Suite 100
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Chelsio KK (Japan)
SHIMA Akasaka Bldg.
Minato-ku, Tokyo
Japan 107-0052
Tel: [phone]

Chelsio T5/T4 Unified Wire for Windows iii
Version History
Version
Revision Date
1.0.0
10/10/2011
1.0.1
10/18/2011
1.0.2
10/25/2011
1.0.3
10/31/2011
1.0.4
01/09/2012
1.0.5
01/18/2012
1.0.6
06/07/2012
1.0.7
07/11/2012
1.0.8
03/04/2013
1.0.9
03/21/2013
1.1.0
06/24/2013
1.1.1
07/19/2013
1.1.2
08/21/2013
1.1.3
09/04/2013

Chelsio T5/T4 Unified Wire for Windows iv
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 6
1. Introduction 7
1.1. Features 7
1.2. Hardware Requirements 7
1.3. Software Requirements 8
1.4. Package Contents 8
2. Hardware Installation 9
3. Software/Driver Installation 10
4. Firmware Update 11
5. Software/Driver Update 12
II. NDIS FUNCTION 13
1. Introduction 14
1.1. Hardware Requirements 14
1.2. Software Requirements 14
2. Software/Driver Configuration and Fine-tuning 15
2.1. Assigning IP address to the Chelsio Network Adapter 15
2.2. Advanced Configuration 15
2.3. Enabling SMB Direct 21
2.4. Verifying RDMA 21
2.5. Troubleshooting 21
III. APPENDIX 23
Chelsio End-User License Agreement (EULA) 24

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Windows 6
I.Chelsio Unified Wire

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Windows 7
1.Introduction
Thank you for choosing Chelsio T5/T4 Unified Wire adapters. These high speed, single chip,
single firmware cards provide enterprises and data centers with high performance solutions for
various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, ISCSI,
FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
The T4 adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified
Wire. The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and
Filtering.
Ideal for all data, storage and high performance clustering applications, the T5/T4 Adapters
enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets,
Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio T5/T4 adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
1.1. Features
Chelsio's T5/T4 Unified Wire package contains Chelsio's NDIS function driver. The
driver supports NIC and NDK functionalities.
1.2. Hardware Requirements
The Chelsio T5/T4 Unified Wire supports all x64 architectures supporting PCIE (x4, x8) slots.
AMD CPUs, 64-bit (x86_64/amd64)
Intel CPUs, 64-bit (x86_64)
The Chelsio Unified Wire supports 3.3v PCI bus only. Running an adapter on a
PCI x4 slot is not recommended as performance will be significantly reduced by
the limitations of PCI.
Note

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Windows 8
1.3. Software Requirements
Chelsio T5/T4 Unified Wire has been developed to run on Windows based platforms. To know
more about the complete list of versions supported by each driver/software, please refer to their
respective sections.
1.4. Package Contents
Chelsio T5/T4 Unified Wire driver package contains driver files and support documentation
(docs folder).

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Windows 9
2.Hardware Installation
1. Shutdown/power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIE x8
slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 40/10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Verify if the adapter was installed successfully. To do so, open Device Manager in Control
Panel.
13. Under Other devices section, Chelsio adapter should be listed as Ethernet Controller. If
the adapter is not listed, right-click on the system name or click on the Actions menu and
select Scan for hardware changes
For Chelsio adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions
Physical function 4: for all NIC functions of the card
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
14. Once the Unified Wire package is installed, open Device Manager again. Expand Network
adapters section and now Chelsio adapter should be listed.
Network device names for Chelsio’s physical ports are assigned using the
following convention: the port farthest from the motherboard will appear as the
first ethernet network interface. However, for T580-LP-CR,T580-SO-CR and
T420-BT adapters, the association of physical ethernet ports and their
corresponding network device names is opposite. For these adapters, the port
nearest to the motherboard will appear as the first network interface.
Note

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Windows 10
3.Software/Driver Installation
Chelsio Unified Wire can be installed using Windows Update or Chelsio’s Unified Wire
Installer. Alternatively, you can install or remove driver files in an offline Windows image using
procedures mentioned in the following links:
http://technet.microsoft.com/en-us/library/hh825070.aspx
http://blogs.technet.com/b/heyscriptingguy/archive/2012/09/27/use-the-powershell-dism-
cmdlets-to-manage-windows-8.aspx

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Windows 11
4.Firmware Update
The driver will auto-load the T5 and T4 firmwares if an update is required. The firmware version
can be verified using cxgbtool:
C:\Users\Administrator> cxgbtool nic0 version

Chapter I. Chelsio Unified Wire
Chelsio T5/T4 Unified Wire for Windows 12
5.Software/Driver Update
For any distribution specific problems, please check README and Release Notes included in
the release for possible workaround.
Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various
software/drivers. You can also subscribe to our newsletter for the latest software updates.

Chapter II. NDIS Function
Chelsio T5/T4 Unified Wire for Windows 13
II.NDIS Function

Chapter II. NDIS Function
Chelsio T5/T4 Unified Wire for Windows 14
1.Introduction
Chelsio’s T5/T4 series of Unified Wire Adapters provide extensive support for NIC operation,
including all stateless offload mechanisms for both IPv4 and IPv6 (IP, TCP and UDP checksum
offload, LSO - Large Send Offload aka TSO - TCP Segmentation Offload, Network Direct and
assist mechanisms for accelerating LRO - Large Receive Offload).
1.1. Hardware Requirements
1.1.1. Supported Adapters
The following are the currently shipping Chelsio Adapters that are compatible with Chelsio NDIS
driver:
T520-LL-CR
T520-SO-CR*
T580-SO-CR *
T520-CR
T540-CR
T580-LP-CR
T420-CR
T440-CR
T422-CR
T420-SO-CR*
T404-BT
T440-LP-CR
T420-BT
T420-LL-CR
T420-CX
* NDK Functionality not supported
1.2. Software Requirements
1.2.1. Windows Requirement
Currently Chelsio T5/T4 NDIS Function driver is available for Windows server 2012 R2.

Chapter II. NDIS Function
Chelsio T5/T4 Unified Wire for Windows 15
2.Software/Driver Configuration and Fine-tuning
2.1. Assigning IP address to the Chelsio Network Adapter
1. Double click on the Network Connections icon and choose the Chelsio card entry and
double click it.
2. Click on the Properties button from the Local Area Connection X Status.
3. Select “Internet Protocol (TCP/IP)” from the list and click on Properties button below it.
4. From the Internet Protocol (TCP/IP) Properties window, assign an IP Address (e.g.
192.169.1.10) and subnet mask (e.g. 255.255.255.0).
5. Click on Ok and close on the other window.
6. Check to see if you can ping to some other address on this subnet.
2.2. Advanced Configuration
The Chelsio Network driver provides advanced configuration options under the Device
Properties. In the Network Connections window, select the Chelsio Adapter’s Local Area
Connection interface (right-click -> properties). Click the Configure button.
2.2.1. VBD Driver Parameters
To see all tunable VBD driver parameters, open the Device Manager (Start > Control Panel ->
System & Security-> System -> Device Manager), click on the System Devices, double click on
the Chelsio T5/T4 40G/10G Bus Enumerator, and then click the Advanced tab.
Max VMQueues:
Description: Specifies the maximum number of virtual machine queues.
Value: 0-16
Default: 4
Nic Port x:
Description: Enables/Disables the Nic port x.
Value: Enabled | Disabled
Default: Enabled
Port x Speed & Duplex:
Description: Specifies the Port x operation speed and mode.
Value:
o100Mbs/Full Duplex
o1Gbps/Full Duplex
oAuto Detect
Default: Auto Detect

Chapter II. NDIS Function
Chelsio T5/T4 Unified Wire for Windows 16
iScsi Port x:
Description: Enables/Disables the iScsi port x.
Value: Enabled | Disabled
Default: Disabled
2.2.2. NDIS Miniport Driver Parameters
To see all tunable NDIS Miniport Driver parameters, open the Device Manager (Start > Control
Panel -> System & Security-> System -> Device Manager), click on the Network adapters,
double click on any Chelsio Network Adapter, and then click the Advanced tab.
Flow Control:
Description: Enable or disable the receipt or transmission of PAUSE frames. If Rx is
enabled, all incoming PAUSE frames will be honored. If Tx is enabled, the HW will send
PAUSE frames when its MAC Rx FIFO usage is beyond a high-watermark. If Disabled
option is selected, no PAUSE will be sent. When Rx MAC FIFO gets full, further ingress
packets will be dropped. If Rx & Tx Enabled is selected, both Rx and Tx options will be
enabled.
Value:
oDisabled
oRx & Tx Enabled
oRx Enabled
oTx Enabled
Default: Rx & Tx Enabled
IPv4 Checksum Offload:
Description: Allows the checksum to be computed by the adapter for IPV4 packets.
Value:
oDisabled
oRx & Tx Enabled
oRx Enabled
oTx Enabled
Default: Rx & Tx Enabled
Jumbo Packet (Maximum Transmission Unit):
Description: Specifies the Maximum Transmission Unit (MTU) value.
Value: 1500-9000
Default: 1500
This parameter is valid only for 1G port. User can specify any of the above 1G or
100M full duplex modes. If Auto-Detect is selected, the 1G port will do auto-
negotiation.
Note

Chapter II. NDIS Function
Chelsio T5/T4 Unified Wire for Windows 17
Large Send Offload V2 (IPv4):
Description: Allows for configuring Large Send Offload (LSO) using version 2 for the
IPv4 traffic.
Value: Enabled | Disabled
Default: Enabled
Large Send Offload V2 (IPv6):
Description: Allows for configuring Large Send Offload (LSO) using version 2 for the
IPv6 traffic.
Value: Enabled | Disabled
Default: Enabled
Locally Administered Address:
Description: Specifies a new MAC address for the port. If Not Present, the default MAC
from EEPROM is used. If specified, the new MAC overwrites the default MAC. This
specified MAC is persistent across machine reboot.
Value: Not Present | Any legal 6-byte MAC address
Default: Not Present
Maximum Number of RSS Queues:
Description: For changing the number of RSS Queues.
Value: 1-8
Default: 4
NetworkDirect Functionality:
Description: For enabling or disabling Network Direct Functionality.
Value: Enabled | Disabled
Default: Enabled
NetworkDirect Interrupt Moderation:
Description: This parameter controls the interrupts generated during NDK traffic.
Value: Enabled | Disabled
Default: Enabled
For more information, refer http://msdn.microsoft.com/en-
us/library/windows/hardware/ff556017%28v=vs.85%29.aspx
Receive Side Scaling (RSS):
Description: This parameter controls the RSS functions. If on, the Microsoft RSS function
is enabled. If off, the Chelsio RSS function is enabled.
Value: Enabled | Disabled
Default: Enabled

Chapter II. NDIS Function
Chelsio T5/T4 Unified Wire for Windows 18
Recv Segment Coalescing (IPv4):
Description: Enables or Disables the Receive Segment Coalescing feature for IPv4
traffic. This is a receiver packets aggregating feature which helps reduce the receive
host CPU load and improve throughput in a 10Gb network environment where CPU can
be the bottleneck.
Value: Enabled | Disabled
Default: Enabled
Recv Segment Coalescing (IPv6):
Description: Enables or Disables the Receive Segment Coalescing feature for IPv6
traffic. This is a receiver packets aggregating feature which helps reduce the receive
host CPU load and improve throughput in a 10Gb network environment where CPU can
be the bottleneck.
Value: Enabled | Disabled
Default: Enabled
RSS load balancing profile:
Description: This parameter controls the RSS load balancing profile.
Value: Closest Processor | Closest Processor Static | Conservative Scaling | NUMA
Scaling | NUMA Scaling Static
Default: Closest Processor
For more information, refer http://technet.microsoft.com/en-us/library/hh997036.aspx
Rx Ethernet Queue Size:
Description: This parameter is used to set different Rx Queue sizes. Driver ships with
reasonable defaults.
Value: 512-4096
Default: 3072
Rx Offload Queue Size:
Description: This parameter is used to set different Offload Rx Queue sizes. Driver ships
with reasonable defaults.
Value: 256-2048
Default: 512
This feature is not fully tested and available for experimental usage only.
Note

Chapter II. NDIS Function
Chelsio T5/T4 Unified Wire for Windows 19
TCP Checksum Offload (IPv4):
Description: This parameter enables or disables computation of TCP checksum by the
adapter for IPv4 packets.
Value:
oDisabled
oRx & Tx Enabled
oRx Enabled
oTx Enabled
Default: Rx & Tx Enabled
TCP Checksum Offload (IPv6):
Description: This parameter enables or disables computation of TCP checksum by the
adapter for IPv6 packets.
Value:
oDisabled
oRx & Tx Enabled
oRx Enabled
oTx Enabled
oDefault: Rx & Tx Enabled
TCP Offload:
Description: Controls the enable or disable of TCP Offload.
Range: Enabled | Disabled
Default: Disabled
Tx Ethernet Queue Size:
Description: This parameter is used to set different Tx Queue sizes. Driver ships with
reasonable defaults.
Value: 512-4096
Default: 1024
Tx Offload Queue Size:
Description: This parameter is used to set different Offload Tx Queue sizes. Driver ships
with reasonable defaults.
Value: 256-4096
Default: 512
This feature is not fully tested and available for experimental usage only.
Note
This feature is not fully tested. This feature is available for experimental usage
only.
Note

Chapter II. NDIS Function
Chelsio T5/T4 Unified Wire for Windows 20
UDP Checksum Offload (IPv4):
Description: This parameter enables or disables computation of UDP checksum by the
adapter for IPv4 packets.
Value:
oDisabled
oRx & Tx Enabled
oRx Enabled
oTx Enabled
Default: Rx & Tx Enabled
UDP Checksum Offload (IPv6):
Description: This parameter enables or disables computation of UDP checksum by the
adapter for IPv6 packets.
Value:
oDisabled
oRx & Tx Enabled
oRx Enabled
oTx Enabled
Default: Rx & Tx Enabled
Virtual Machine Queues:
Description: Enable or disable the virtual machine queue feature on the interface.
Value: Enabled | Disabled
Default: Enabled
VLAN Identifier:
Description: Specifies the VLAN ID to be inserted.
Value: 0-4094
Default: 0
VMQ LookAhead Split:
Description: Enable or disable the ability to split receive buffers into lookahead and post-
lookahead buffers.
Value: Enabled | Disabled
Default: Enabled
VMQ VLAN ID Filtering:
Description: Enable or disable the ability to filter network packets by using the VLAN
identifier in the media access control (MAC) header.
Value: Enabled | Disabled
Default: Enabled

Chapter II. NDIS Function
Chelsio T5/T4 Unified Wire for Windows 21
2.3. Enabling SMB Direct
SMB Direct is installed and enabled by default on Windows Server 2012 R2 systems. RDMA
functionality on Chelsio adapters will be enabled after installing the driver from the package.
You can use the following PowerShell command Get-NetAdapterRdma to get a list of RDMA
capable interfaces in your system.
The next step is to configure your network so that SMB server and client machines can
communicate with each other. Disable firewall for Chelsio adapters when using SMB.
2.4. Verifying RDMA
There are number of ways to verify if RDMA is working:
2.4.1. Verify active connections using PowerShell
Open PowerShell window and execute the command Get-SmbConnection
2.4.2. Use Event Viewer
You can use Event Viewer to view SMB connection events. To do this:
i. Open Event Viewer
ii. In the console tree, expand Event Viewer.
iii. Navigate to Application and Service Logs-> Microsoft->Windows->SMBClient
->Operational
2.5. Troubleshooting
If RDMA is not working in your system, please check for the following:
Operating system version should be Windows Server 2012 R2.
Verify network interface configurations on both sides.
Please ensure that the interfaces are reachable using ping command.
In the Event Viewer (on the client side), please ensure that the traffic running is not over
TCP.
At least one adapter should indicate “RDMA Capable” true to take advantage of
SMB RDMA capability.
Note