Chelsio Communications Terminator 5 User manual

FPS

Chelsio T5 Unified Wire for Windows ii
This document and related products are distributed under licenses restricting their use, copying, distribution,
and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO COMMUNICATIONS,
INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [e-mail]
Copyright © 2016. Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Chelsio KK (Japan)
Yamato Building 8F,
5-27-3 Sendagaya,
Shibuya-ku,
Tokyo 151-0051,
Japan

Chelsio T5 Unified Wire for Windows iii
Document History
Version
Revision Date
1.3.6
11/19/2015
1.3.7
12/11/2015
1.3.8
01/28/2016
1.3.9
03/04/2016
1.4.0
04/21/2016
1.4.1
05/13/2016
1.4.2
06/27/2016
1.4.3
07/25/2016
1.4.4
07/29/2016
1.4.5
08/16/2016
1.4.6
08/26/2016
1.4.7
10/14/2016

Chelsio T5 Unified Wire for Windows iv
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 7
1. Introduction 8
1.1. Features 8
1.2. Hardware Requirements 9
1.3. Software Requirements 9
1.4. Package Contents 9
2. Hardware Installation 10
3. Software/Driver Installation 11
3.1. Prerequisites 11
3.2. Unified Wire Installer 12
3.3. Nano Server Installer 21
3.4. Zip Package 22
4. Configuring Chelsio Network Interfaces 26
4.1. Configuring 40G Adapters 26
4.2. Assigning IP address 29
5. Mass Deployment (Installer) 31
5.1. Pre-deployment Configuration 31
5.2. Flashing Option ROM 33
5.3. Erasing Option ROM 41
5.4. Configuring using UM CLI 42
6. cxgbtool/cxgbnano help 48
7. Firmware Update 93
8. Software/Driver Uninstallation 94
8.1. Installer 94
8.2. Zip Package 96
9. Software/Driver Update 97
II. NDIS FUNCTION 98
1. Introduction 99
1.1. Hardware Requirements 99
1.2. Software Requirements 99
2. Software/Driver Configuration and Fine-tuning 100
2.1. Advanced Configuration 100
2.2. NVGRE Offload 107
2.3. VXLAN Task Offload 107
2.4. Virtual RSS (vRSS) 107
III. IWARP (ND) 109
1. Introduction 110
1.1. Hardware Requirements 110

Chelsio T5 Unified Wire for Windows v
1.2. Software Requirements 110
2. Software/Driver Configuration and Fine-tuning 111
2.1. Registering iWARP(ND) driver 111
IV. SMB DIRECT 112
3. Introduction 113
3.1. Hardware Requirements 113
3.2. Software Requirements 113
4. Software/Driver Configuration and Fine-tuning 114
4.1. Enabling SMB Direct 114
4.2. Verifying RDMA 114
4.3. RDMA/NVGRE concurrent (Mode 2) 115
4.4. Troubleshooting 116
V. NDIS SR-IOV 117
1. Introduction 118
1.1. Hardware Requirements 118
1.2. Software Requirements 118
2. Software/Driver Configuration and Fine-tuning 119
2.1. Enabling SR-IOV 119
2.2. vSwitch Configuration 122
2.3. Guest (VM) Configuration 122
VI. DATA CENTER BRIDGING (DCB) 123
1. Introduction 124
1.1. Hardware Requirements 124
1.2. Software Requirements 124
2. Software/Driver Configuration and Fine-tuning 125
2.1. Installing DCB 125
2.2. Quality of Service (QoS) 125
2.3. Configuring Dell Force10 Switch 126
VII. UNIFIED WIRE MANAGER (UM) 129
1. Introduction 130
1.1. Features 130
1.2. Reference Architecture 131
1.3. Unified Wire Manager Components 131
1.4. Authentication and encryption 132
2. Hardware and Software 133
2.1. Supported Adapters 133
2.2. Platform/Component Matrix 133
2.3. Platform/Driver Matrix 133

Chelsio T5 Unified Wire for Windows vi
3. Installation (Zip Package) 134
4. Configuration 139
4.1. Configuring Management Station 139
5. Verifying UM components status 141
5.1. Verifying Management Agent 141
5.2. Verifying Management Client 142
5.3. Verifying Management Station 142
6. Management Agent 143
6.1. Communication 143
6.2. Service configuration 143
6.3. Firewall 143
7. CLI client 144
7.1. CLI Help system 144
7.2. Client conflict resolution 144
8. Web GUI client 145
8.1. Management Station 145
8.2. Accessing Web Management Interface 146
8.3. Layout and Navigation 148
8.4. Home page 149
8.5. System page 159
8.6. Network page 163
8.7. Storage 178
8.8. Hardware Features 202
9. Uninstallation 214
VIII. UM FOR WIN PE 218
1. Introduction 219
1.1. Hardware Requirements 219
1.2. Software Requirements 220
1.3. Examples 220
IX. APPENDIX 224
Chelsio End-User License Agreement (EULA) 225

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 7
I.Chelsio Unified Wire

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 8
1.Introduction
Thank you for choosing Chelsio T5 Unified Wire adapters. These high speed, single chip,
single firmware cards provide enterprises and data centers with high performance solutions for
various Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a completeUnifiedWire solution comprising NIC, TOE, iWARPRDMA, iSCSI, FCoE
and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
Ideal for all data, storage and high performance clustering applications, the T5 adapters
enable a unified fabric over a single wire by simultaneously running all unmodified IP sockets,
Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio T5 adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
1.1. Features
Chelsio Unified Wire for Windows is an easy to use utility developed to provide installation of 64-
bit Windows based drivers and tools for Chelsio's T5 Unified Wire adapters.
It consists of the following components:
NDIS Function driver (NIC)
NVGRE Offload
VXLAN Task Offload
SMB Direct
RDMA/NVGRE concurrent (Mode 2)
iWARP (ND)
NDIS SR-IOV
Virtual RSS (vRSS)
Data Center Bridging (DCB)
Unified Wire Manager (UM)
UM for Win PE

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 9
1.2. Hardware Requirements
The Chelsio T5 Unified Wire supports all x64 architectures supporting PCIE (x4, x8) slots.
AMD CPUs, 64-bit (x86_64/amd64)
Intel CPUs, 64-bit (x86_64)
1.3. Software Requirements
The Chelsio T5 Unified Wire software has been developed to run on Windows based platforms.
To know more about the complete list of versions supported by each driver/software, please refer
their respective sections.
1.4. Package Contents
Unified Wire Installer
Chelsio T5 Unified Wire comes with an interactive installer and support documentation. The
documentation, consisting of README, Release Notes and User’s Guide (this document), can
be found in the <system_drive>\ChelsioUwire\docs\ directory after installing Chelsio UnifiedWire.
Zip Package
The zip package contains driver files, UM installer, UM for Win PE application, firmware binaries,
adapter configuration binaries and support documentation (docs folder).
Nano Server Installer
The Nano Server Installer package contains driver files, firmware binaries, adapter configuration
binaries and support documentation (docs folder)
The Chelsio Unified Wire supports 3.3v PCI bus only. Running an adapter on a
PCI x4 slot is not recommended as performance will be significantly reduced by
the limitations of PCI.
Note
T4 family of adapters are not supported on Windows 2016 Server & 10 AU Client
versions.
Note
Drivers are WHQL certified.
Note

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 10
2.Hardware Installation
i. Shutdown/power off your system.
ii. Power off all remaining peripherals attached to your system.
iii. Unpack the Chelsio adapter and place it on an anti-static surface.
iv. Remove the system case cover according to the system manufacturer’s instructions.
v. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
vi. For maximum performance, it is highly recommended to install the adapter into a PCIE x8
slot.
vii. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It may
be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
viii.Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
ix. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
x. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
xi. Power on your system.
xii. Verify if the adapter was installed successfully. To do so, open Device Manager in Control
Panel.
xiii.Under Other devices section, Chelsio adapter should be listed as Ethernet Controller. If the
adapter is not listed, right-click on the system name or click on the Actions menu and select
Scan for hardware changes
For Chelsio adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions
Physical function 4: for all NIC functions of the card
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
xiv.Once the Unified Wire package is installed, open Device Manager again. Expand Network
adapters section and now Chelsio adapter should be listed.
Network device names for Chelsio’s physical ports are assigned using the
following convention: the port farthest from the motherboard will appear as the
first Ethernet interface. However, for T5 40G adapters, the association of physical
Ethernet ports and their corresponding network device names is opposite. For
these adapters, the port nearest to the motherboard will appear as the first
network interface.
Note

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 11
3.Software/Driver Installation
Chelsio Unified Wire can be installed using the Installer or the zip package. Refer the relevant
section below depending on the method of installation selected.
3.1. Prerequisites
Please ensure that the following requirements are met, before proceeding with the installation:
Unified Wire Installer
Download and install the latest Microsoft Visual C++ Redistributable Packages for Visual Studio
(x86 and x64).
NDIS SR-IOV
Enable SR-IOV in the machine.
Unified Wire Manager (Installer & Zip Package)
The Unified Wire installer has been designed to install Unified Wire Manager (UM) along with
driver components by default, hence no separate installation is required. In case of zip package,
you will have to manually run the UM installer located in ChelsioUwire-x.x.x.xx/UM folder (See
Unified Wire Manager chapter for for more information). Based on the Windows version running
on the system, the three UM components, i.e. Management Agent, Client and Station will be
installed. If an older version of UM exists, the installer will upgrade it to the version provided in
the package.
Management Agent
If you wish to install Management Agent, please make sure that Microsoft .Net Framework 3.5
is installed before proceeding with the installation.
Management Station
If you wish to install Management Station, please make sure that the following requirements are
met before proceeding with the installation:
i. Install Python 2.6.6 (32-bit). (Download from here)
ii. Ensure that the path to python binary (typically C:\Python26), is added to PATH system
variable.
iii. Install Apache HTTP Server 2.2 with SSL. (Download from here)
iv. If the Apache Server is running, it should be stopped before starting the installation process.

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 12
3.2. Unified Wire Installer
Chelsio Unified Wire Installer provides two methods of installation: GUI or CLI mode. GUI mode
provides an interactive GUI installer with customizable options. Whereas, CLI mode enables
unattended installation of Chelsio drivers and software thereby relieving the user from monitoring
the installation process and providing input via dialog boxes.
Enabling Test Signing
In case the drivers in the package are not WHQL certified, follow the steps mentioned below to
enable test signing on your system:
i. Goto Start->Run command option, enter "cmd" and press OK. This will open the command
prompt utility.
ii. Run the following command:
C:\Users\Administrator> bcdedit /set testsigning on
iii. Reboot the machine for the changes to take effect.
GUI mode (Installer)
i. Run the ChelsioUwire-x.x.x.xx.exe installer application.
ii. Click Next for the Chelsio End User License Agreement Window.
Figure 1 - Unified Wire installer welcome window

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 13
iii. Select the radio button I accept the terms of the license agreement and click Next.
Figure 2 - Chelsio EULA window
iv. Now, either select Complete for complete package installation or else select Custom radio
button to customize the installation.
Figure 3 - Select setup (installation) type

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 14
Figure 4 - Customizing the installation
v. Click Install to start the installation.
Figure 5 - Start installation

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 15
vi. Selected driver components will now be installed.
Figure 6 - Installation in progress
vii. After successful installation of driver(s), the Unified Wire Manager installer will be invoked.
Click Next for the Chelsio End User License Agreement Window.
Figure 7 - UM installer welcome window

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 16
viii.Select the radio button I accept the terms of the license agreement and click Next.
Figure 8 - Chelsio EULA window
ix. The next window will display the prerequisites for various UM components. Ensure that they
are met before proceeding. Click Next.
Figure 9 - UM prerequisites window
If prerequisites mentioned are not met, installation of UM will fail. Driver installation
will however continue.
Note

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 17
x. Now, either select Complete for complete package installation or else select Custom radio
button to customize the installation. The Custom option provides the option to change the
installation path and which UM components to install. Click Next.
Figure 10 - Select setup (installation) type
If Management Station is selected, please make sure that all related prerequisites
are met before proceeding (See Prerequisites) or else the component will be skipped
during installation.
Note

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 18
Figure 11 - Changing UM installation path
Figure 12 - Selecting UM components to install

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 19
xi. Click Install to start the installation. Unified Wire Manager will now be installed with the
selected options.
Figure 13 - Starting UM installation
xii. Click Finish to exit from the UM Installer.
Figure 14 - Finishing UM installation

Chapter I. Chelsio Unified Wire
Chelsio T5 Unified Wire for Windows 20
xiii.Click Finish to exit from the Unified Wire Installer.
Figure 15 - Finishing Unified Wire installation
CLI mode (Silent Installation)
To install drivers/software using this feature, open command prompt and execute the following
command:
C:\Users\Administrator>ChelsioUwire-x.x.x.xx.exe –in <driver(s)>
E.g.:
C:\Users\Administrator>ChelsioUwire-x.x.x.xx.exe –in all
The above command will install all the drivers and UM Agent.
To add a driver to an existing list of already installed drivers, use the following command:
C:\Users\Administrator>ChelsioUwire-x.x.x.x.exe –add <driver(s)>