Chelsio Communications Chelsio T5 User manual

pef3.12segmenUe mo

Chelsio T5/T4 Unified Wire For Linux ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [email protected]
Copyright © 2015.Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
370 San Aleso Ave.
Suite 100
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Fax: [phone]
Chelsio KK (Japan)
SHIMA Akasaka Bldg.
Minato-ku, Tokyo
Japan 107-0052
Tel: [phone]

Chelsio T5/T4 Unified Wire For Linux iii
Document History
Version
Revision Date
1.0.0
12/08/2011
1.0.1
01/09/2013
1.0.2
01/27/2013
1.0.3
03/26/2013
1.0.4
04/12/2013
1.0.5
06/20/2013
1.0.6
08/17/2013
1.0.7
10/22/2013
1.0.8
03/08/2013
1.0.9
05/15/2013
1.1.0
07/26/2013
1.1.1
08/14/2013
1.1.2
12/06/2013
1.1.3
12/19/2013
1.1.4
03/13/2014
1.1.5
05/02/2014
1.1.6
06/30/2014
1.1.7
10/22/2014
1.1.8
11/04/2014
1.1.9
02/05/2015

Chelsio T5/T4 Unified Wire For Linux iv
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 12
1. Introduction 13
1.1. Features 13
1.2. Hardware Requirements 14
1.3. Software Requirements 14
1.4. Package Contents 14
2. Hardware Installation 17
3. Software/Driver Installation 20
3.1. Pre-requisites 21
3.2. Installing Chelsio Unified Wire from source 21
3.3. Installing Chelsio Unified Wire from RPM 32
3.4. Firmware update 34
4. Software/Driver Uninstallation 35
4.1. Uninstalling Chelsio Unified Wire from source 35
4.2. Uninstalling Chelsio Unified Wire from RPM 39
5. Configuring Chelsio Network Interfaces 41
5.1. Configuring 40G adapters 41
5.2. Configuring network-scripts 43
5.3. Creating network-scripts 43
5.4. Checking Link 44
6. Software/Driver Update 45
II. NETWORK (NIC/TOE) 46
1. Introduction 47
1.1. Hardware Requirements 47
1.2. Software Requirements 48
2. Software/Driver Loading 49
2.1. Loading in NIC mode (without full offload support) 49
2.2. Loading in TOE mode (with full offload support) 49
3. Software/Driver Unloading 50
3.1. Unloading the NIC driver 50
3.2. Unloading the TOE driver 50
4. Software/Driver Configuration and Fine-tuning 51
4.1. Instantiate Virtual Functions (SR-IOV) 51
4.2. Performance tuning 51
III. VIRTUAL FUNCTION NETWORK (VNIC) 58
1. Introduction 59
1.1. Hardware Requirements 59

Chelsio T5/T4 Unified Wire For Linux v
1.2. Software Requirements 60
2. Software/Driver Loading 61
2.1. Loading the driver 61
3. Software/Driver Unloading 62
3.1. Unloading the driver 62
4. Software/Driver Configuration and Fine-tuning 63
4.1. Instantiate Virtual Functions 63
IV. IWARP (RDMA) 64
1. Introduction 65
1.1. Hardware Requirements 65
1.2. Software Requirements 65
2. Software/Driver Loading 67
2.1. Compiling and Loading iWARP driver 67
3. Software/Driver Unloading 68
4. Software/Driver Configuration and Fine-tuning 69
4.1. Testing connectivity with ping and rping 69
4.2. Enabling various MPIs 70
4.3. Setting up NFS-RDMA 78
4.4. Performance tuning 79
V. WD-UDP 80
1. Introduction 81
1.1. Hardware Requirements 81
1.2. Software Requirements 81
2. Software/Driver Loading 83
3. Software/Driver Unloading 84
4. Software/Driver Configuration and Fine-tuning 85
4.1. Accelerating UDP Socket communications 85
VI. WD-TOE 91
1. Introduction 92
1.1. Hardware Requirements 92
1.2. Software Requirements 92
2. Software/Driver Loading 94
3. Software/Driver Unloading 95
4. Software/Driver Configuration and Fine-tuning 96
4.1. Running the application 96
VII. ISCSI PDU OFFLOAD TARGET 97
1. Introduction 98

Chelsio T5/T4 Unified Wire For Linux vi
1.1. Features 98
1.2. Hardware Requirements 99
1.3. Software Requirements 101
2. Software/Driver Loading 103
2.1. Latest iSCSI Software Stack Driver Software 103
2.2. Generating single RPM for T3 and T4 adapters 105
2.3. Obtaining the iSCSI Software License 106
3. Software/Driver Unloading 108
4. Software/Driver Configuration and Fine-tuning 109
4.1. Command Line Tools 109
4.2. iSCSI Configuration File 109
4.3. A Quick Start Guide for Target 110
4.4. The iSCSI Configuration File 113
4.5. Challenge-Handshake Authenticate Protocol (CHAP) 122
4.6. Target Access Control List (ACL) Configuration 124
4.7. Target Storage Device Configuration 125
4.8. Target Redirection Support 128
4.9. The command line interface tools “iscsictl” & “chisns” 129
4.10. Rules of Target Reload (i.e. “on the fly” changes) 134
4.11. System Wide Parameters 135
4.12. Performance tuning 136
VIII. ISCSI PDU OFFLOAD INITIATOR 138
1. Introduction 139
1.1. Hardware Requirements 139
1.2. Software Requirements 140
2. Software/Driver Loading 141
3. Software/Driver Unloading 143
4. Software/Driver Configuration and Fine-tuning 144
4.1. Accelerating open-iSCSI Initiator 144
4.2. Auto login from cxgb4i initiator at OS bootup 147
IX. DATA CENTER BRIDGING (DCB) 148
1. Introduction 149
1.1. Hardware Requirements 149
1.2. Software Requirements 149
2. Software/Driver Loading 150
3. Software/Driver Unloading 151
4. Software/Driver Configuration and Fine-tuning 152
4.1. Configuring Cisco Nexus 5010 switch 152
4.2. Configuring the Brocade 8000 switch 155

Chelsio T5/T4 Unified Wire For Linux vii
X. FCOE PDU OFFLOAD TARGET 157
1. Introduction 158
1.1. Hardware Requirements 158
1.2. Software Requirements 158
2. Software/Driver Loading 161
3. Software/Driver Configuration and Fine-tuning 162
3.1. Configuring Cisco Nexus 5010 switch 162
3.2. Collecting port information 165
3.3. Configuring LUNs on Target 169
3.4. Configuring Persistent Target 171
3.5. Verifying initiators connected to the target 172
3.6. Removing LUNs 173
3.7. Performance tuning 173
4. Software/Driver Unloading 176
XI. FCOE FULL OFFLOAD INITIATOR 177
1. Introduction 178
1.1. Hardware Requirements 178
1.2. Software Requirements 178
2. Software/Driver Loading 179
3. Software/Driver Unloading 180
4. Software/Driver Configuration and Fine-tuning 181
4.1. Configuring Cisco Nexus 5010 and Brocade switch 181
4.2. FCoE fabric discovery verification 181
4.3. Formatting the LUNs and Mounting the Filesystem 185
4.4. Creating Filesystem 187
4.5. Mounting the formatted LUN 188
XII. OFFLOAD BONDING DRIVER 189
1. Introduction 190
1.1. Hardware Requirements 190
1.2. Software Requirements 190
2. Software/Driver Loading 192
3. Software/Driver Unloading 193
4. Software/Driver Configuration and Fine-tuning 194
4.1. Offloading TCP traffic over a bonded interface 194
XIII. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) 195
1. Introduction 196
1.1. Hardware Requirements 196
1.2. Software Requirements 197

Chelsio T5/T4 Unified Wire For Linux viii
2. Software/Driver Loading 198
3. Software/Driver Unloading 199
4. Software/Driver Configuration and Fine-tuning 200
4.1. Offloading TCP traffic over a bonded interface 200
4.2. Network Device Configuration 201
XIV. UDP SEGMENTATION OFFLOAD AND PACING 202
1. Introduction 203
1.1. Hardware Requirements 203
1.2. Software Requirements 204
2. Software/Driver Loading 205
3. Software/Driver Unloading 206
4. Software/Driver Configuration and Fine-tuning 207
4.1. Modifying the application 207
4.2. Configuring UDP Pacing 209
XV. OFFLOAD IPV6 DRIVER 211
1. Introduction 212
1.1. Hardware Requirements 212
1.2. Software Requirements 212
2. Software/Driver Loading 214
3. Software/Driver Unloading 215
3.1. Unloading the NIC driver 215
3.2. Unloading the TOE driver 215
XVI. BYPASS DRIVER 216
1. Introduction 217
1.1. Features 217
1.2. Hardware Requirements 218
1.3. Software Requirements 219
2. Software/Driver Loading 220
3. Software/Driver Unloading 221
4. Software/Driver Configuration and Fine-tuning 222
4.1. Starting ba server 222
4.2. Bypass API (CLI) 222
XVII. WD SNIFFING AND TRACING 228
1. Theory of Operation 229
1.1. Hardware Requirements 230
1.2. Software Requirements 231
2. Installation and Usage 232

Chelsio T5/T4 Unified Wire For Linux ix
2.1. Installing basic support 232
2.2. Using Sniffer (wd_sniffer) 232
2.3. Using Tracer (wd_tcpdump_trace) 232
XVIII.CLASSIFICATION AND FILTERING 234
1. Introduction 235
1.1. Hardware Requirements 235
1.2. Software Requirements 236
2. Usage 237
2.1. Configuration 237
2.2. Creating Filter Rules 237
2.3. Listing Filter Rules 239
2.4. Removing Filter Rules 239
2.5. Layer 3 example 240
2.6. Layer 2 example 243
3. Hash/DDR Filters 247
3.1. Creating Filter Rules 247
3.2. Listing Filter Rules 248
3.3. Removing Filter Rules 249
3.4. Swap MAC feature 249
3.5. Hit Counters 249
XIX. TRAFFIC MANAGEMENT 252
1. Introduction 253
1.1. Hardware Requirements 253
1.2. Software Requirements 254
2. Software/Driver Loading 255
3. Software/Driver Unloading 256
4. Software/Driver Configuration and Fine-tuning 257
4.1. Traffic Management Rules 257
4.2. Configuring Traffic Management 259
5. Usage 262
5.1. Non-Offloaded Connections 262
5.2. Offloaded Connections 262
5.3. Offloaded Connections with Modified Application 263
XX. UNIFIED WIRE MANAGER (UM) 264
1. Introduction 265
1.1. Features 265
1.2. Reference Architecture 266
1.3. Unified Wire Manager Components 266

Chelsio T5/T4 Unified Wire For Linux x
1.4. Authentication and encryption 267
2. Hardware and Software 268
2.1. Supported Adapters 268
2.2. Platform/Component Matrix 269
2.3. Platform/Driver Matrix 269
3. Installing Unified Wire Manager 270
4. Verifying UM components status 271
4.1. Verifying Management Agent 271
4.2. Verifying Management Client 272
4.3. Verifying Management Station 272
5. Management Agent 273
5.1. Communication 273
5.2. Configuration 273
5.3. Service configuration 273
5.4. Firewall 274
6. CLI client 275
6.1. CLI Help system 275
6.2. Client conflict resolution 275
7. Web GUI client 276
7.1. Management Station 276
7.2. Accessing Web Management Interface 277
7.3. Layout and Navigation 280
7.4. Home page 281
7.5. System page 294
7.6. Network page 306
7.7. Storage Page 335
7.8. Hardware Features 368
8. Uninstalling Unified Wire Manager 384
8.1. Uninstalling Management Agent 384
8.2. Uninstalling Management Client 384
8.3. Uninstalling Management Station 385
XXI. UNIFIED BOOT 386
1. Introduction 387
1.1. Hardware Requirements 387
1.2. Software Requirements 388
2. Flashing firmware and option ROM 389
2.1. Using Flash Utility 389
2.2. Using cxgbtool 393
3. Configuring PXE Server 394
4. PXE boot process 395