Chelsio Communications Chelsio T5 User manual

RHEL6.7 19pef3.12segmenUe mo

Chelsio T5/T4 Unified Wire for Linux ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to sa[email protected]
Support
For all support related questions please send email to [email protected]
Copyright © 2016.Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Fax: [phone]
Chelsio KK (Japan)
Regus Place Canada
Akasaka 7-3-37
Minato-ku, Tokyo,
Japan 107-0052
Tel: [phone]

Chelsio T5/T4 Unified Wire for Linux iii
Document History
Version
Revision Date
1.0.0
12/08/2011
1.0.1
01/09/2013
1.0.2
01/27/2013
1.0.3
03/26/2013
1.0.4
04/12/2013
1.0.5
06/20/2013
1.0.6
08/17/2013
1.0.7
10/22/2013
1.0.8
03/08/2013
1.0.9
05/15/2013
1.1.0
07/26/2013
1.1.1
08/14/2013
1.1.2
12/06/2013
1.1.3
12/19/2013
1.1.4
03/13/2014
1.1.5
05/02/2014
1.1.6
06/30/2014
1.1.7
10/22/2014
1.1.8
11/04/2014
1.1.9
02/05/2015
1.2.0
03/04/2015
1.2.1
03/25/2015
1.2.2
06/03/2015
1.2.3
08/05/2015
1.2.4
02/29/2016

Chelsio T5/T4 Unified Wire for Linux iv
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 12
1. Introduction 13
1.1. Features 13
1.2. Hardware Requirements 14
1.3. Software Requirements 14
1.4. Package Contents 15
2. Hardware Installation 18
3. Software/Driver Installation 21
3.1. Pre-requisites 22
3.2. Installing Chelsio Unified Wire from source 22
3.3. Installing Chelsio Unified Wire from RPM 33
3.4. Firmware update 35
4. Software/Driver Uninstallation 37
4.1. Uninstalling Chelsio Unified Wire from source 37
4.2. Uninstalling Chelsio Unified Wire from RPM 41
5. Configuring Chelsio Network Interfaces 43
5.1. Configuring 40G adapters 43
5.2. Configuring network-scripts 45
5.3. Creating network-scripts 45
5.4. Checking Link 46
6. Performance Tuning 47
7. Software/Driver Update 48
II. NETWORK (NIC/TOE) 49
1. Introduction 50
1.1. Hardware Requirements 50
1.2. Software Requirements 51
2. Software/Driver Loading 52
2.1. Loading in NIC mode (without full offload support) 52
2.2. Loading in TOE mode (with full offload support) 52
3. Software/Driver Unloading 53
3.1. Unloading the NIC driver 53
3.2. Unloading the TOE driver 53
4. Software/Driver Configuration and Fine-tuning 55
4.1. Instantiate Virtual Functions (SR-IOV) 55
4.2. Enabling Busy waiting 55
4.3. Performance Tuning 56
III. VIRTUAL FUNCTION NETWORK (VNIC) 62

Chelsio T5/T4 Unified Wire for Linux v
1. Introduction 63
1.1. Hardware Requirements 63
1.2. Software Requirements 64
2. Software/Driver Loading 65
2.1. Instantiate Virtual Functions 65
2.2. Loading the driver 65
3. Software/Driver Unloading 66
3.1. Unloading the driver 66
IV. IWARP (RDMA) 67
1. Introduction 68
1.1. Hardware Requirements 68
1.2. Software Requirements 68
2. Software/Driver Loading 70
2.1. Loading iWARP driver 70
3. Software/Driver Unloading 71
4. Software/Driver Configuration and Fine-tuning 72
4.1. Testing connectivity with ping and rping 72
4.2. Enabling various MPIs 73
4.3. Setting up NFS-RDMA 80
4.4. Performance Tuning 82
V. RDMA BLOCK DEVICE DRIVER (RBD) 83
1. Introduction 84
1.1. Hardware Requirements 84
1.2. Software Requirements 84
2. Software/Driver Loading 85
3. Software/Driver Unloading 86
4. Software/Driver Configuration and Fine-tuning 87
VI. WD-UDP 88
1. Introduction 89
1.1. Hardware Requirements 89
1.2. Software Requirements 89
2. Software/Driver Loading 91
3. Software/Driver Unloading 92
4. Software/Driver Configuration and Fine-tuning 93
4.1. Accelerating UDP Socket communications 93
VII. ISCSI PDU OFFLOAD TARGET 99
1. Introduction 100

Chelsio T5/T4 Unified Wire for Linux vi
1.1. Features 100
1.2. Hardware Requirements 101
1.3. Software Requirements 102
2. Software/Driver Loading 104
2.1. Latest iSCSI Software Stack Driver Software 104
3. Software/Driver Unloading 106
4. Software/Driver Configuration and Fine-tuning 107
4.1. Command Line Tools 107
4.2. iSCSI Configuration File 107
4.3. A Quick Start Guide for Target 108
4.4. The iSCSI Configuration File 110
4.5. Challenge-Handshake Authenticate Protocol (CHAP) 121
4.6. Target Access Control List (ACL) Configuration 123
4.7. Target Storage Device Configuration 125
4.8. Target Redirection Support 127
4.9. The command line interface tools “iscsictl” & “chisns” 128
4.10. Rules of Target Reload (i.e. “on the fly” changes) 133
4.11. System Wide Parameters 135
4.12. Performance Tuning 136
VIII. ISCSI PDU OFFLOAD INITIATOR 137
1. Introduction 138
1.1. Hardware Requirements 138
1.2. Software Requirements 139
2. Software/Driver Loading 140
3. Software/Driver Unloading 141
4. Software/Driver Configuration and Fine-tuning 142
4.1. Accelerating open-iSCSI Initiator 142
4.2. Auto login from cxgb4i initiator at OS bootup 144
IX. DATA CENTER BRIDGING (DCB) 146
1. Introduction 147
1.1. Hardware Requirements 147
1.2. Software Requirements 147
2. Software/Driver Loading 149
3. Software/Driver Unloading 150
4. Software/Driver Configuration and Fine-tuning 151
4.1. Configuring Cisco Nexus 5010 switch 151
4.2. Configuring the Brocade 8000 switch 154
5. Running NIC & iSCSI Traffic together with DCBx 156

Chelsio T5/T4 Unified Wire for Linux vii
X. FCOE PDU OFFLOAD TARGET 157
1. Introduction 158
1.1. Hardware Requirements 158
1.2. Software Requirements 158
2. Software/Driver Loading 159
3. Software/Driver Unloading 160
4. Software/Driver Configuration and Fine-tuning 161
4.1. Configuring Cisco Nexus 5010 switch 161
4.2. Collecting port information 164
4.3. Configuring LUNs on Target 167
4.4. Configuring Persistent Target 169
4.5. Verifying initiators connected to the target 170
4.6. Removing LUNs 171
4.7. Performance Tuning 171
XI. FCOE FULL OFFLOAD INITIATOR 174
1. Introduction 175
1.1. Hardware Requirements 175
1.2. Software Requirements 175
2. Software/Driver Loading 177
3. Software/Driver Unloading 178
4. Software/Driver Configuration and Fine-tuning 179
4.1. Configuring Cisco Nexus 5010 and Brocade switch 179
4.2. FCoE fabric discovery verification 179
4.3. Formatting the LUNs and Mounting the Filesystem 183
4.4. Creating Filesystem 184
4.5. Mounting the formatted LUN 185
XII. OFFLOAD BONDING DRIVER 186
1. Introduction 187
1.1. Hardware Requirements 187
1.2. Software Requirements 187
2. Software/Driver Loading 189
3. Software/Driver Unloading 190
4. Software/Driver Configuration and Fine-tuning 191
4.1. Offloading TCP traffic over a bonded interface 191
XIII. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) 192
1. Introduction 193
1.1. Hardware Requirements 193
1.2. Software Requirements 194

Chelsio T5/T4 Unified Wire for Linux viii
2. Software/Driver Loading 195
3. Software/Driver Unloading 196
4. Software/Driver Configuration and Fine-tuning 197
4.1. Offloading TCP traffic over a bonded interface 197
XIV. UDP SEGMENTATION OFFLOAD AND PACING 198
1. Introduction 199
1.1. Hardware Requirements 199
1.2. Software Requirements 200
2. Software/Driver Loading 201
3. Software/Driver Unloading 202
4. Software/Driver Configuration and Fine-tuning 203
4.1. Modifying the application 203
4.2. Configuring UDP Pacing 205
XV. OFFLOAD IPV6 DRIVER 207
1. Introduction 208
1.1. Hardware Requirements 208
1.2. Software Requirements 208
2. Software/Driver Loading 210
3. Software/Driver Unloading 211
3.1. Unloading the NIC driver 211
3.2. Unloading the TOE driver 211
XVI. BYPASS DRIVER 212
1. Introduction 213
1.1. Features 213
1.2. Hardware Requirements 214
1.3. Software Requirements 215
2. Software/Driver Loading 216
3. Software/Driver Unloading 217
4. Software/Driver Configuration and Fine-tuning 218
4.1. Starting ba server 218
4.2. Bypass API (CLI) 218
XVII. WD SNIFFING AND TRACING 224
1. Theory of Operation 225
1.1. Hardware Requirements 226
1.2. Software Requirements 227
2. Installation and Usage 228

Chelsio T5/T4 Unified Wire for Linux ix
2.1. Installing basic support 228
2.2. Using Sniffer (wd_sniffer) 228
2.3. Using Tracer (wd_tcpdump_trace) 228
XVIII.CLASSIFICATION AND FILTERING 230
1. Introduction 231
1.1. Hardware Requirements 231
1.2. Software Requirements 232
2. Usage 233
2.1. Configuration 233
2.2. Creating Filter Rules 234
2.3. Listing Filter Rules 235
2.4. Removing Filter Rules 235
2.5. Layer 3 example 236
2.6. Layer 2 example 239
3. Hash/DDR Filters 243
3.1. Creating Filter Rules 243
3.2. Listing Filter Rules 244
3.3. Removing Filter Rules 245
3.4. Swap MAC feature 245
3.5. Hit Counters 246
XIX. TRAFFIC MANAGEMENT 248
1. Introduction 249
1.1. Hardware Requirements 249
1.2. Software Requirements 250
2. Software/Driver Loading 251
3. Software/Driver Unloading 252
4. Software/Driver Configuration and Fine-tuning 253
4.1. Traffic Management Rules 253
4.2. Configuring Traffic Management 255
5. Usage 258
5.1. Non-Offloaded Connections 258
5.2. Offloaded Connections 258
5.3. Offloaded Connections with Modified Application 259
XX. UNIFIED WIRE MANAGER (UM) 260
1. Introduction 261
1.1. Features 261
1.2. Reference Architecture 262
1.3. Unified Wire Manager Components 262

Chelsio T5/T4 Unified Wire for Linux x
1.4. Authentication and encryption 263
2. Hardware and Software 264
2.1. Supported Adapters 264
2.2. Platform/Component Matrix 265
2.3. Platform/Driver Matrix 265
3. Installing Unified Wire Manager 266
4. Verifying UM components status 267
4.1. Verifying Management Agent 267
4.2. Verifying Management Client 268
4.3. Verifying Management Station 268
5. Management Agent 269
5.1. Communication 269
5.2. Configuration 269
5.3. Service configuration 269
5.4. Firewall 270
6. CLI client 271
6.1. CLI Help system 271
6.2. Client conflict resolution 271
7. Web GUI client 272
7.1. Management Station 272
7.2. Accessing Web Management Interface 273
7.3. Layout and Navigation 276
7.4. Home page 277
7.5. System page 290
7.6. Network page 302
7.7. Storage Page 331
7.8. Hardware Features 364
8. Uninstalling Unified Wire Manager 380
8.1. Uninstalling Management Agent 380
8.2. Uninstalling Management Client 380
8.3. Uninstalling Management Station 381
XXI. UNIFIED BOOT 382
1. Introduction 383
1.1. Hardware Requirements 383
1.2. Software Requirements 385
2. Flashing firmware and option ROM 386
2.1. Preparing USB flash drive 386
2.2. Legacy 387
2.3. uEFI 390
3. Configuring PXE Server 395