Chelsio Communications Terminator Series User manual

Chelsio Unified Wire for Linux i

Chelsio Unified Wire for Linux ii
This document and related products are distributed under licenses restricting their use, copying, distribution,
and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third-party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO COMMUNICATIONS,
INC.
Sales
For all sales inquiries please send email to [e-mail]
Support
For all support related questions please send email to [e-mail]
Copyright © 2018. Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Chelsio KK (Japan)
Yamato Building 8F,
5-27-3 Sendagaya,
Shibuya-ku,
Tokyo 151-0051,
Japan

Chelsio Unified Wire for Linux iii
Document History
Version
Revision Date
1.0.0
12/08/2011
1.0.1
01/09/2013
1.0.2
01/27/2013
1.0.3
03/26/2013
1.0.4
04/12/2013
1.0.5
06/20/2013
1.0.6
08/17/2013
1.0.7
10/22/2013
1.0.8
03/08/2013
1.0.9
05/15/2013
1.1.0
07/26/2013
1.1.1
08/14/2013
1.1.2
12/06/2013
1.1.3
12/19/2013
1.1.4
03/13/2014
1.1.5
05/02/2014
1.1.6
06/30/2014
1.1.7
10/22/2014
1.1.8
11/04/2014
1.1.9
02/05/2015
1.2.0
03/04/2015
1.2.1
03/25/2015
1.2.2
06/03/2015
1.2.3
08/05/2015
1.2.4
02/29/2016
1.2.5
04/27/2016
1.2.6
08/25/2016
1.2.7
10/07/2016
1.2.8
10/18/2016
1.2.9
11/11/2016
1.3.0
12/05/2016
1.3.1
12/30/2016
1.3.2
01/30/2017
1.3.3
02/27/2017
1.3.4
05/11/2017
1.3.5
09/05/2017

Chelsio Unified Wire for Linux iv
1.3.6
09/29/2017
1.3.7
12/29/2017
1.3.8
02/28/2018
1.3.9
03/30/2018
1.4.0
04/18/2018
1.4.1
07/05/2018
1.4.2
07/18/2018
1.4.3
10/01/2018

Chelsio Unified Wire for Linux v
TABLE OF CONTENTS
I. CHELSIO UNIFIED WIRE 15
Introduction 16
1.1. Features 16
1.2. Hardware Requirements 17
1.3. Software Requirements 17
1.4. Package Contents 17
Hardware Installation 21
Software/Driver Installation 23
3.1. Pre-requisites 23
3.2. Enabling RDMA on ARM Platforms 24
3.3. Mounting debugfs 24
3.4. Allowing unsupported modules on SLES 25
3.5. Installing Chelsio Unified Wire from source 25
3.6. Installing Chelsio Unified Wire from RPM 32
3.7. Firmware Update 36
3.8. Removing Drivers from initramfs 36
Configuring Chelsio Network Interfaces 37
4.1. Configuring Adapters 37
4.2. Configuring network-scripts 41
4.3. Creating network-scripts 41
4.4. Checking Link 42
Performance Tuning 43
5.1. Generic 43
5.2. Throughput 43
5.3. Latency 43
Software/Driver Update 45
Software/Driver Uninstallation 46
7.1. Uninstalling Chelsio Unified Wire from source 46
7.2. Uninstalling Chelsio Unified Wire from RPM 50
II. NETWORK (NIC/TOE) 51
Introduction 52
1.1. Hardware Requirements 52
1.2. Software Requirements 53
Software/Driver Installation 54
Software/Driver Loading 55
3.1. Loading in NIC mode (without full offload support) 55
3.2. Loading in TOE mode (with full offload support) 55
Software/Driver Configuration 56
4.1. Enabling TCP Offload 56

Chelsio Unified Wire for Linux vi
4.2. Enabling Busy waiting 56
4.3. Precision Time Protocol (PTP) 57
4.4. VXLAN Offload 59
4.5. Performance Tuning 62
Software/Driver Unloading 68
5.1. Unloading the NIC Driver 68
5.2. Unloading the TOE Driver 68
III. VIRTUAL FUNCTION NETWORK (VNIC) 70
Introduction 71
1.1. Hardware Requirements 71
1.2. Software Requirements 72
Software/Driver Installation 73
2.1. Pre-requisites 73
2.2. Installation 73
Software/Driver Loading 74
3.1. Instantiate Virtual Functions (SR-IOV) 74
3.2. Loading the Driver 74
Software/Driver Configuration and Fine-tuning 76
4.1. VF Rate Limiting 76
4.2. Bonding 77
4.3. High Capacity VF Configuration 78
Software/Driver Unloading 81
5.1. Unloading the Driver 81
IV.IWARP (RDMA) 82
Introduction 83
1.1. Hardware Requirements 83
1.2. Software Requirements 84
Software/Driver Installation 85
2.1. Pre-requisites 85
2.2. Installation 85
Software/Driver Loading 86
3.1. Loading iWARP Driver 86
Software/Driver Configuration and Fine-tuning 87
4.1. Testing connectivity with ping and rping 87
4.2. Enabling various MPIs 88
4.3. Setting up NFS-RDMA 96
4.4. Performance Tuning 97
Software/Driver Unloading 98
V. ISER 99

Chelsio Unified Wire for Linux vii
Introduction 100
1.1. Hardware Requirements 100
1.2. Software Requirements 100
Kernel Installation 101
Software/Driver Installation 102
3.1. Pre-requisites 102
3.2. Installation 102
Software/Driver Loading 103
Software/Driver Configuration and Fine-tuning 104
5.1. HMA 105
5.2. Performance Tuning 105
Software/Driver Unloading 106
VI. WD-UDP 107
Introduction 108
1.1. Hardware Requirements 108
1.2. Software Requirements 108
Software/Driver Installation 110
Software/Driver Loading 111
Software/Driver Configuration and Fine-tuning 112
4.1. Accelerating UDP Socket Communications 112
Software/Driver Unloading 118
VII. WD-TOE 119
Introduction 120
1.1. Hardware Requirements 120
1.2. Software Requirements 120
Software/Driver Installation 121
2.1. Pre-requisites 121
2.2. Installation 121
Software/Driver Loading 122
Software/Driver Configuration and Fine-tuning 123
4.1. Running the Application 123
Software/Driver Unloading 124
VIII. NVME-OF 125
Introduction 126
1.1. Hardware Requirements 126
1.2. Software Requirements 126
Kernel Installation 128
Software/Driver Installation 129
3.1. Pre-requisites 129

Chelsio Unified Wire for Linux viii
3.2. Installation 129
Software/Driver Loading 130
Software/Driver Configuration and Fine-tuning 131
5.1. Target 131
5.2. Initiator 132
5.3. HMA 132
5.4. Performance Tuning 133
Software/Driver Unloading 134
IX. LIO ISCSI TARGET OFFLOAD 135
Introduction 136
1.1. Hardware Requirements 136
1.2. Software Requirements 136
Kernel Configuration 138
Software/Driver Installation 141
3.1. Pre-requisites 141
3.2. Installation 141
Software/Driver Loading 143
Software/Driver Configuration and Fine-tuning 144
5.1. Configuring LIO iSCSI Target 144
5.2. Offloading LIO iSCSI Connection 144
5.3. Running LIO iSCSI and Network Traffic Concurrently 145
5.4. Performance Tuning 146
Software/Driver Unloading 147
6.1. Unloading the LIO iSCSI Target Offload Driver 147
6.2. Unloading the NIC Driver 147
X. ISCSI PDU OFFLOAD TARGET 148
Introduction 149
1.1. Features 149
1.2. Hardware Requirements 150
1.3. Software Requirements 151
Software/Driver Installation 153
Software/Driver Loading 154
3.1. Latest iSCSI Software Stack Driver Software 154
Software/Driver Configuration and Fine-tuning 156
4.1. Command Line Tools 156
4.2. iSCSI Configuration File 156
4.3. A Quick Start Guide for Target 157
4.4. The iSCSI Configuration File 159
4.5. Challenge-Handshake Authenticate Protocol (CHAP) 170
4.6. Target Access Control List (ACL) Configuration 172

Chelsio Unified Wire for Linux ix
4.7. Target Storage Device Configuration 174
4.8. Target Redirection Support 176
4.9. The command line interface tools “iscsictl” & “chisns” 178
4.10. Rules of Target Reload (i.e. “on the fly” changes) 183
4.11. System Wide Parameters 184
4.12. Performance Tuning 185
Software/Driver Unloading 186
XI. ISCSI PDU OFFLOAD INITIATOR 187
Introduction 188
1.1. Hardware Requirements 188
1.2. Software Requirements 189
Software/Driver Installation 190
2.1. Pre-requisites 190
2.2. Installation 190
Software/Driver Loading 191
Software/Driver Configuration and Fine-tuning 192
4.1. Accelerating open-iSCSI Initiator 192
4.2. HMA 194
4.3. Auto login from cxgb4i initiator at OS bootup 195
4.4. Performance Tuning 196
Software/Driver Unloading 197
XII. CRYPTO OFFLOAD 198
Introduction 199
1.1. Hardware Requirements 199
1.2. Software Requirements 199
Kernel Configuration 200
Software/Driver Installation 203
3.1. Pre-requisites 203
3.2. Installation 203
Software/Driver Loading 204
4.1. Co-processor 204
4.2. Inline 204
Software/Driver Configuration and Fine-tuning 205
5.1. Co-processor 205
5.2. Inline 206
5.3. Performance Tuning 209
Software/Driver Unloading 210
XIII. DATA CENTER BRIDGING (DCB) 211
Introduction 212

Chelsio Unified Wire for Linux x
1.1. Hardware Requirements 212
1.2. Software Requirements 212
Software/Driver Installation 214
Software/Driver Loading 215
Software/Driver Configuration and Fine-tuning 217
4.1. Configuring Cisco Nexus 5010 switch 217
4.2. Configuring the Brocade 8000 switch 220
Running NIC & iSCSI Traffic together with DCBx 222
XIV. FCOE FULL OFFLOAD INITIATOR 223
Introduction 224
1.1. Hardware Requirements 224
1.2. Software Requirements 224
Software/Driver Installation 225
Software/Driver Loading 226
Software/Driver Configuration and Fine-tuning 227
4.1. Configuring Cisco Nexus 5010 and Brocade switch 227
4.2. FCoE fabric discovery verification 227
4.3. Formatting the LUNs and Mounting the Filesystem 231
4.4. Creating Filesystem 232
4.5. Mounting the formatted LUN 233
Software/Driver Unloading 234
XV. OFFLOAD BONDING 235
Introduction 236
1.1. Hardware Requirements 236
1.2. Software Requirements 236
Software/Driver Installation 238
Software/Driver Loading 239
Software/Driver Configuration and Fine-tuning 240
4.1. Offloading TCP traffic over a bonded interface 240
Software/Driver Unloading 241
XVI. OFFLOAD MULTI-ADAPTER FAILOVER (MAFO) 242
Introduction 243
1.1. Hardware Requirements 243
1.2. Software Requirements 244
Software/Driver Installation 245
Software/Driver Loading 246
Software/Driver Configuration and Fine-tuning 247
4.1. Offloading TCP traffic over a bonded interface 247
Software/Driver Unloading 248