Cavium 41 Series User manual

AH0054602-00 J
Third party information brought to
you courtesy of Dell EMC.
User’s Guide
Converged Network Adapters
41xxx Series

ii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Document Revision History
Revision A, April 28, 2017
Revision B, August 24, 2017
Revision C, October 1, 2017
Revision D, January 24, 2018
Revision E, March 15, 2018
Revision F, April 19, 2018
Revision G, May 22, 2018
Revision H, August 23, 2018
Revision J, January 24, 2019
Changes Sections Affected
In Table 2-2, updated the OS requirements:
Removed support for Windows Nano. Added
support for 2008 SP2 R2 SP2 and Windows
2019.
Removed support for RHEL 6.8, 7.2, 7.3, 7.4.
Added support for RHEL 7.5 and 7.6.
Removed support for SLES 11 SP4 and
SLES 12 SP2. Added support for SLES 15.
Removed support for ESXi 6.0 U3 and later for
25G adapters. Added support for vSphere ESXi
6.5 U2 and vSphere ESXi 6.7 U1.
“System Requirements” on page 4
Following Step 4, added a note: “The maximum
number of supported qedentv Ethernet interfaces
on an ESXi host is 32, because the vmkernel
allows only 32 interfaces to register for manage-
ment callback.”
“Installing VMware Drivers” on page 29
Added a note describing the driver limitation as it
applies to the iSCSI vmnic.
“iSCSI Support” on page 34
Moved iSCSI boot from SAN information from the
iSCSI Configuration chapter to a new chapter.
Added and corrected iSCSI BFS information.
Chapter 6 Boot from SAN Configuration
Moved FCoE boot from SAN information from the
FCoE Configuration chapter to a new chapter.
Added and corrected FCoE BFS information.
Chapter 6 Boot from SAN Configuration

iii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Updated Ta ble 7-1 to reflect the current OS
support for RoCE v1, RoCE v2, iWARP, iSER, and
OFED.
“Supported Operating Systems and OFED” on
page 131
In Table 7-2, replaced the RDMA Mode property
with the NetworkDirect Technology prop-
erty.
“Configuring RoCE on the Adapter for Windows
Server” on page 135
In Step 1, updated the supported kernels for
RoCE v2 for Linux.
“RoCE v2 Configuration for Linux” on page 148
In Step 5, updated the guest OS for VM for
PVRDMA.
“Configuring a Paravirtual RDMA Device
(PVRDMA)” on page 157
Updated the following iWARP procedures:
In Step 1 c of the To enable iWARP on the
Windows host and verify RDMA procedure,
changed the parameter to NetworkDirect
Technology (was RDMA Mode).
In Step 4 b of the To verify iWARP traffic
procedure, changed the command to netstat
-xan (was net -xan).
“Configuring iWARP on Windows” on page 167
In the To configure the NFS server procedure:
Added a new Step 1 that describes how to
create an nfs-server directory and grant
permission.
In Step 2, added rw, to the command.
Replaced Step 4 for loading the service.
“Configuring NFS-RDMA” on page 175
In the To configure the NFS client procedure:
Added a new Step 1 that describes how to
create an nfs-client directory and grant
permission.
In Step 3, updated the command used for NFS
Version 4.
“Configuring NFS-RDMA” on page 175
Updated the list of OSs for configuring iSER in
inbox OFED.
“Before You Begin” on page 179
Following Step 12, added a fourth bullet to the
note: “Switch dependent teaming (IEEE 802.3ad
LACP and Generic/Static Link Aggregation (Trunk-
ing) cannot use a switch independent partitioned
virtual adapter. IEEE standards require Switch
Dependent Teaming (IEEE 802.3ad LACP and
Generic/Static Link Aggregation (Trunking)) mode
to work per the entire port instead of just the MAC
address (fraction of a port) granularity.”
“Configuring Microsoft Initiator to Use Cavium’s
iSCSI Offload” on page 192

iv AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Added support for Windows Server 2019 to the
section.
“Windows Server 2012 R2, 2016, and 2019 iSCSI
Boot Installation” on page 199
In the To create a Hyper-V virtual switch with an
RDMA NIC procedure:
Changed the section title to “...RDMA NIC” (was
“...RDMA Virtual NIC”.
Changed procedure title (was To create a
Hyper-V virtual switch with an RDMA virtual
NIC),
Added a new Step 1: “On all physical inter-
faces, set the value of the NetworkDirect
Functionality parameter to Enabled.”
“Creating a Hyper-V Virtual Switch with an RDMA
NIC” on page 239
In the To add host virtual NICs procedure:
In Step 2, corrected the last digit of the com-
mand to “S” (was “5”).
Changed the note’s first bullet, second sen-
tence to: “The same VLAN ID must be assigned
to ports on the switch” (was: “The same VLAN
ID must be assigned to all the interfaces, and
on the switch”).
“Adding a vLAN ID to Host Virtual NIC” on
page 240
Deleted the obsolete section, Default and
Maximum VMMQ Virtual NIC.
“Configuring VMMQ” on page 254
In Step 4, deleted the note and example. “Creating a VM and Enabling VMMQ on VMNet-
workadapters in the VM” on page 257
In the first bullet, deleted the statement, “The
Management OS VNIC has four VMMQs.”
“Enabling and Disabling VMMQ on a Management
NIC” on page 258
Added a note referencing the new parameter
added for Windows Server 2016 and Windows
Server 2019 to configure the maximum quantity of
queue pairs on a virtual port.
“Monitoring Traffic Statistics” on page 258
Deleted the obsolete “Deploying and Managing a
Nano Server” section.
Chapter 14 Windows Server 2016
Added a new chapter describing the features of
Windows Server 2019.
Chapter 15 Windows Server 2019
In Table B-1, added support for the Dell 10/25G
dual rate transceiver.
“Tested Cables and Optical Modules” on page 280
Added terms and definitions. Glossary

v AH0054602-00 J
Table of Contents
Preface
Supported Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii
What Is in This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx
Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
Laser Safety—FDA Notice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
Agency Certification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
EMI and EMC Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
KCC: Class A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
VCCI: Class A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
Product Safety Compliance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
1 Product Overview
Functional Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Adapter Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Physical Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Standards Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Hardware Installation
System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Preinstallation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Installing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3 Driver Installation
Installing Linux Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Installing the Linux Drivers Without RDMA . . . . . . . . . . . . . . . . . . . . . 10
Removing the Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Installing Linux Drivers Using the src RPM Package . . . . . . . . . 12
Installing Linux Drivers Using the kmp/kmod RPM Package . . . 13
Installing Linux Drivers Using the TAR File. . . . . . . . . . . . . . . . . 13

vi AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Installing the Linux Drivers with RDMA . . . . . . . . . . . . . . . . . . . . . . . . 14
Linux Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Linux Driver Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Importing a Public Key for Secure Boot. . . . . . . . . . . . . . . . . . . . . . . . 16
Installing Windows Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Installing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Running the DUP in the GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
DUP Installation Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
DUP Installation Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Removing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Managing Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Setting Power Management Options. . . . . . . . . . . . . . . . . . . . . . . . . . 27
Installing VMware Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
VMware Drivers and Driver Packages. . . . . . . . . . . . . . . . . . . . . . . . . 28
Installing VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
VMware NIC Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . 31
VMware Driver Parameter Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Removing the VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
FCoE Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
iSCSI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4 Upgrading the Firmware
Running the DUP by Double-Clicking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Running the DUP from a Command Line. . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Running the DUP Using the .bin File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5 Adapter Preboot Configuration
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Displaying Firmware Image Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Configuring Device-level Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Configuring NIC Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Configuring Data Center Bridging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Configuring FCoE Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Configuring iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Configuring Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Partitioning for VMware ESXi 6.0 and ESXi 6.5 . . . . . . . . . . . . . . . . . 62

vii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
6 Boot from SAN Configuration
iSCSI Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
iSCSI Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 65
iSCSI Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Setting the BIOS Boot Mode to UEFI . . . . . . . . . . . . . . . . . . . . . 66
Enabling NPAR and the iSCSI HBA . . . . . . . . . . . . . . . . . . . . . . 68
Selecting the iSCSI UEFI Boot Protocol. . . . . . . . . . . . . . . . . . . 69
Configuring the Storage Target. . . . . . . . . . . . . . . . . . . . . . . . . . 70
Configuring iSCSI Boot Parameters . . . . . . . . . . . . . . . . . . . . . . 70
Configuring BIOS Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Configuring iSCSI Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . 73
Configuring the DHCP Server to Support iSCSI Boot . . . . . . . . 85
Configuring iSCSI Boot from SAN on Windows . . . . . . . . . . . . . . . . . 89
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Selecting the Preferred iSCSI Boot Mode . . . . . . . . . . . . . . . . . 90
Configuring iSCSI General Parameters . . . . . . . . . . . . . . . . . . . 90
Configuring the iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Configuring the iSCSI Targets . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Detecting the iSCSI LUN and Injecting the QLogic Drivers . . . .92
Configuring iSCSI Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 94
Configuring iSCSI Boot from SAN for RHEL 7.5 and Later . . . . 95
Configuring iSCSI Boot from SAN for SLES 12 SP3
and Later . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Configuring iSCSI Boot from SAN for Other Linux
Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Configuring iSCSI Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 109
Setting the UEFI Main Configuration . . . . . . . . . . . . . . . . . . . . . 109
Configuring the System BIOS for iSCSI Boot (L2) . . . . . . . . . . . 111
Mapping the CD or DVD for OS Installation . . . . . . . . . . . . . . . . 113
FCoE Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
FCoE Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 115
FCoE Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Specifying the BIOS Boot Protocol. . . . . . . . . . . . . . . . . . . . . . . 116
Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 116
Configuring FCoE Boot from SAN on Windows . . . . . . . . . . . . . . . . . 121
Windows Server 2012 R2 and 2016 FCoE Boot Installation . . . 121
Configuring FCoE on Windows . . . . . . . . . . . . . . . . . . . . . . . . . 122
FCoE Crash Dump on Windows. . . . . . . . . . . . . . . . . . . . . . . . . 122
Injecting (Slipstreaming) Adapter Drivers into Windows
Image Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

viii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Configuring FCoE Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 124
Prerequisites for Linux FCoE Boot from SAN. . . . . . . . . . . . . . . 124
Configuring Linux FCoE Boot from SAN . . . . . . . . . . . . . . . . . . 125
Configuring FCoE Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 127
Injecting (Slipstreaming) ESXi Adapter Drivers into Image Files . 127
Installing the Customized ESXi ISO . . . . . . . . . . . . . . . . . . . . . . 128
7 RoCE Configuration
Supported Operating Systems and OFED . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Planning for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Preparing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Preparing the Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Configuring the Cisco Nexus 6000 Ethernet Switch . . . . . . . . . . . . . . 134
Configuring the Dell Z9100 Ethernet Switch . . . . . . . . . . . . . . . . . . . . 135
Configuring RoCE on the Adapter for Windows Server . . . . . . . . . . . . . . . . 135
Viewing RDMA Counters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Configuring RoCE on the Adapter for Linux . . . . . . . . . . . . . . . . . . . . . . . . . 143
RoCE Configuration for RHEL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
RoCE Configuration for SLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Verifying the RoCE Configuration on Linux . . . . . . . . . . . . . . . . . . . . . 145
vLAN Interfaces and GID Index Values. . . . . . . . . . . . . . . . . . . . . . . . 147
RoCE v2 Configuration for Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Identifying the RoCE v2 GID Index or Address . . . . . . . . . . . . . 148
Verifying the RoCE v1 or RoCE v2 GID Index and Address
from sys and class Parameters . . . . . . . . . . . . . . . . . . . . . . . . 149
Verifying the RoCE v1 or RoCE v2 Function Through
perftest Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Configuring RoCE on the Adapter for VMware ESX . . . . . . . . . . . . . . . . . . 154
Configuring RDMA Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
Configuring MTU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
RoCE Mode and Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Configuring a Paravirtual RDMA Device (PVRDMA). . . . . . . . . . . . . . 157
Configuring DCQCN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
DCQCN Terminology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
DCQCN Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
DCB-related Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Global Settings on RDMA Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Setting vLAN Priority on RDMA Traffic . . . . . . . . . . . . . . . . . . . . 162
Setting ECN on RDMA Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Setting DSCP on RDMA Traffic . . . . . . . . . . . . . . . . . . . . . . . . . 162

ix AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Configuring DCQCN (continued)
Configuring DSCP-PFC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Enabling DCQCN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Configuring CNP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
DCQCN Algorithm Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
MAC Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Script Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
8 iWARP Configuration
Preparing the Adapter for iWARP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Configuring iWARP on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Configuring iWARP on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Installing the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Configuring iWARP and RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Detecting the Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Supported iWARP Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Running Perftest for iWARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Configuring NFS-RDMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
iWARP RDMA-Core Support on SLES 12 SP3 and OFED 4.8x . . . . . 176
9 iSER Configuration
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Configuring iSER for RHEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Configuring iSER for SLES 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Using iSER with iWARP on RHEL and SLES. . . . . . . . . . . . . . . . . . . . . . . . 184
Optimizing Linux Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Configuring CPUs to Maximum Performance Mode . . . . . . . . . . . . . . 186
Configuring Kernel sysctl Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Configuring IRQ Affinity Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Configuring Block Device Staging . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Configuring iSER on ESXi 6.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Before You Begin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Configuring iSER for ESXi 6.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
10 iSCSI Configuration
iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
iSCSI Offload in Windows Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Installing Cavium QLogic Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Installing the Microsoft iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . 192
Configuring Microsoft Initiator to Use Cavium’s iSCSI Offload . . . . . . 192

x AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
iSCSI Offload in Windows Server (continued)
iSCSI Offload FAQs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Windows Server 2012 R2, 2016, and 2019 iSCSI Boot Installation . . 199
iSCSI Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
iSCSI Offload in Linux Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Differences from bnx2i. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Configuring qedi.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Verifying iSCSI Interfaces in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
11 FCoE Configuration
Configuring Linux FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Differences Between qedf and bnx2fc. . . . . . . . . . . . . . . . . . . . . . . . . 205
Configuring qedf.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Verifying FCoE Devices in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
12 SR-IOV Configuration
Configuring SR-IOV on Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
Configuring SR-IOV on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Enabling IOMMU for SR-IOV in UEFI-based Linux OS Installations . . . 220
Configuring SR-IOV on VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
13 NVMe-oF Configuration with RDMA
Installing Device Drivers on Both Servers . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Configuring the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Configuring the Initiator Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Preconditioning the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Testing the NVMe-oF Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Optimizing Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
.IRQ Affinity (multi_rss-affin.sh) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
CPU Frequency (cpufreq.sh). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
14 Windows Server 2016
Configuring RoCE Interfaces with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . 238
Creating a Hyper-V Virtual Switch with an RDMA NIC . . . . . . . . . . . . 239
Adding a vLAN ID to Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . 240
Verifying If RoCE is Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Adding Host Virtual NICs (Virtual Ports) . . . . . . . . . . . . . . . . . . . . . . . 242
Mapping the SMB Drive and Running RoCE Traffic . . . . . . . . . . . . . . 242
RoCE over Switch Embedded Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Creating a Hyper-V Virtual Switch with SET and RDMA
Virtual NICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244