Cavium 41 Series User manual

AH0054602-00 J
Third party information brought to
you courtesy of Dell EMC.
User’s Guide
Converged Network Adapters
41xxx Series

ii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Document Revision History
Revision A, April 28, 2017
Revision B, August 24, 2017
Revision C, October 1, 2017
Revision D, January 24, 2018
Revision E, March 15, 2018
Revision F, April 19, 2018
Revision G, May 22, 2018
Revision H, August 23, 2018
Revision J, January 24, 2019
Changes Sections Affected
In Table 2-2, updated the OS requirements:
Removed support for Windows Nano. Added
support for 2008 SP2 R2 SP2 and Windows
2019.
Removed support for RHEL 6.8, 7.2, 7.3, 7.4.
Added support for RHEL 7.5 and 7.6.
Removed support for SLES 11 SP4 and
SLES 12 SP2. Added support for SLES 15.
Removed support for ESXi 6.0 U3 and later for
25G adapters. Added support for vSphere ESXi
6.5 U2 and vSphere ESXi 6.7 U1.
“System Requirements” on page 4
Following Step 4, added a note: “The maximum
number of supported qedentv Ethernet interfaces
on an ESXi host is 32, because the vmkernel
allows only 32 interfaces to register for manage-
ment callback.”
“Installing VMware Drivers” on page 29
Added a note describing the driver limitation as it
applies to the iSCSI vmnic.
“iSCSI Support” on page 34
Moved iSCSI boot from SAN information from the
iSCSI Configuration chapter to a new chapter.
Added and corrected iSCSI BFS information.
Chapter 6 Boot from SAN Configuration
Moved FCoE boot from SAN information from the
FCoE Configuration chapter to a new chapter.
Added and corrected FCoE BFS information.
Chapter 6 Boot from SAN Configuration

iii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Updated Ta ble 7-1 to reflect the current OS
support for RoCE v1, RoCE v2, iWARP, iSER, and
OFED.
“Supported Operating Systems and OFED” on
page 131
In Table 7-2, replaced the RDMA Mode property
with the NetworkDirect Technology prop-
erty.
“Configuring RoCE on the Adapter for Windows
Server” on page 135
In Step 1, updated the supported kernels for
RoCE v2 for Linux.
“RoCE v2 Configuration for Linux” on page 148
In Step 5, updated the guest OS for VM for
PVRDMA.
“Configuring a Paravirtual RDMA Device
(PVRDMA)” on page 157
Updated the following iWARP procedures:
In Step 1 c of the To enable iWARP on the
Windows host and verify RDMA procedure,
changed the parameter to NetworkDirect
Technology (was RDMA Mode).
In Step 4 b of the To verify iWARP traffic
procedure, changed the command to netstat
-xan (was net -xan).
“Configuring iWARP on Windows” on page 167
In the To configure the NFS server procedure:
Added a new Step 1 that describes how to
create an nfs-server directory and grant
permission.
In Step 2, added rw, to the command.
Replaced Step 4 for loading the service.
“Configuring NFS-RDMA” on page 175
In the To configure the NFS client procedure:
Added a new Step 1 that describes how to
create an nfs-client directory and grant
permission.
In Step 3, updated the command used for NFS
Version 4.
“Configuring NFS-RDMA” on page 175
Updated the list of OSs for configuring iSER in
inbox OFED.
“Before You Begin” on page 179
Following Step 12, added a fourth bullet to the
note: “Switch dependent teaming (IEEE 802.3ad
LACP and Generic/Static Link Aggregation (Trunk-
ing) cannot use a switch independent partitioned
virtual adapter. IEEE standards require Switch
Dependent Teaming (IEEE 802.3ad LACP and
Generic/Static Link Aggregation (Trunking)) mode
to work per the entire port instead of just the MAC
address (fraction of a port) granularity.”
“Configuring Microsoft Initiator to Use Cavium’s
iSCSI Offload” on page 192

iv AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Added support for Windows Server 2019 to the
section.
“Windows Server 2012 R2, 2016, and 2019 iSCSI
Boot Installation” on page 199
In the To create a Hyper-V virtual switch with an
RDMA NIC procedure:
Changed the section title to “...RDMA NIC” (was
“...RDMA Virtual NIC”.
Changed procedure title (was To create a
Hyper-V virtual switch with an RDMA virtual
NIC),
Added a new Step 1: “On all physical inter-
faces, set the value of the NetworkDirect
Functionality parameter to Enabled.”
“Creating a Hyper-V Virtual Switch with an RDMA
NIC” on page 239
In the To add host virtual NICs procedure:
In Step 2, corrected the last digit of the com-
mand to “S” (was “5”).
Changed the note’s first bullet, second sen-
tence to: “The same VLAN ID must be assigned
to ports on the switch” (was: “The same VLAN
ID must be assigned to all the interfaces, and
on the switch”).
“Adding a vLAN ID to Host Virtual NIC” on
page 240
Deleted the obsolete section, Default and
Maximum VMMQ Virtual NIC.
“Configuring VMMQ” on page 254
In Step 4, deleted the note and example. “Creating a VM and Enabling VMMQ on VMNet-
workadapters in the VM” on page 257
In the first bullet, deleted the statement, “The
Management OS VNIC has four VMMQs.”
“Enabling and Disabling VMMQ on a Management
NIC” on page 258
Added a note referencing the new parameter
added for Windows Server 2016 and Windows
Server 2019 to configure the maximum quantity of
queue pairs on a virtual port.
“Monitoring Traffic Statistics” on page 258
Deleted the obsolete “Deploying and Managing a
Nano Server” section.
Chapter 14 Windows Server 2016
Added a new chapter describing the features of
Windows Server 2019.
Chapter 15 Windows Server 2019
In Table B-1, added support for the Dell 10/25G
dual rate transceiver.
“Tested Cables and Optical Modules” on page 280
Added terms and definitions. Glossary

v AH0054602-00 J
Table of Contents
Preface
Supported Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii
What Is in This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx
Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
Laser Safety—FDA Notice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii
Agency Certification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
EMI and EMC Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
KCC: Class A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
VCCI: Class A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
Product Safety Compliance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
1 Product Overview
Functional Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Adapter Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Physical Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Standards Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Hardware Installation
System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Preinstallation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Installing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3 Driver Installation
Installing Linux Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Installing the Linux Drivers Without RDMA . . . . . . . . . . . . . . . . . . . . . 10
Removing the Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Installing Linux Drivers Using the src RPM Package . . . . . . . . . 12
Installing Linux Drivers Using the kmp/kmod RPM Package . . . 13
Installing Linux Drivers Using the TAR File. . . . . . . . . . . . . . . . . 13

vi AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Installing the Linux Drivers with RDMA . . . . . . . . . . . . . . . . . . . . . . . . 14
Linux Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Linux Driver Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Importing a Public Key for Secure Boot. . . . . . . . . . . . . . . . . . . . . . . . 16
Installing Windows Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Installing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Running the DUP in the GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
DUP Installation Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
DUP Installation Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Removing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Managing Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Setting Power Management Options. . . . . . . . . . . . . . . . . . . . . . . . . . 27
Installing VMware Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
VMware Drivers and Driver Packages. . . . . . . . . . . . . . . . . . . . . . . . . 28
Installing VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
VMware NIC Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . 31
VMware Driver Parameter Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Removing the VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
FCoE Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
iSCSI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4 Upgrading the Firmware
Running the DUP by Double-Clicking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Running the DUP from a Command Line. . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Running the DUP Using the .bin File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5 Adapter Preboot Configuration
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Displaying Firmware Image Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Configuring Device-level Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Configuring NIC Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Configuring Data Center Bridging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Configuring FCoE Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Configuring iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Configuring Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Partitioning for VMware ESXi 6.0 and ESXi 6.5 . . . . . . . . . . . . . . . . . 62

vii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
6 Boot from SAN Configuration
iSCSI Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
iSCSI Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 65
iSCSI Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Setting the BIOS Boot Mode to UEFI . . . . . . . . . . . . . . . . . . . . . 66
Enabling NPAR and the iSCSI HBA . . . . . . . . . . . . . . . . . . . . . . 68
Selecting the iSCSI UEFI Boot Protocol. . . . . . . . . . . . . . . . . . . 69
Configuring the Storage Target. . . . . . . . . . . . . . . . . . . . . . . . . . 70
Configuring iSCSI Boot Parameters . . . . . . . . . . . . . . . . . . . . . . 70
Configuring BIOS Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Configuring iSCSI Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . 73
Configuring the DHCP Server to Support iSCSI Boot . . . . . . . . 85
Configuring iSCSI Boot from SAN on Windows . . . . . . . . . . . . . . . . . 89
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Selecting the Preferred iSCSI Boot Mode . . . . . . . . . . . . . . . . . 90
Configuring iSCSI General Parameters . . . . . . . . . . . . . . . . . . . 90
Configuring the iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Configuring the iSCSI Targets . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Detecting the iSCSI LUN and Injecting the QLogic Drivers . . . .92
Configuring iSCSI Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 94
Configuring iSCSI Boot from SAN for RHEL 7.5 and Later . . . . 95
Configuring iSCSI Boot from SAN for SLES 12 SP3
and Later . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Configuring iSCSI Boot from SAN for Other Linux
Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Configuring iSCSI Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 109
Setting the UEFI Main Configuration . . . . . . . . . . . . . . . . . . . . . 109
Configuring the System BIOS for iSCSI Boot (L2) . . . . . . . . . . . 111
Mapping the CD or DVD for OS Installation . . . . . . . . . . . . . . . . 113
FCoE Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
FCoE Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 115
FCoE Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Specifying the BIOS Boot Protocol. . . . . . . . . . . . . . . . . . . . . . . 116
Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 116
Configuring FCoE Boot from SAN on Windows . . . . . . . . . . . . . . . . . 121
Windows Server 2012 R2 and 2016 FCoE Boot Installation . . . 121
Configuring FCoE on Windows . . . . . . . . . . . . . . . . . . . . . . . . . 122
FCoE Crash Dump on Windows. . . . . . . . . . . . . . . . . . . . . . . . . 122
Injecting (Slipstreaming) Adapter Drivers into Windows
Image Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

viii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Configuring FCoE Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 124
Prerequisites for Linux FCoE Boot from SAN. . . . . . . . . . . . . . . 124
Configuring Linux FCoE Boot from SAN . . . . . . . . . . . . . . . . . . 125
Configuring FCoE Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 127
Injecting (Slipstreaming) ESXi Adapter Drivers into Image Files . 127
Installing the Customized ESXi ISO . . . . . . . . . . . . . . . . . . . . . . 128
7 RoCE Configuration
Supported Operating Systems and OFED . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Planning for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Preparing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Preparing the Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Configuring the Cisco Nexus 6000 Ethernet Switch . . . . . . . . . . . . . . 134
Configuring the Dell Z9100 Ethernet Switch . . . . . . . . . . . . . . . . . . . . 135
Configuring RoCE on the Adapter for Windows Server . . . . . . . . . . . . . . . . 135
Viewing RDMA Counters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Configuring RoCE on the Adapter for Linux . . . . . . . . . . . . . . . . . . . . . . . . . 143
RoCE Configuration for RHEL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
RoCE Configuration for SLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Verifying the RoCE Configuration on Linux . . . . . . . . . . . . . . . . . . . . . 145
vLAN Interfaces and GID Index Values. . . . . . . . . . . . . . . . . . . . . . . . 147
RoCE v2 Configuration for Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Identifying the RoCE v2 GID Index or Address . . . . . . . . . . . . . 148
Verifying the RoCE v1 or RoCE v2 GID Index and Address
from sys and class Parameters . . . . . . . . . . . . . . . . . . . . . . . . 149
Verifying the RoCE v1 or RoCE v2 Function Through
perftest Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Configuring RoCE on the Adapter for VMware ESX . . . . . . . . . . . . . . . . . . 154
Configuring RDMA Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
Configuring MTU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
RoCE Mode and Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Configuring a Paravirtual RDMA Device (PVRDMA). . . . . . . . . . . . . . 157
Configuring DCQCN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
DCQCN Terminology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
DCQCN Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
DCB-related Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Global Settings on RDMA Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Setting vLAN Priority on RDMA Traffic . . . . . . . . . . . . . . . . . . . . 162
Setting ECN on RDMA Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Setting DSCP on RDMA Traffic . . . . . . . . . . . . . . . . . . . . . . . . . 162

ix AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
Configuring DCQCN (continued)
Configuring DSCP-PFC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Enabling DCQCN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Configuring CNP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
DCQCN Algorithm Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
MAC Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Script Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
8 iWARP Configuration
Preparing the Adapter for iWARP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Configuring iWARP on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Configuring iWARP on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Installing the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Configuring iWARP and RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Detecting the Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Supported iWARP Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Running Perftest for iWARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Configuring NFS-RDMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
iWARP RDMA-Core Support on SLES 12 SP3 and OFED 4.8x . . . . . 176
9 iSER Configuration
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Configuring iSER for RHEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Configuring iSER for SLES 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Using iSER with iWARP on RHEL and SLES. . . . . . . . . . . . . . . . . . . . . . . . 184
Optimizing Linux Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Configuring CPUs to Maximum Performance Mode . . . . . . . . . . . . . . 186
Configuring Kernel sysctl Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Configuring IRQ Affinity Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Configuring Block Device Staging . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Configuring iSER on ESXi 6.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Before You Begin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Configuring iSER for ESXi 6.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
10 iSCSI Configuration
iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
iSCSI Offload in Windows Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Installing Cavium QLogic Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Installing the Microsoft iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . 192
Configuring Microsoft Initiator to Use Cavium’s iSCSI Offload . . . . . . 192

x AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
iSCSI Offload in Windows Server (continued)
iSCSI Offload FAQs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Windows Server 2012 R2, 2016, and 2019 iSCSI Boot Installation . . 199
iSCSI Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
iSCSI Offload in Linux Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Differences from bnx2i. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Configuring qedi.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Verifying iSCSI Interfaces in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
11 FCoE Configuration
Configuring Linux FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Differences Between qedf and bnx2fc. . . . . . . . . . . . . . . . . . . . . . . . . 205
Configuring qedf.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Verifying FCoE Devices in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
12 SR-IOV Configuration
Configuring SR-IOV on Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
Configuring SR-IOV on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Enabling IOMMU for SR-IOV in UEFI-based Linux OS Installations . . . 220
Configuring SR-IOV on VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
13 NVMe-oF Configuration with RDMA
Installing Device Drivers on Both Servers . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Configuring the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Configuring the Initiator Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Preconditioning the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Testing the NVMe-oF Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Optimizing Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
.IRQ Affinity (multi_rss-affin.sh) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
CPU Frequency (cpufreq.sh). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
14 Windows Server 2016
Configuring RoCE Interfaces with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . 238
Creating a Hyper-V Virtual Switch with an RDMA NIC . . . . . . . . . . . . 239
Adding a vLAN ID to Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . 240
Verifying If RoCE is Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Adding Host Virtual NICs (Virtual Ports) . . . . . . . . . . . . . . . . . . . . . . . 242
Mapping the SMB Drive and Running RoCE Traffic . . . . . . . . . . . . . . 242
RoCE over Switch Embedded Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Creating a Hyper-V Virtual Switch with SET and RDMA
Virtual NICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244

xi AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
RoCE over Switch Embedded Teaming (continued)
Enabling RDMA on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Assigning a vLAN ID on SET. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Running RDMA Traffic on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Configuring QoS for RoCE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Configuring QoS by Disabling DCBX on the Adapter . . . . . . . . . . . . . 246
Configuring QoS by Enabling DCBX on the Adapter. . . . . . . . . . . . . . 250
Configuring VMMQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
Enabling VMMQ on the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
Creating a Virtual Machine Switch with or Without SR-IOV . . . . . . . . 255
Enabling VMMQ on the Virtual Machine Switch . . . . . . . . . . . . . . . . . 256
Getting the Virtual Machine Switch Capability. . . . . . . . . . . . . . . . . . . 257
Creating a VM and Enabling VMMQ on VMNetworkadapters
in the VM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Enabling and Disabling VMMQ on a Management NIC . . . . . . . . . . . 258
Monitoring Traffic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Configuring VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Enabling VXLAN Offload on the Adapter. . . . . . . . . . . . . . . . . . . . . . . 259
Deploying a Software Defined Network. . . . . . . . . . . . . . . . . . . . . . . . 259
Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Configuring the Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Deploying a Hyper-Converged System . . . . . . . . . . . . . . . . . . . . . . . . 261
Deploying the Operating System . . . . . . . . . . . . . . . . . . . . . . . . 261
Configuring the Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . 263
15 Windows Server 2019
RSSv2 for Hyper-V. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
RSSv2 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Known Event Log Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Windows Server 2019 Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
VMMQ Is Enabled by Default . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Inbox Driver Network Direct (RDMA) Is Disabled by Default. . . . . . . . 268
New Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Max Queue Pairs (L2) Per VPort. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Network Direct Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Virtualization Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
VMQ and VMMQ Default Accelerations . . . . . . . . . . . . . . . . . . . . . . . 271
Single VPort Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

xii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
16 Troubleshooting
Troubleshooting Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Verifying that Current Drivers Are Loaded . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Verifying Drivers in Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Verifying Drivers in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Verifying Drivers in VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Testing Network Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Testing Network Connectivity for Windows . . . . . . . . . . . . . . . . . . . . . 275
Testing Network Connectivity for Linux . . . . . . . . . . . . . . . . . . . . . . . . 276
Microsoft Virtualization with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Linux-specific Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Miscellaneous Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Collecting Debug Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
AAdapter LEDS
B Cables and Optical Modules
Supported Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
Tested Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
C Dell Z9100 Switch Configuration
D Feature Constraints
Glossary

xiii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
List of Figures
Figure Page
3-1 Dell Update Package Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3-2 QLogic InstallShield Wizard: Welcome Window . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3-3 QLogic InstallShield Wizard: License Agreement Window. . . . . . . . . . . . . . . . . . . . 20
3-4 InstallShield Wizard: Setup Type Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3-5 InstallShield Wizard: Custom Setup Window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3-6 InstallShield Wizard: Ready to Install the Program Window . . . . . . . . . . . . . . . . . . 22
3-7 InstallShield Wizard: Completed Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3-8 Dell Update Package Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3-9 Setting Advanced Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3-10 Power Management Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4-1 Dell Update Package: Splash Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4-2 Dell Update Package: Loading New Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4-3 Dell Update Package: Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4-4 Dell Update Package: Finish Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4-5 DUP Command Line Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5-1 System Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5-2 System Setup: Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5-3 Main Configuration Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5-4 Main Configuration Page, Setting Partitioning Mode to NPAR. . . . . . . . . . . . . . . . . 42
5-5 Firmware Image Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5-6 Device Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5-7 NIC Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5-8 System Setup: Data Center Bridging (DCB) Settings . . . . . . . . . . . . . . . . . . . . . . . 50
5-9 FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5-10 FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5-11 iSCSI General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5-12 iSCSI Initiator Configuration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5-13 iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5-14 iSCSI Second Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5-15 NIC Partitioning Configuration, Global Bandwidth Allocation. . . . . . . . . . . . . . . . . . 57
5-16 Global Bandwidth Allocation Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5-17 Partition 1 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5-18 Partition 2 Configuration: FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5-19 Partition 3 Configuration: iSCSI Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5-20 Partition 4 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6-1 System Setup: Boot Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6-2 System Setup: Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6-3 System Setup: NIC Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6-4 System Setup: Boot Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6-5 Selecting NIC Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6-6 System Setup: NIC Configuration, Boot Protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . 73
6-7 System Setup: iSCSI Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6-8 System Setup: Selecting General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

xiv AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
6-9 System Setup: iSCSI General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6-10 System Setup: Selecting iSCSI Initiator Parameters . . . . . . . . . . . . . . . . . . . . . . . . 77
6-11 System Setup: iSCSI Initiator Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6-12 System Setup: Selecting iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . 79
6-13 System Setup: iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6-14 System Setup: iSCSI Second Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6-15 System Setup: Saving iSCSI Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6-16 System Setup: iSCSI General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6-17 System Setup: iSCSI General Parameters, VLAN ID . . . . . . . . . . . . . . . . . . . . . . . 89
6-18 Detecting the iSCSI LUN Using UEFI Shell (Version 2). . . . . . . . . . . . . . . . . . . . . . 93
6-19 Windows Setup: Selecting Installation Destination . . . . . . . . . . . . . . . . . . . . . . . . . 93
6-20 Windows Setup: Selecting Driver to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6-21 Integrated NIC: Device Level Configuration for VMware . . . . . . . . . . . . . . . . . . . . . 109
6-22 Integrated NIC: Partition 2 Configuration for VMware . . . . . . . . . . . . . . . . . . . . . . . 110
6-23 Integrated NIC: System BIOS, Boot Settings for VMware . . . . . . . . . . . . . . . . . . . . 111
6-24 Integrated NIC: System BIOS, Connection 1 Settings for VMware . . . . . . . . . . . . . 112
6-25 Integrated NIC: System BIOS, Connection 1 Settings (Target) for VMware . . . . . . 112
6-26 VMware iSCSI BFS: Selecting a Disk to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6-27 VMware iSCSI Boot from SAN Successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6-28 System Setup: Selecting Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
6-29 System Setup: Device Settings, Port Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6-30 System Setup: NIC Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
6-31 System Setup: FCoE Mode Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6-32 System Setup: FCoE General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6-33 System Setup: FCoE General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6-34 Selecting the Driver Disk Image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6-35 Selecting the Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6-36 ESXi-Customizer Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6-37 Select a VMware Disk to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6-38 VMware Generic USB Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
7-1 Configuring RoCE Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7-2 Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7-3 Performance Monitor: Cavium FastLinQ Counters . . . . . . . . . . . . . . . . . . . . . . . . . 140
7-4 Switch Settings, Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7-5 Switch Settings, Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7-6 Configuring RDMA_CM Applications: Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7-7 Configuring RDMA_CM Applications: Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7-8 Configuring a New Distributed Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
7-9 Assigning a vmknic for PVRDMA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7-10 Setting the Firewall Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8-1 Windows PowerShell Command: Get-NetAdapterRdma. . . . . . . . . . . . . . . . . . . . . 168
8-2 Windows PowerShell Command: Get-NetOffloadGlobalSetting . . . . . . . . . . . . . . . 168
8-3 Perfmon: Add Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
8-4 Perfmon: Verifying iWARP Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
9-1 RDMA Ping Successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

xv AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
9-2 iSER Portal Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9-3 Iface Transport Confirmed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
9-4 Checking for New iSCSI Device. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
9-5 LIO Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
10-1 iSCSI Initiator Properties, Configuration Page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
10-2 iSCSI Initiator Node Name Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
10-3 iSCSI Initiator—Discover Target Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
10-4 Target Portal IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
10-5 Selecting the Initiator IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
10-6 Connecting to the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10-7 Connect To Target Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
12-1 System Setup for SR-IOV: Integrated Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
12-2 System Setup for SR-IOV: Device Level Configuration . . . . . . . . . . . . . . . . . . . . . . 209
12-3 Adapter Properties, Advanced: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . 210
12-4 Virtual Switch Manager: Enabling SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
12-5 Settings for VM: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
12-6 Device Manager: VM with QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
12-7 Windows PowerShell Command: Get-NetadapterSriovVf . . . . . . . . . . . . . . . . . . . . 214
12-8 System Setup: Processor Settings for SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
12-9 System Setup for SR-IOV: Integrated Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
12-10 Editing the grub.conf File for SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
12-11 Command Output for sriov_numvfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
12-12 Command Output for ip link show Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
12-13 RHEL68 Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
12-14 Add New Virtual Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
12-15 VMware Host Edit Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
13-1 NVMe-oF Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
13-2 Subsystem NQN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
13-3 Confirm NVMe-oF Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
13-4 FIO Utility Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
14-1 Enabling RDMA in Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
14-2 Hyper-V Virtual Ethernet Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
14-3 Windows PowerShell Command: Get-VMNetworkAdapter . . . . . . . . . . . . . . . . . . . 241
14-4 Windows PowerShell Command: Get-NetAdapterRdma. . . . . . . . . . . . . . . . . . . . . 241
14-5 Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
14-6 Performance Monitor Shows RoCE Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
14-7 Windows PowerShell Command: New-VMSwitch . . . . . . . . . . . . . . . . . . . . . . . . . . 244
14-8 Windows PowerShell Command: Get-NetAdapter. . . . . . . . . . . . . . . . . . . . . . . . . . 245
14-9 Advanced Properties: Enable QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
14-10 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
14-11 Advanced Properties: Enabling QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
14-12 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
14-13 Advanced Properties: Enabling Virtual Switch RSS. . . . . . . . . . . . . . . . . . . . . . . . . 255
14-14 Virtual Switch Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
14-15 Windows PowerShell Command: Get-VMSwitch. . . . . . . . . . . . . . . . . . . . . . . . . . . 257

xvi AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
14-16 Advanced Properties: Enabling VXLAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
14-17 Example Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
15-1 RSSv2 Event Log Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268

xvii AH0054602-00 J
User’s Guide—Converged Network Adapters
41xxx Series
List of Tables
Table Page
2-1 Host Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2-2 Minimum Host Operating System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3-1 Cavium QLogic 41xxx Series Adapters Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . 9
3-2 qede Driver Optional Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3-3 Linux Driver Operation Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3-4 VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3-5 ESXi Driver Packages by Release. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3-6 VMware NIC Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3-7 VMware Driver Parameter Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5-1 Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6-1 iSCSI Out-of-Box and Inbox Boot from SAN Support . . . . . . . . . . . . . . . . . . . . . . . 65
6-2 iSCSI General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6-3 DHCP Option 17 Parameter Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6-4 DHCP Option 43 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6-5 DHCP Option 17 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6-6 FCoE Out-of-Box and Inbox Boot from SAN Support . . . . . . . . . . . . . . . . . . . . . . . 115
7-1 OS Support for RoCE v1, RoCE v2, iWARP, iSER, and OFED . . . . . . . . . . . . . . . . 131
7-2 Advanced Properties for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7-3 Cavium FastLinQ RDMA Error Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7-4 DCQCN Algorithm Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
13-1 Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
15-1 Windows 2019 Virtualization Resources for Dell 41xxx Series Adapters. . . . . . . . . 270
15-2 Windows 2019 VMQ and VMMQ Accelerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
16-1 Collecting Debug Data Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
A-1 Adapter Port Link and Activity LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
B-1 Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
B-2 Switches Tested for Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284

xviii AH0054602-00 J
Preface
This preface lists the supported products, specifies the intended audience,
explains the typographic conventions used in this guide, and describes legal
notices.
Supported Products
This user’s guide describes the following Cavium® products:
QL41112HFCU-DE 10Gb Converged Network Adapter, full-height bracket
QL41112HLCU-DE 10Gb Converged Network Adapter, low-profile bracket
QL41162HFRJ-DE 10Gb Converged Network Adapter, full-height bracket
QL41162HLRJ-DE 10Gb Converged Network Adapter, low-profile bracket
QL41162HMRJ-DE 10Gb Converged Network Adapter
QL41164HMCU-DE 10Gb Converged Network Adapter
QL41164HMRJ-DE 10Gb Converged Network Adapter
QL41262HFCU-DE 10/25Gb Converged Network Adapter, full-height
bracket
QL41262HLCU-DE 10/25Gb Converged Network Adapter, low-profile
bracket
QL41262HMCU-DE 10/25Gb Converged Network
QL41264HMCU-DE 10/25Gb Converged Network Adapter
Intended Audience
This guide is intended for system administrators and other technical staff
members responsible for configuring and managing adapters installed on Dell®
PowerEdge® servers in Windows®, Linux®, or VMware® environments.

Preface
What Is in This Guide
xix AH0054602-00 J
What Is in This Guide
Following this preface, the remainder of this guide is organized into the following
chapters and appendices:
Chapter 1 Product Overview provides a product functional description, a list
of features, and the adapter specifications.
Chapter 2 Hardware Installation describes how to install the adapter,
including the list of system requirements and a preinstallation checklist.
Chapter 3 Driver Installation describes the installation of the adapter drivers
on Windows, Linux, and VMware.
Chapter 4 Upgrading the Firmware describes how to use the Dell Update
Package (DUP) to upgrade the adapter firmware.
Chapter 5 Adapter Preboot Configuration describes the preboot adapter
configuration tasks using the Human Infrastructure Interface (HII)
application.
Chapter 6 Boot from SAN Configuration covers boot from SAN configuration
for both iSCSI and FCoE.
Chapter 7 RoCE Configuration describes how to configure the adapter, the
Ethernet switch, and the host to use RDMA over converged Ethernet
(RoCE).
Chapter 8 iWARP Configuration provides procedures for configuring Internet
wide area RDMA protocol (iWARP) on Windows, Linux, and VMware
ESXi 6.7 systems.
Chapter 9 iSER Configuration describes how to configure iSCSI Extensions
for RDMA (iSER) for Linux RHEL, SLES, Ubuntu, and ESXi 6.7.
Chapter 10 iSCSI Configuration describes iSCSI boot and iSCSI offload for
Windows and Linux.
Chapter 11 FCoE Configuration covers configuring Linux FCoE offload.
Chapter 12 SR-IOV Configuration provides procedures for configuring single
root input/output virtualization (SR-IOV) on Windows, Linux, and VMware
systems.
Chapter 13 NVMe-oF Configuration with RDMA demonstrates how to
configure NVMe-oF on a simple network for 41xxx Series Adapters.
Chapter 14 Windows Server 2016 describes the Windows Server 2016
features.

Preface
Documentation Conventions
xx AH0054602-00 J
Chapter 15 Windows Server 2019 describes the Windows Server 2019
features.
Chapter 16 Troubleshooting describes a variety of troubleshooting methods
and resources.
Appendix A Adapter LEDS lists the adapter LEDs and their significance.
Appendix B Cables and Optical Modules lists the cables, optical modules,
and switches that the 41xxx Series Adapters support.
Appendix C Dell Z9100 Switch Configuration describes how to configure the
Dell Z9100 switch port for 25Gbps.
Appendix D Feature Constraints provides information about feature
constraints implemented in the current release.
At the end of this guide is a glossary of terms.
Documentation Conventions
This guide uses the following documentation conventions:
 provides additional information.
 without an alert symbol indicates the presence of a hazard
that could cause damage to equipment or loss of data.
 with an alert symbol indicates the presence of a hazard that
could cause minor or moderate injury.
 indicates the presence of a hazard that could cause serious
injury or death.
Text in blue font indicates a hyperlink (jump) to a figure, table, or section in
this guide, and links to Web sites are shown in underlined blue. For
example:
Table 9-2 lists problems related to the user interface and remote agent.
See “Installation Checklist” on page 6.
For more information, visit www.marvell.com.
Text in bold font indicates user interface elements such as a menu items,
buttons, check boxes, or column headings. For example:
Click the Start button, point to Programs, point to Accessories, and
then click Command Prompt.
Under Notification Options, select the Warning Alarms check box.
NOTE
CAUTION
CAUTION
!
!
WARNING