Cavium FastLinQ 45000 Series User manual

BC0154501-00 P
User’s Guide
Converged Network Adapters and
Intelligent Ethernet Adapters
FastLinQ 45000 Series

ii BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
Document Revision History
Revision A, April 19, 2016
Revision B, April 26, 2016
Revision C, May 11, 2016
Revision D, August 29, 2016
Revision E, August 30, 2016
Revision F, September 30, 2016
Revision G, March 10, 2017
Revision H, June 26, 2017
Revision J, November 21, 2017
Revision K, February 7, 2018
Revision L, May 22, 2018
Revision M, June 18, 2018
Revision N, August 24, 2018
Revision P, January 17, 2019
Changes Sections Affected
Updated the Cavium logos, preface content, and
copyright information.
Front cover, Preface, and back page
Removed references to Windows Nano Server. “System Requirements” on page 7 (Table 2-2)
Chapter 3 Driver Installation
Chapter 4 Firmware Upgrade Utility
Chapter 14 Windows Server 2016
Before Table 3-4, added “Cavium has certified
qedrntv for ESXi 6.5 and ESXi 6.7, which are bun-
dled along with qedentv as one package in vCG
listings.”
“VMware Drivers and Driver Packages” on
page 24
Updated the Linux firmware update procedure. “Upgrading Adapter Firmware on Linux” on
page 31
Updated the Windows firmware update procedure. “Upgrading Adapter Firmware on Windows” on
page 31
Added L2B Firmware Version as a read-only item
on the Firmware Information page.
“Displaying Firmware Information” on page 38
For device-level configuration, added the Port
Mode parameter to the bullet list.
“Configuring Device-level Parameters” on page 38

iii BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
For port-level configuration:
Updated Figure 5-7.
In Step 4, updated the RoCE Priority descrip-
tion.
In Step 10, added iWARP+RoCE as a new
option.
“Configuring Port-level Parameters” on page 41
Added a footnote referencing the QConvergeCon-
sole Help system for procedures for changing the
adapter port mode in QConvergeConsole GUI.
“Changing the Adapter Port Mode” on page 53
Moved iSCSI boot from SAN information from the
iSCSI Configuration chapter to a new chapter.
Chapter 6 Boot from SAN Configuration
Moved FCoE boot from SAN information from the
FCoE Configuration chapter to a new chapter.
Chapter 6 Boot from SAN Configuration
In the VMware bullet, expanded the information to
explain the limitations of iSCSI boot in the ESXi
OS.
“Supported Operating Systems for iSCSI Boot” on
page 60
Following Step 1 of the To install SLES 12 SP3
and later procedure, added this note: “Although
they are not necessarily required for iSCSI boot
from SAN for SUSE, Cavium recommends that
you also complete Step 2 through Step 15 of the
Configuring iSCSI Boot from SAN for RHEL 7.5
and Later procedure”
“Configuring iSCSI Boot from SAN for SLES 12
SP3 and Later” on page 91
In Step 4 of the To install RHEL 7.4 and later pro-
cedure, changed the command to inst.dd
modprobe.blacklist=qed,qede,qedr,qedi
,qedf.
“Configuring FCoE Boot from SAN for RHEL 7.4
and Later” on page 120
In Table 7-1 for OFED:
Removed support for RHEL 7.4, SLES 12 SP2.
Added support for RHEL 7.6, SLES 12 SP4.
Replaced CentOS 7.4 with CentOS 7.6.
Changed Ubuntu 16.04 LTS support for RoCE
v2 to include “(4.5 + kernel)”.
“Supported Operating Systems and OFED” on
page 127
In Table 7-2, replaced the RDMA Mode property
with the NetworkDirect Technology prop-
erty.
“Configuring RoCE on the Adapter for Windows
Server” on page 135

iv BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
Updated the following iWARP procedures:
In Step 1 c of the To enable iWARP on the
Windows host and verify RDMA procedure,
changed the parameter to NetworkDirect
Technology (was RDMA Mode).
In Step 4 b of the To verify iWARP traffic pro-
cedure, changed the command to netstat
-xan (was net -xan).
“Configuring iWARP on Windows” on page 169
In the To configure the NFS server procedure:
Added a new Step 1 that describes how to
create an nfs-server directory and grant
permission.
Replaced Step 4 for loading the service.
“Configuring NFS-RDMA” on page 175
In the To configure the NFS client procedure:
Added a new Step 1 that describes how to
create an nfs-client directory and grant
permission.
In Step 3, updated the command used for NFS
Version 4.
“Configuring NFS-RDMA” on page 175
Following Step 12, added a fourth bullet to the
note: “Switch dependent teaming (IEEE 802.3ad
LACP and Generic/Static Link Aggregation (Trunk-
ing) cannot use a switch independent partitioned
virtual adapter. IEEE standards require Switch
Dependent Teaming (IEEE 802.3ad LACP and
Generic/Static Link Aggregation (Trunking)) mode
to work per the entire port instead of just the MAC
address (fraction of a port) granularity.”
“Configuring Microsoft Initiator to Use Cavium’s
iSCSI Offload” on page 197
Added new section addressing an issue with iSCSI
offload on RHEL 7.6.
“” on page 208
In the note, added the following: “For NDKPI
Mode-2, you must enable Network Direct Function-
ality of the physical interface.”
“Configuring RoCE Interfaces with Hyper-V” on
page 241

v BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
In the To create a Hyper-V virtual switch with an
RDMA NIC procedure:
Changed the section title to “...RDMA NIC” (was
“...RDMA Virtual NIC”.
Changed procedure title (was To create a
Hyper-V virtual switch with an RDMA virtual
NIC),
Added a new Step 1: “On all physical interfaces,
set the value of the NetworkDirect Functional-
ity parameter to Enabled.”
“Creating a Hyper-V Virtual Switch with an RDMA
NIC” on page 242
In the To add host virtual NICs procedure:
In Step 2, corrected the last digit of the com-
mand to “S” (was “5”).
Changed the note’s first bullet, second sen-
tence to: “The same VLAN ID must be assigned
to ports on the switch” (was: “The same VLAN
ID must be assigned to all the interfaces, and
on the switch”).
“Adding a vLAN ID to Host Virtual NIC” on
page 243
Added terms and definitions. Glossary

vi BC0154501-00 P
Table of Contents
Preface
Supported Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
What Is in This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Related Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx
Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi
Technical Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
Downloading Updates and Documentation . . . . . . . . . . . . . . . . . . . . . xxiii
Knowledgebase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
Laser Safety—FDA Notice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv
Agency Certification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv
EMI and EMC Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv
KCC: Class A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvi
VCCI: Class A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvi
Product Safety Compliance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvi
1 Product Overview
Functional Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Adapter Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
QLogic Control Suite CLI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
QLogic QConvergeConsole GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
QLogic QConvergeConsole vCenter Plug-in. . . . . . . . . . . . . . . . . . . . 4
QConvergeConsole PowerKit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
FastLinQ ESXCLI VMware Plug-in . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Adapter Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Physical Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Standards Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Hardware Installation
System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Preinstallation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Installing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

vii BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
3 Driver Installation
Installing Linux Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Installing the Linux Drivers Without RDMA . . . . . . . . . . . . . . . . . . . . . 13
Removing the Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Installing Linux Drivers Using the src RPM Package . . . . . . . . . 15
Installing Linux Drivers Using the kmp/kmod RPM Package . . . 16
Installing Ubuntu Linux Drivers. . . . . . . . . . . . . . . . . . . . . . . . . . 16
Installing Linux Drivers Using the TAR File. . . . . . . . . . . . . . . . . 16
Installing the Linux Drivers with RDMA . . . . . . . . . . . . . . . . . . . . . . . . 17
Linux Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Linux Driver Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Importing a Public Key for Secure Boot. . . . . . . . . . . . . . . . . . . . . . . . 19
Installing Windows Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Installing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Removing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Managing Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Setting Power Management Options. . . . . . . . . . . . . . . . . . . . . . . . . . 23
Installing VMware Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
VMware Drivers and Driver Packages. . . . . . . . . . . . . . . . . . . . . . . . . 24
Installing VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
VMware NIC Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . 26
VMware Driver Parameter Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Removing the VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
FCoE Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
iSCSI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4 Firmware Upgrade Utility
Image Verification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Upgrading Adapter Firmware on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Upgrading Adapter Firmware on Windows. . . . . . . . . . . . . . . . . . . . . . . . . . 31
Converting a 100G Adapter to Four-port 25G Adapter. . . . . . . . . . . . . . . . . 32
5 Adapter Preboot Configuration
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Displaying Firmware Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Configuring Device-level Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Configuring Port-level Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Configuring FCoE Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

viii BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
Configuring iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Configuring Partitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Changing the Adapter Port Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Converting a 100G Adapter to Four-port 25G Adapter . . . . . . . . . . . . 53
Converting a 25G Adapter to Single-port 100G Adapter . . . . . . . . . . . 54
Converting a 40G Adapter to a Four-port 10G Adapter. . . . . . . . . . . . 55
Converting a 10G Adapter to a Four-port 40G Adapter. . . . . . . . . . . . 56
6 Boot from SAN Configuration
iSCSI Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
iSCSI Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 59
iSCSI Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Supported Operating Systems for iSCSI Boot . . . . . . . . . . . . . . 60
Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 61
Selecting the iSCSI UEFI Boot Protocol. . . . . . . . . . . . . . . . . . . 64
Configuring the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Configuring iSCSI Boot Parameters . . . . . . . . . . . . . . . . . . . . . . 65
Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 65
Configuring iSCSI Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . 67
Configuring the DHCP Server to Support iSCSI Boot . . . . . . . . 77
Configuring iSCSI Boot from SAN on Windows . . . . . . . . . . . . . . . . . 81
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Selecting the Preferred iSCSI Boot Mode . . . . . . . . . . . . . . . . . 82
Configuring iSCSI General Parameters . . . . . . . . . . . . . . . . . . . 82
Configuring the iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Configuring the iSCSI Targets . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Detecting the iSCSI LUN and Injecting the QLogic Drivers . . . .84
Configuring iSCSI Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 86
Configuring iSCSI Boot from SAN for RHEL 7.4 . . . . . . . . . . . . 86
Configuring iSCSI Boot from SAN for RHEL 7.5 and Later . . . . 90
Configuring iSCSI Boot from SAN for SLES 12 SP3
and Later . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Configuring iSCSI Boot from SAN for Other
Linux Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Configuring iSCSI Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 107
FCoE Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
FCoE Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 107
FCoE Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Specifying the BIOS Boot Protocol. . . . . . . . . . . . . . . . . . . . . . . 108
Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 109

ix BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
Configuring FCoE Boot from SAN on Windows . . . . . . . . . . . . . . . . . 114
Windows Server 2012 R2 and 2016 FCoE Boot Installation . . . 114
Configuring FCoE on Windows . . . . . . . . . . . . . . . . . . . . . . . . . 115
FCoE Crash Dump on Windows. . . . . . . . . . . . . . . . . . . . . . . . . 115
Injecting (Slipstreaming) Adapter Drivers into Windows
Image Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Configuring FCoE Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 117
Prerequisites for Linux FCoE Boot from SAN. . . . . . . . . . . . . . . 117
Configuring Linux FCoE Boot from SAN . . . . . . . . . . . . . . . . . . 118
Configuring FCoE Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 123
Injecting (Slipstreaming) ESXi Adapter Drivers into
Image Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Installing the Customized ESXi ISO . . . . . . . . . . . . . . . . . . . . . . 124
7 RoCE Configuration
Supported Operating Systems and OFED . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Planning for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
Preparing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Preparing the Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Configuring the Cisco Nexus 6000 Ethernet Switch . . . . . . . . . . . . . . 129
Configuring the Dell Z9100 Ethernet Switch . . . . . . . . . . . . . . . . . . . . 131
Configuring the Arista 7060X Ethernet Switch. . . . . . . . . . . . . . . . . . . 133
Configuring RoCE on the Adapter for Windows Server . . . . . . . . . . . . . . . . 135
Viewing RDMA Counters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Configuring RoCE on the Adapter for Linux . . . . . . . . . . . . . . . . . . . . . . . . . 143
RoCE Configuration for RHEL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
RoCE Configuration for SLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
RoCE Configuration for Ubuntu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Verifying the RoCE Configuration on Linux . . . . . . . . . . . . . . . . . . . . . 147
vLAN Interfaces and GID Index Values. . . . . . . . . . . . . . . . . . . . . . . . 150
RoCE v2 Configuration for Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Identifying the RoCE v2 GID Index or Address . . . . . . . . . . . . . 151
Verifying the RoCE v1 or RoCE v2 GID Index and Address
from sys and class Parameters . . . . . . . . . . . . . . . . . . . . . . . . 151
Verifying the RoCE v1 or RoCE v2 Function Through
perftest Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Configuring RoCE on the Adapter for VMware ESX . . . . . . . . . . . . . . . . . . 156
Configuring RDMA Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Configuring MTU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
RoCE Mode and Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Configuring a Paravirtual RDMA Device (PVRDMA). . . . . . . . . . . . . . 159

x BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
Configuring DCQCN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
DCQCN Terminology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
DCQCN Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
DCB-related Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Global Settings on RDMA Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Setting vLAN Priority on RDMA Traffic . . . . . . . . . . . . . . . . . . . . 164
Setting ECN on RDMA Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Setting DSCP on RDMA Traffic . . . . . . . . . . . . . . . . . . . . . . . . . 165
Configuring DSCP-PFC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Enabling DCQCN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Configuring CNP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
DCQCN Algorithm Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
MAC Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Script Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
8 iWARP Configuration
Preparing the Adapter for iWARP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Configuring iWARP on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Configuring iWARP on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Installing the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Configuring iWARP and RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Detecting the Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Supported iWARP Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Running Perftest for iWARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Configuring NFS-RDMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
iWARP RDMA-Core Support on SLES 12 SP3 and RHEL 7.4 . . . . . . 177
9 iSER Configuration
Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Configuring iSER for RHEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Configuring iSER for SLES 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Configuring iSER for Ubuntu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Configuring LIO as a Target. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Configuring the Initiator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
Optimizing Linux Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Configuring CPUs to Maximum Performance Mode . . . . . . . . . . . . . . 191
Configuring Kernel sysctl Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Configuring IRQ Affinity Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Configuring Block Device Staging . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

xi BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
Configuring iSER on ESXi 6.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Before You Begin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Configuring iSER for ESXi 6.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
10 iSCSI Configuration
iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
iSCSI Offload in Windows Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Installing Cavium QLogic Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Installing the Microsoft iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . 197
Configuring Microsoft Initiator to Use Cavium’s iSCSI Offload . . . . . . 197
iSCSI Offload FAQs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Windows Server 2012 R2, 2016, and 2019 iSCSI Boot Installation . . 204
iSCSI Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
iSCSI Offload in Linux Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Differences from bnx2i. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Configuring qedi.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Verifying iSCSI Interfaces in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
11 FCoE Configuration
Configuring Linux FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Differences Between qedf and bnx2fc. . . . . . . . . . . . . . . . . . . . . . . . . 210
Configuring qedf.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Verifying FCoE Devices in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
12 SR-IOV Configuration
Configuring SR-IOV on Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Configuring SR-IOV on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Configuring SR-IOV on VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
13 NVMe-oF Configuration with RDMA
Installing Device Drivers on Both Servers . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Configuring the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Configuring the Initiator Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
Preconditioning the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Testing the NVMe-oF Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Optimizing Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
.IRQ Affinity (multi_rss-affin.sh) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
CPU Frequency (cpufreq.sh). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239

xii BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
14 Windows Server 2016
Configuring RoCE Interfaces with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . 241
Creating a Hyper-V Virtual Switch with an RDMA NIC . . . . . . . . . . . . 242
Adding a vLAN ID to Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . 243
Verifying If RoCE is Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Adding Host Virtual NICs (Virtual Ports) . . . . . . . . . . . . . . . . . . . . . . . 245
Mapping the SMB Drive and Running RoCE Traffic . . . . . . . . . . . . . . 245
RoCE over Switch Embedded Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Creating a Hyper-V Virtual Switch with SET and RDMA
Virtual NICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Enabling RDMA on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Assigning a vLAN ID on SET. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
Running RDMA Traffic on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
Configuring QoS for RoCE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
Configuring QoS by Disabling DCBX on the Adapter . . . . . . . . . . . . . 249
Configuring QoS by Enabling DCBX on the Adapter. . . . . . . . . . . . . . 253
Configuring VMMQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Enabling VMMQ on the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Creating a Virtual Machine Switch with or Without SR-IOV . . . . . . . . 258
Enabling VMMQ on the Virtual Machine Switch . . . . . . . . . . . . . . . . . 259
Getting the Virtual Machine Switch Capability. . . . . . . . . . . . . . . . . . . 260
Creating a VM and Enabling VMMQ on VMNetworkadapters
in the VM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Default and Maximum VMMQ Virtual NIC . . . . . . . . . . . . . . . . . . . . . . 262
Enabling and Disabling VMMQ on a Management NIC . . . . . . . . . . . 262
Monitoring Traffic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Configuring VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Enabling VXLAN Offload on the Adapter. . . . . . . . . . . . . . . . . . . . . . . 263
Deploying a Software Defined Network. . . . . . . . . . . . . . . . . . . . . . . . 263
Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Configuring the Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Deploying a Hyper-Converged System . . . . . . . . . . . . . . . . . . . . . . . . 265
Deploying the Operating System . . . . . . . . . . . . . . . . . . . . . . . . 265
Configuring the Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . 267
15 Traffic Control Offload
Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Current Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

xiii BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
Ingress Packet Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Ingress Packet Redirection for SR-IOV VFs . . . . . . . . . . . . . . . . . . . . 272
Ingress Packet Redirection with MAC-vLAN Offloaded
Devices and SR-IOV VFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
TC Drop Action Support (Ingress Drop) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
TC MQPRIO Offload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
16 Troubleshooting
Troubleshooting Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Verifying that Current Drivers Are Loaded . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Verifying Drivers in Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Verifying Drivers in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Verifying Drivers in VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Testing Network Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Testing Network Connectivity for Windows . . . . . . . . . . . . . . . . . . . . . 276
Testing Network Connectivity for Linux . . . . . . . . . . . . . . . . . . . . . . . . 277
Microsoft Virtualization with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Linux-specific Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Miscellaneous Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Troubleshooting Windows FCoE and iSCSI Boot from SAN . . . . . . . . . . . . 278
Collecting Debug Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
AAdapter LEDS
B Cables and Optical Modules
Supported Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
Tested Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
C Feature Constraints
Glossary
Index

xiv BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
List of Figures
Figure Page
3-1 Setting Advanced Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3-2 Power Management Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
5-1 System Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5-2 System Setup: Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5-3 Main Configuration Page, Setting Default Partitioning Mode. . . . . . . . . . . . . . . . . . 35
5-4 Main Configuration Page, Setting NPAR Partitioning Mode. . . . . . . . . . . . . . . . . . . 36
5-5 Firmware Information Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5-6 Device Level Configuration Page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5-7 Port Level Configuration Page: Setting Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . 42
5-8 FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5-9 FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5-10 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5-11 iSCSI Initiator Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5-12 iSCSI First Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5-13 iSCSI Second Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5-14 Partitions Configuration Page (No FCoE Offload or iSCSI Offload). . . . . . . . . . . . . 48
5-15 Partitions Configuration Page (with FCoE Offload and iSCSI Offload) . . . . . . . . . . 48
5-16 Global Bandwidth Allocation Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5-17 Partition 1 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5-18 Partition 2 Configuration: FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5-19 Partition 3 Configuration: iSCSI Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5-20 Partition 4 Configuration: Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5-21 Changing the Port Mode to 4x25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5-22 Changing the Port Mode to 1x100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5-23 Changing the Port Mode to 4x10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5-24 Changing the Port Mode to 1x40 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6-1 Systems Utilities at Boot Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
6-2 Configuration Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6-3 Selecting Port Level Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6-4 Port Level Configuration, Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6-5 Port Level Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
6-6 Systems Utilities at Boot Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6-7 Selecting Port Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6-8 Port Level Configuration, Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6-9 Selecting iSCSI Boot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6-10 Selecting General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6-11 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6-12 Selecting iSCSI Initiator Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
6-13 iSCSI Initiator Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6-14 iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6-15 iSCSI First Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
6-16 iSCSI Second Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
6-17 Saving iSCSI Changes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

xv BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
6-18 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6-19 iSCSI Initiator Configuration, VLAN ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6-20 Detecting the iSCSI LUN Using UEFI Shell (Version 2). . . . . . . . . . . . . . . . . . . . . . 84
6-21 Windows Setup: Selecting Installation Destination . . . . . . . . . . . . . . . . . . . . . . . . . 85
6-22 Windows Setup: Selecting Driver to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6-23 Prompt for Out-of-Box Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6-24 Red Hat Enterprise Linux 7.4 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6-25 System Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6-26 System Configuration, Port Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6-27 Port Level Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6-28 Boot Mode in Port Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6-29 FCoE Offload Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6-30 Selecting FCoE Boot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6-31 Selecting General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6-32 FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6-33 FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6-34 Selecting the Driver Disk Image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6-35 Selecting the Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6-36 Prompt for Out-of-Box Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6-37 Red Hat Enterprise Linux 7.4 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6-38 ESXi-Customizer Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
6-39 Select a VMware Disk to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6-40 VMware Generic USB Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7-1 Configuring RoCE Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7-2 Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7-3 Performance Monitor: Cavium FastLinQ Counters . . . . . . . . . . . . . . . . . . . . . . . . . 140
7-4 Switch Settings, Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
7-5 Switch Settings, Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
7-6 Configuring RDMA_CM Applications: Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
7-7 Configuring RDMA_CM Applications: Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
7-8 Configuring a New Distributed Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
7-9 Assigning a vmknic for PVRDMA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
7-10 Setting the Firewall Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
8-1 Windows PowerShell Command: Get-NetAdapterRdma. . . . . . . . . . . . . . . . . . . . . 169
8-2 Windows PowerShell Command: Get-NetOffloadGlobalSetting . . . . . . . . . . . . . . . 170
8-3 Perfmon: Add Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
8-4 Perfmon: Verifying iWARP Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
9-1 RDMA Ping Successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9-2 iSER Portal Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9-3 Iface Transport Confirmed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
9-4 Checking for New iSCSI Device. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
10-1 iSCSI Initiator Properties, Configuration Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
10-2 iSCSI Initiator Node Name Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
10-3 iSCSI Initiator—Discover Target Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
10-4 Target Portal IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

xvi BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
10-5 Selecting the Initiator IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
10-6 Connecting to the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
10-7 Connect To Target Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
12-1 Device Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
12-2 Adapter Properties, Advanced: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . 215
12-3 Virtual Switch Manager: Enabling SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
12-4 Settings for VM: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
12-5 Device Manager: VM with QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
12-6 Windows PowerShell Command: Get-NetadapterSriovVf . . . . . . . . . . . . . . . . . . . . 219
12-7 Integrated Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
12-8 Editing the grub.conf File for SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
12-9 Command Output for sriov_numvfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
12-10 Command Output for ip link show Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
12-11 RHEL68 Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
12-12 Add New Virtual Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
12-13 VMware Host Edit Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
13-1 NVMe-oF Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
13-2 Subsystem NQN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
13-3 Confirm NVMe-oF Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
13-4 FIO Utility Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
14-1 Enabling RDMA in Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
14-2 Hyper-V Virtual Ethernet Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
14-3 Windows PowerShell Command: Get-VMNetworkAdapter . . . . . . . . . . . . . . . . . . . 244
14-4 Windows PowerShell Command: Get-NetAdapterRdma . . . . . . . . . . . . . . . . . . . . . 244
14-5 Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
14-6 Performance Monitor Shows RoCE Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
14-7 Windows PowerShell Command: New-VMSwitch . . . . . . . . . . . . . . . . . . . . . . . . . . 247
14-8 Windows PowerShell Command: Get-NetAdapter. . . . . . . . . . . . . . . . . . . . . . . . . . 248
14-9 Advanced Properties: Enable QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
14-10 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
14-11 Advanced Properties: Enabling QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
14-12 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
14-13 Advanced Properties: Enabling Virtual Switch RSS. . . . . . . . . . . . . . . . . . . . . . . . . 258
14-14 Virtual Switch Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
14-15 Windows PowerShell Command: Get-VMSwitch. . . . . . . . . . . . . . . . . . . . . . . . . . . 260
14-16 Advanced Properties: Enabling VXLAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
14-17 Example Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
16-1 Windows Setup Error Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278

xvii BC0154501-00 P
User’s Guide—Converged Network Adapters and Intelligent Ethernet Adapters
FastLinQ 45000 Series
List of Tables
Table Page
2-1 Host Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2-2 Minimum Host Operating System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3-1 Cavium QLogic 45000 Series Adapters Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . 12
3-2 qede Driver Optional Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3-3 Linux Driver Operation Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3-4 VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3-5 ESXi Driver Packages by Release. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3-6 VMware NIC Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3-7 VMware Driver Parameter Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5-1 Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6-1 iSCSI Out-of-Box and Inbox Boot from SAN Support . . . . . . . . . . . . . . . . . . . . . . . 59
6-2 iSCSI General Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
6-3 DHCP Option 17 Parameter Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6-4 DHCP Option 43 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6-5 DHCP Option 17 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6-6 FCoE Out-of-Box and Inbox Boot from SAN Support . . . . . . . . . . . . . . . . . . . . . . . 107
7-1 OS Support for RoCE v1, RoCE v2, iWARP, iSER, and OFED . . . . . . . . . . . . . . . . 127
7-2 Advanced Properties for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7-3 Cavium FastLinQ RDMA Error Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7-4 DCQCN Algorithm Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
13-1 Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
16-1 Collecting Debug Data Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
A-1 Adapter Port Link and Activity LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
B-1 Tested Cables and Optical Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
B-2 Switches Tested for Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

xviii BC0154501-00 P
Preface
This preface lists the supported products, specifies the intended audience,
explains the typographic conventions used in this guide, lists related documents,
provides technical support information, and describes legal notices.
Supported Products
This user’s guide describes the following Cavium FastLinQ products:
25Gb Intelligent Ethernet Adapters:
QL45211HLCU-BK/SP/CK
QL45212HLCU-BK/SP/CK
40Gb Intelligent Ethernet Adapters:
QL45411HLCU-BK/SP/CK
QL45412HLCU-BK/SP/CK
40Gb Converged Network Adapter1:
QL45462HLCU-BK/SP/CK
100Gb Intelligent Ethernet Adapters:
QL45611HLCU-BK/SP/CK
QL45631HOCU-BK/SP/CK
NOTE
QConvergeConsole® (QCC) GUI is the only GUI management tool across
all Cavium® QLogic® adapters. QLogic Control Suite™ (QCS) GUI is no
longer supported for the FastLinQ® 45000 Series Adapters and adapters
based on 57xx/57xxx controllers, and has been replaced by the QCC GUI
management tool. The QCC GUI provides single-pane-of-glass GUI
management for all Cavium adapters.
In Windows® environments, when you run QCS CLI and Management
Agents Installer, it will uninstall the QCS GUI (if installed on the system) and
any related components from your system. To obtain the new GUI, download
QCC GUI for your adapter from the Marvell Web site (see “Downloading
Updates and Documentation” on page xxiii).
1Windows and Linux only. VMware support for FCoE and iSCSI offload is coming soon.

Preface
Intended Audience
xix BC0154501-00 P
Intended Audience
This guide is intended for system administrators and other technical staff
members responsible for configuring and managing adapters installed on servers
in Windows, Linux®, or VMware® environments.
What Is in This Guide
Following this preface, the remainder of this guide is organized into the following
chapters and appendices:
Chapter 1 Product Overview provides a product functional description, a list
of features, adapter management tool descriptions, and the adapter
specifications.
Chapter 2 Hardware Installation describes how to install the adapter,
including the list of system requirements and a preinstallation checklist.
Chapter 3 Driver Installation describes the installation of the adapter drivers
on Windows, Linux, and VMware.
Chapter 4 Firmware Upgrade Utility describes the use of the utility to
upgrade adapter firmware and boot code.
Chapter 5 Adapter Preboot Configuration describes the preboot adapter
configuration tasks using the Human Infrastructure Interface (HII)
application.
Chapter 6 Boot from SAN Configuration covers boot from SAN configuration
for both iSCSI and FCoE.
Chapter 7 RoCE Configuration describes how to configure the adapter, the
Ethernet switch, and the host to use RDMA over converged Ethernet
(RoCE).
Chapter 8 iWARP Configuration provides procedures for configuring Internet
wide area RDMA protocol (iWARP) on Windows, Linux, and VMware
ESXi 6.7 systems.
Chapter 9 iSER Configuration describes how to configure iSCSI Extensions
for RDMA (iSER) for Linux RHEL, SLES, Ubuntu, and ESXi 6.7.
Chapter 10 iSCSI Configuration describes iSCSI boot and iSCSI offload for
Windows and Linux.
Chapter 11 FCoE Configuration covers configuring Linux FCoE offload.
Chapter 12 SR-IOV Configuration provides procedures for configuring single
root input/output virtualization (SR-IOV) on Windows, Linux, and VMware
systems.

Preface
Related Materials
xx BC0154501-00 P
Chapter 13 NVMe-oF Configuration with RDMA demonstrates how to
configure NVMe-oF on a simple network for 45000 Series Adapters.
Chapter 14 Windows Server 2016 describes the Windows Server 2016
features.
Chapter 15 Traffic Control Offload describes the traffic control (TC) offload
feature.
Chapter 16 Troubleshooting describes a variety of troubleshooting methods
and resources.
Appendix A Adapter LEDS lists the adapter LEDs and their significance.
Appendix B Cables and Optical Modules lists the cables, optical modules,
and switches that the 45000 Series Adapters support.
Appendix C Feature Constraints provides information about feature
constraints implemented in the current release.
At the end of this guide are a glossary of terms and an index to help you quickly
locate the information you need.
Related Materials
For additional information, refer to the following documents that are available on
the Marvell Web Site:
Installation Guide—QConvergeConsole GUI (part number SN0051105-00)
contains detailed information on how to install and use the
QConvergeConsole GUI management tool.
User's Guide—QLogic Control Suite CLI (part number BC0054511-00)
contains detailed information on how to install, start, and use QLogic Control
Suite CLI.
User's Guide—PowerShell (part number BC0054518-00) contains detailed
information on how to install QConvergeConsole FastLinQ PowerKit to
manage the FastLinQ adapters on the system.
User’s Guide—QConvergeConsole Plug-ins for vSphere (part number
SN0054677-00) provides details for using the two QCC plug-ins to extend
the capabilities of VMware vCenter Server and the vSphere Web Client.
User’s Guide—FastLinQ ESXCLI VMware Plug-in (part number
BC0151101-00) describes the plug-in that extends the capabilities of the
ESX® CLI to manage Cavium QLogic 3400, 8400, 41000, and 45000 Series
Adapters installed in VMware ESX/ESXi hosts.
For information about downloading documentation from the Marvell Web site, see
“Downloading Updates and Documentation” on page xxiii.