Broadcom NetXreme-C User manual

NetXtreme-E-UG100
February 26, 2018
User’sManual
NetXtreme-E
Broadcom®NetXreme-C and NetXtreme-E
USER’S MANUAL

© 2018 by Broadcom. All rights reserved.
Broadcom®, the pulse logo, Connecting everything®, Avago Technologies, and the A logo are among the
trademarks of Broadcom and/or its affiliates in the United States, certain other countries and/or the EU. The
term “Broadcom” refers to Broadcom Limited and/or its subsidiaries. For more information, please visit
www.broadcom.com.
Broadcom reserves the right to make changes without further notice to any products or data herein to improve
reliability, function, or design. Information furnished by Broadcom is believed to be accurate and reliable.
However, Broadcom does not assume any liability arising out of the application or use of this information, nor
the application or use of any product or circuit described herein, neither does it convey any license under its
patent rights nor the rights of others.
Revision History
Revision Date Change Description
NetXtreme-E-UG100 2/26/18 Initial release for 20.6.

Table of ContentsNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 3
Table of Contents
Regulatory and Safety Approvals...............................................................................................................7
Regulatory...............................................................................................................................................7
Safety......................................................................................................................................................7
Electromagnetic Compatibility (EMC)......................................................................................................8
Electrostatic Discharge (ESD) Compliance.............................................................................................8
FCC Statement........................................................................................................................................8
Functional Description.................................................................................................................................9
Network Link and Activity Indication........................................................................................................14
BCM957402AXXXX/BCM957412AXXXX .............................................................................................14
BCM957404AXXXX/BCM957414AXXXX .............................................................................................15
BCM957406AXXXX/BCM957416AXXXX .............................................................................................16
BCM957414M4140D.............................................................................................................................17
BCM957412M4120D.............................................................................................................................18
BCM957416M4160 ...............................................................................................................................19
Features.......................................................................................................................................................20
Software and Hardware Features .........................................................................................................20
Virtualization Features...........................................................................................................................21
VXLAN...................................................................................................................................................22
NVGRE/GRE/IP-in-IP/Geneve..............................................................................................................22
Stateless Offloads.................................................................................................................................22
RSS................................................................................................................................................22
TPA................................................................................................................................................22
Header-Payload Split.....................................................................................................................22
UDP Fragmentation Offload..................................................................................................................22
Stateless Transport Tunnel Offload.......................................................................................................23
Multiqueue Support for OS....................................................................................................................23
NDIS VMQ .....................................................................................................................................23
VMWare NetQueue........................................................................................................................23
KVM/Xen Multiqueue .....................................................................................................................23
SR-IOV Configuration Support Matrix...................................................................................................23
SR-IOV..................................................................................................................................................24
Network Partitioning (NPAR).................................................................................................................24
RDMA over Converge Ethernet – RoCE...............................................................................................24
Supported Combinations.......................................................................................................................25
NPAR, SR-IOV, and RoCE............................................................................................................25
NPAR, SR-IOV, and DPDK............................................................................................................25
Unsupported Combinations............................................................................................................25

Table of ContentsNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 4
Installing the Hardware..............................................................................................................................26
Safety Precautions................................................................................................................................26
System Requirements...........................................................................................................................26
Hardware Requirements................................................................................................................26
Preinstallation Checklist.................................................................................................................26
Installing the Adapter.............................................................................................................................27
Connecting the Network Cables............................................................................................................27
Supported Cables and Modules.....................................................................................................27
Copper ...........................................................................................................................................28
SFP+..............................................................................................................................................28
SFP28............................................................................................................................................28
Software Packages and Installation .........................................................................................................29
Supported Operating Systems..............................................................................................................29
Installing Drivers....................................................................................................................................29
Windows.........................................................................................................................................29
Dell DUP ................................................................................................................................29
GUI Install ..............................................................................................................................29
Silent Install............................................................................................................................29
INF Install...............................................................................................................................29
Linux...............................................................................................................................................30
Module Install.........................................................................................................................30
Linux Ethtool Commands.......................................................................................................31
VMware..........................................................................................................................................32
Firmware Update...................................................................................................................................33
Dell Update Package .....................................................................................................................33
Windows.........................................................................................................................................33
Linux...............................................................................................................................................33
Windows Driver Advanced Properties and Event Log Messages .........................................................34
Driver Advanced Properties ..................................................................................................................34
Event Log Messages.............................................................................................................................35
Teaming.......................................................................................................................................................37
Windows................................................................................................................................................37
Linux......................................................................................................................................................37
System-level Configuration.......................................................................................................................38
UEFI HII Menu.......................................................................................................................................38
Main Configuration Page................................................................................................................38
Firmware Image Properties............................................................................................................38
Device Level Configuration............................................................................................................38
NIC Configuration...........................................................................................................................38

Table of ContentsNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 5
iSCSI Configuration........................................................................................................................38
Comprehensive Configuration Management.........................................................................................39
Device Hardware Configuration.....................................................................................................39
MBA Configuration Menu...............................................................................................................39
iSCSI Boot Main Menu...................................................................................................................39
Auto-Negotiation Configuration.............................................................................................................40
Operational Link Speed..........................................................................................................43
Firmware Link Speed.............................................................................................................43
Auto-negotiation Protocol.......................................................................................................43
Windows Driver Settings........................................................................................................43
Linux Driver Settings..............................................................................................................43
ESXi Driver Settings...............................................................................................................44
FEC Auto-Negotiation ...........................................................................................................................45
Link Training..........................................................................................................................................46
Media Auto Detect.................................................................................................................................47
ISCSI Boot...................................................................................................................................................49
Supported Operating Systems for iSCSI Boot ......................................................................................49
Setting up iSCSI Boot............................................................................................................................49
Configuring the iSCSI Target.........................................................................................................49
Configuring iSCSI Boot Parameters...............................................................................................50
MBA Boot Protocol Configuration ..................................................................................................50
iSCSI Boot Configuration...............................................................................................................51
Static iSCSI Boot Configuration .............................................................................................51
Dynamic iSCSI Boot Configuration........................................................................................52
Enabling CHAP Authentication ......................................................................................................53
Configuring the DHCP Server to Support iSCSI Boot...........................................................................54
DHCP iSCSI Boot Configurations for IPv4.....................................................................................54
DHCP Option 17, Root Path ..................................................................................................54
DHCP Option 43, Vendor-Specific Information......................................................................54
Configuring the DHCP Server................................................................................................55
DHCP iSCSI Boot Configuration for IPv6.......................................................................................55
DHCPv6 Option 16, Vendor Class Option .............................................................................55
DHCPv6 Option 17, Vendor-Specific Information ..................................................................55
Configuring the DHCP Server................................................................................................56
VXLAN: Configuration and Use Case Examples.....................................................................................56
SR-IOV: Configuration and Use Case Examples.....................................................................................57
Linux Use Case.....................................................................................................................................57
Windows Case ......................................................................................................................................58
VMWare SRIOV Case...........................................................................................................................59

Table of ContentsNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 6
NPAR – Configuration and Use Case Example .......................................................................................62
Features and Requirements..................................................................................................................62
Limitations.............................................................................................................................................62
Configuration.........................................................................................................................................62
Notes on Reducing NIC Memory Consumption ....................................................................................65
RoCE – Configuration and Use Case Examples......................................................................................66
Linux Configuration ...............................................................................................................................66
Requirements.................................................................................................................................66
BNXT_RE Driver Dependencies....................................................................................................66
Installation......................................................................................................................................67
Limitations......................................................................................................................................67
Known Issues.................................................................................................................................67
Windows................................................................................................................................................68
Kernel Mode...................................................................................................................................68
Verifying RDMA..............................................................................................................................68
User Mode......................................................................................................................................69
VMware ESX.........................................................................................................................................70
Limitations......................................................................................................................................70
BNXT RoCE Driver Requirements.................................................................................................70
Installation......................................................................................................................................70
Configuring Paravirtualized RDMA Network Adapters...................................................................71
Configuring a Virtual Center for PVRDMA.............................................................................71
Tagging vmknic for PVRDMA on ESX Hosts.........................................................................71
Setting the Firewall Rule for PVRDMA...................................................................................71
Adding a PVRDMA Device to the VM ....................................................................................71
Configuring the VM on Linux Guest OS.................................................................................72
DCBX – Data Center Bridging ...................................................................................................................73
QoS Profile – Default QoS Queue Profile .............................................................................................73
DCBX Mode = Enable (IEEE only)........................................................................................................74
DCBX Willing Bit....................................................................................................................................74
Frequently Asked Questions.....................................................................................................................77

Regulatory and Safety ApprovalsNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 7
Regulatory and Safety Approvals
The following sections detail the Regulatory, Safety, Electromagnetic Compatibility (EMC), and Electrostatic
Discharge (ESD) standard compliance for the NetXtreme-E Network Interface Card.
Regulatory
Safety
Table 1: Regulatory Approvals
Item Applicable Standard Approval/Certificate
CE/European Union EN 62368-1:2014 CB report and certificate
UL/USA IEC 62368-1 ed. 2 CB report and certificate
CSA/Canada CSA 22.2 No. 950 CSA report and certificate.
Taiwan CNS14336 Class B –
Table 2: Safety Approvals
Country Certification Type/Standard Compliance
International CB Scheme
ICES 003 – Digital Device
UL 1977 (connector safety)
UL 796 (PCB wiring safety)
UL 94 (flammability of parts)
Yes

Regulatory and Safety ApprovalsNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 8
Electromagnetic Compatibility (EMC)
Electrostatic Discharge (ESD) Compliance
FCC Statement
This equipment has beentested and found to comply with the limits for a Class B digital device, pursuant to Part
15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference
in a residential installation. This equipment generates uses and can radiate radio frequency energy and, if not
installedandusedinaccordancewiththeinstructions,may cause harmful interferenceto radio communications.
However, there is no guarantee that interference will not occur in a particular installation. If this equipment does
cause harmful interference to radio television reception, which can be determined by turning the equipment off
and on, the user is encouraged to try to correct the interference by one or more of the following measures:
• Reorient or relocate the receiving antenna.
• Increase the separation between the equipment and receiver.
• Consult the dealer or an experienced radio/TV technician for help.
Table 3: Electromagnetic Compatibility
Standard / Country Certification Type Compliance
CE/EU EN 55032:2012/AC:2013 Class B
EN 55024:2010
EN 61000-3-2:2014
EN 61000-3-3:2013
CE report and CE DoC
FCC/USA CFR47, Part 15 Class B FCC/IC DoC and EMC report
referencing FCC and IC standards
IC/Canada ICES-003 Class B FCC/IC DoC and report referencing
FCC and IC standards
ACA/Australia, New Zealand AS/NZS CISPR 22:2009 +A1:2010 /
AS/NZS CISPR 32:2015 ACA certificate RCM Mark
BSMI/Taiwan CNS13438 Class B BSMI certificate
BSMI/Taiwan CNS15663 BSMI certificate
MIC/S. Korea KN32 Class B
KN35 Korea certificate MSIP Mark
VCCI/Japan V-3/2014/04 (take effect 3/31/2015) Copy of VCCI on-line certificate
Table 4: ESD Compliance Summary
Standard Certification Type Compliance
EN55024:2010
(EN 61000-4-2) Air/Direct discharge Yes
Note: Changes or modifications not expressly approved by the manufacture responsible for
compliance could void the user’s authority to operate the equipment.

Functional DescriptionNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 9
Functional Description
Dell supports 10GBase-T, 10G SFP+, and 25G SFP28 Network Interface Cards (NICs). These NICs are
described in Table 5.
Table 5: Functional Description
Network Interface Card Description
BCM957402A4020DLPC/BCM957402A4020DC/BCM957412A4120D/BCM957412M4120D
Speed Dual Port 10 Gbps Ethernet
PCI-E Gen 3 x8a
a. The NIC supports PCI-E Gen 3, Gen 2, and Gen 1 speeds, however, PCI Gen 3 is recommended to achieve
nominal throughput when 2 ports of 25G links transmit and receive traffic at the same time.
Interface SFP+ for 10 Gbps
Device Broadcom BCM57402/BCM57412 10 Gbps MAC controller with Integrated dual
channel 10 Gbps SFI transceiver.
NDIS Name Broadcom NetXtreme E-Series Dual-port 10Gb SFP+ Ethernet PCIe Adapter
UEFI Name Broadcom Dual 10Gb SFP+ Ethernet
BCM57404A4041DLPC/BCM57404A4041DC/BCM957414A4141D/BCM957414M4140D
Speed Dual Port 25 Gbps or 10 Gbps Ethernet
PCI-E Gen 3 x8a
Interface SFP28 for 25 Gbps and SFP+ for 10 Gbps
Device Broadcom BCM57404/BCM57414 25 Gbps MAC controller with Integrated dual
channel 25 Gbps SFI transceiver.
NDIS Name Broadcom NetXtreme E-Series Dual-port 25 Gb SFP28 Ethernet PCIe Adapter
UEFI Name Broadcom Dual 25 Gb SFP 28 Ethernet
BCM957406A4060DLPC/BCM957406A4060DC/BCM957416A4160D/BCM957416M4160
Speed Dual Port 10GBase-T Ethernet
PCI-E Gen 3 x8a
Interface RJ45 for 10 Gbps and 1 Gbps
Device Broadcom BCM57406/BCM57416 10 Gbps MAC controller with Integrated dual
channel 10GBase-T transceiver.
NDIS Name Broadcom NetXtreme E-Series Dual-port 10GBASE-T Ethernet PCIe Adapter
UEFI Name Broadcom Dual 10GBASE-T Ethernet

Functional DescriptionNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 10
Figure 1: BCM957402A4020DC, BCM957412A4120D Network Interface Card

Functional DescriptionNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 11
Figure 2: BCM957404A4041DLPC, BCM957414A4141D Network Interface Card

Functional DescriptionNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 12
Figure 3: BCM957406A4060DLPC, BCM957416A4160D Network Interface Card
Figure 4: BCM957414M4140D Network Daughter Card (rNDC)

Functional DescriptionNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 13
Figure 5: BCM957412M4120D Network Daughter Card (rNDC)
Figure 6: BCM957416M4160 Network Daughter Card (rNDC)

Network Link and Activity IndicationNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 14
Network Link and Activity Indication
BCM957402AXXXX/BCM957412AXXXX
The SFP+ port has two LEDs toindicate traffic activities and link speed. The LEDs are visible throughthe cutout
on the bracket as shown in Figure 7. The LED functionality is described in Table 6.
Figure 7: BCM957402AXXXX/BCM957412AXXXX Activity and Link LED Locations
Table 6: BCM957402AXXXX/BCM957412AXXXX Activity and Link LED Locations
LED Type Color/Behavior Note
Activity Off No Activity
Green blinking Traffic Flowing Activity
Link Off No Link
Green Linked at 10 Gbps
Port 1 SFP+ Cage
Port 1 Activity LED Port 1 Link LED
Port 2 Activity LED Port 2 Link LED
Port 2 SFP+ Cage

Network Link and Activity IndicationNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 15
BCM957404AXXXX/BCM957414AXXXX
The SFP28 port has two LEDs to indicate traffic activities and link speed. The LEDs are visible through the
cutout on the bracket as shown in Figure 8. The LED functionality is described in Table 7.
Figure 8: BCM957404AXXXX/BCM957414AXXXX Activity and Link LED Locations
Table 7: BCM957404AXXXX/BCM957414AXXXX Activity and Link LED Locations
LED Type Color/Behavior Note
Activity Off No Activity
Green blinking Traffic Flowing Activity
Link Off No Link
Green Linked at 25 Gbps
Yellow Linked at 10 Gbps
Port 1 SFP28 Cage
Port 2 SFP28 Cage
Port 1 Activity LED Port 1 Link LED
Port 2 Activity LED Port 2 Link LED

Network Link and Activity IndicationNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 16
BCM957406AXXXX/BCM957416AXXXX
The RJ-45 port has two LEDs to indicate traffic activitiesand link speed. The LEDs are visiblethrough the cutout
on the bracket as shown in Figure 9. The LED functionality is described in Table 8.
Figure 9: BCM957406AXXXX/BCM957416AXXXX Activity and Link LED Locations
Table 8: BCM957406AXXXX/BCM957416AXXXX Activity and Link LED Locations
LED Type Color/Behavior Notes
Activity Off No Activity
Green blinking Traffic Flowing Activity
Link Off No Link
Green Linked at 10 Gbps
Amber Linked at 1 Gbps

Network Link and Activity IndicationNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 17
BCM957414M4140D
The SFP28 port has two LEDs to indicate traffic activities and link speed. The LEDs are visible through the
cutout on the bracket as shown in Figure 10.
Figure 10: BCM957414M4140D Network Daughter Card (rNDC) Activity and Link LED Locations
Table 9: BCM957414M4140D Network Daughter Card (rNDC) Activity and Link LED Locations
LED Type Color/Behavior Notes
Activity Off No Activity
Green blinking Traffic Flowing Activity
Link Off No Link
Green Linked at 25 Gbps
Yellow Linked at 10 Gbps

Network Link and Activity IndicationNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 18
BCM957412M4120D
This rNDC has SFP+ and RJ-45ports, each with two LEDs to indicate traffic activities and linkspeed. The LEDs
are visible as shown in Figure 11.
Figure 11: BCM957412M4120D Network Daughter Card (rNDC) Activity and Link LED Locations
Table 10: BCM957412M4120D Network Daughter Card (rNDC) Activity and
Link LED Locations SFP+ Port 1 and 2
LED Type Color/Behavior Notes
Activity Off No Activity
Green blinking Traffic Flowing Activity
Link Off No Link
Green Linked at 10 Gbps
Table 11: 1000BaseT Port 3 and 4
LED Type Color/Behavior Notes
Activity Off No Activity
Green blinking Traffic Flowing Activity
Link Off No Link
Green Linked at 1 Gbps
Amber Linked at 10/100 Mbps

Network Link and Activity IndicationNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 19
BCM957416M4160
This rNDC has 10GBaseT and 1000BaseT RJ-45 ports, each with two LEDs to indicate traffic activities and link
speed. The LEDs are visible as shown in Figure 12.
Figure 12: BCM957416M4160 Network Daughter Card (rNDC) Activity and Link LED Locations
Table 12: BCM957416M4160 Network Daughter Card (rNDC) Activity and
Link LED Locations 10GBaseT Port 1 and 2
LED Type Color/Behavior Notes
Activity Off No Activity
Green blinking Traffic Flowing Activity
Link Off No Link
Green Linked at 10 Gbps
Amber Linked at 1 Gbps

FeaturesNetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100 Page 20
Features
Refer to the following sections for device features.
Software and Hardware Features
Table 13 provides a list of host interface features.
Table 13: Host Interface Features
Feature Details
Host Interface PCIev3.0 (Gen3:8GT/s;Gen 2:5GT/s; Gen1: 2.5
GT/s).
Number of PCIe lanes PCI-E Edge connector: x8.
Vital Product Data (VPD) Supported.
Alternate Routing ID (ARI) Supported.
Function Level Reset (FLR) Supported.
Advanced Error Reporting Supported.
PCIe ECNs Support for TLP Processing Hints (TPH), Latency
Tolerance Reporting (LTR), and Optimized Buffer
Flush/Fill (OBFF).
MSI-X Interrupt vector per queue 1 per RSS queue, 1 per NetQueue, 1 per Virtual
Machine Queue (VMQ).
IP Checksum Offload Support for transmit and receive side.
TCP Checksum Offload Support for transmit and receive side.
UDP Checksum Offload Support for transmit and receive side.
NDIS TCP Large Send Offload Support for LSOV1 and LSOV2.
NDIS Receive Segment Coalescing (RSC) Support for Windows environments.
TCP Segmentation Offload (TSO) Support for Linux and VMware environments.
Large Receive Offload (LRO) Support for Linux and VMware environments.
Generic Receive Offload (GRO) Support for Linux and VMware environments.
Receive Side Scaling (RSS) Support for Windows, Linux, and VMware
environments. Up to 8 queues/port supported for
RSS.
Header-Payload Split Enables the software TCP/IP stack to receive TCP/
IP packets with header and payload data split into
separate buffers. Supports Windows, Linux, and
VMware environments.