Chelsio Terminator Series User manual

noteIf no
Chelsio Unified Boot
Installation and User’s Guide

Chelsio Unified Boot ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third-party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to s[email protected]
Support
For all support related questions please send email to [e-mail]
Copyright © 2020. Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel: [phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Chelsio KK (Japan)
Yamato Building 8F,
5-27-3 Sendagaya,
Shibuya-ku,
Tokyo 151-0051,
Japan

Chelsio Unified Boot iii
Document History
Version
Revision Date
1.0.0
05/18/2012
1.0.1
07/30/2012
1.0.2
10/05/2012
1.0.3
16/05/2012
1.0.4
07/31/2013
1.0.5
04/29/2014
1.0.6
09/05/2014
1.0.7
09/26/2014
1.0.8
10/13/2014
1.0.9
02/24/2015
1.1.0
05/05/2015
1.1.1
07/07/2015
1.1.2
09/22/2016
1.1.3
11/18/2016
1.1.4
10/03/2017
1.1.5
08/07/2018
1.1.6
10/17/2018
1.1.7
05/19/2020

Chelsio Unified Boot iv
TABLE OF CONTENTS
I. UNIFIED BOOT OPTION ROM 5
1. Introduction 6
1.1. Hardware Requirements 6
1.2. Software Requirements 8
1.3. Pre-requisites 8
1.4. Package Contents 8
2. Hardware Installation 9
3. Flashing Firmware and Option ROM 11
3.1. Preparing USB flash drive 11
3.2. Legacy 12
3.3. uEFI 15
3.4. Default Option ROM Settings 25
4. Configuring PXE Server 26
5. PXE boot process 27
5.1. Legacy PXE boot 27
5.2. uEFI PXE Boot 30
6. FCoE boot process 35
6.1. Legacy FCoE boot 35
6.2. uEFI FCoE Boot 41
7. iSCSI boot process 47
7.1. Legacy iSCSI boot 47
7.2. uEFI iSCSI Boot 55
II. APPENDIX 65
Chelsio End-User License Agreement (EULA) 66

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 5
I.Unified Boot Option ROM

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 6
1.Introduction
Thank you for choosing Chelsio Unified Wire adapters. These high speed, single chip, single
firmware cards provide enterprises and data centers with high performance solutions for various
Network and Storage related requirements.
The Terminator series is Chelsio’s next generation of highly integrated, hyper-virtualized
1/10/25/40/50/100GbE controllers. The adapters are built around a programmable protocol-
processing engine, with full offload of a complete Unified Wire solution comprising NIC, TOE,
iWARP RDMA, iSCSI, FCoE and NAT support. It scales to true 40Gb line rate operation from a
single TCP connection to thousands of connections, and allows simultaneous low latency and
high bandwidth operation thanks to multiple physical channels through the ASIC.
Ideal for all data, storage and high-performance clustering applications, the Unified Wire
adapters enable a unified fabric over a single wire by simultaneously running all unmodified IP
sockets, Fibre Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high-
performance computing environments, Chelsio adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
PXE is short for Preboot eXecution Environment and is used for booting computers over an
Ethernet network using a Network Interface Card (NIC). FCoE SAN boot process involves
installation of an operating system (OS) to an FC/FCoE disk and then booting from it. iSCSI
SAN boot process involves installation of an OS to an iSCSI disk and then booting from it.
This section of the guide explains how to configure and use Chelsio Unified Boot Option
ROM which flashes PXE, iSCSI and FCoE Option ROM onto Chelsio’s adapters. It adds
functionalities like PXE, FCoE and iSCSI SAN boot.
1.1. Hardware Requirements
1.1.1. Supported Adapters
The following are the currently shipping Chelsio Adapters that are compatible with Unified Boot
Option ROM software:
T62100-CR
T62100-LP-CR
T62100-SO-CR*
T6425-CR
T6225-CR
T6225-LL-CR
T6225-SO-CR*
T580-CR
T580-LP-CR

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 7
T580-SO-CR*
T580-OCP-SO*
T540-CR
T540-LP-CR
T520-CR
T520-LL-CR
T520-SO-CR*
T520-OCP-SO*
T520-BT
T540-BT
* Only PXE supported
1.1.2. Supported Hardware
The following hardware platforms are supported by Chelsio Unified Boot Option ROM software:
Dell T5600
DELL PowerEdge 2950
DELL PowerEdge T110
DELL PowerEdge T710
DELL PowerEdge R220
DELL PowerEdge R720
IBM X3650 M2
IBM X3650 M4*
HP Proliant DL180 gen9
HP ProLiant DL385G2
Supermicro X7DWE
Supermicro X8DTE-F
Supermicro X8STE
Supermicro X8DT6
Supermicro X9SRL-F
Supermicro X9SRE-3F
Supermicro-X10DRi
ASUS P5KPL
ASUS P8Z68
Lenovo X3650 M5
Intel DQ57TM
* If system BIOS version is lower than 1.5 and both Legacy and uEFI are enabled, system will
hang during POST. Please upgrade the BIOS version to 1.5 or higher to avoid this issue.
1.1.3. Supported Switches
The following switches are supported by Chelsio Unified Boot Option ROM software:
Cisco Nexus 5010 with 5.1(3) N1 (1a) firmware.

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 8
Arista DCS-7124S-F
Mellanox SX_PPC_M460EX
Other platforms/switches have not been tested and are not guaranteed to work.
1.2. Software Requirements
Chelsio Unified Boot Option ROM software requires Disk Operating System to flash Option
ROM onto Chelsio adapters.
The installation of the following Linux distributions is supported using Chelsio inbox drivers.
Linux Distribution
Drivers
RHEL 8.2, 4.18.0-193.el8
PXE, FCoE, iSCSI
RHEL 8.1, 4.18.0-147.el8
RHEL 7.8, 3.10.0-1127.el7
PXE, iSCSI
RHEL 7.7, 3.10.0-1062.el7
1.3. Pre-requisites
A DOS bootable USB flash drive or Floppy Disk is required for updating firmware, option
ROM, creating DUD, etc.
Secure Boot should be disabled in the system BIOS.
1.4. Package Contents
Chelsio Unified Boot Option ROM package contains the following:
OptionROM: This directory contains Unified Boot Option ROM image (cubt4.bin), uEFI
driver (ChelsioUD.efi), default boot configuration file (bootcfg) and a flash utility (cfut4.exe),
which can be used to flash the option ROM onto Chelsio's adapters. It also contains
Firmware files.
EULA: Chelsio’s End User License Agreement.
docs: The docs directory contains support documents - README, Release Notes and
User’s Guide (this document) for the software package.
Other kernel versions have not been tested and are not guaranteed to work.
Note

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 9
2.Hardware Installation
i. Shutdown/power off your system.
ii. Power off all remaining peripherals attached to your system.
iii. Unpack the Chelsio adapter and place it on an anti-static surface.
iv. Remove the system case cover as per the system manufacturer’s instructions.
v. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
vi. For maximum performance, it is highly recommended to install the adapter into a PCIe
x8/x16 slot.
vii. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
viii.Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
ix. After securing the card, ensure that the card is still fully seated in the PCIE x8/x16 slot as
sometimes the process of securing the card causes the card to become unseated.
x. Connect a fiber/twinax cable, multi-mode for short range (SR) optics or single-mode for long
range (LR) optics, to the Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
xi. Power on your system.
xii. On Linux systems, run update-pciids command to download the current version of PCI ID list
xiii.Verify if the adapter was installed successfully:
•On Linux and ESXi systems, run lspci command and you should see a similar output:
•On Windows systems, follow these steps:
a. Open Device Manager in Control Panel.
b. Under Other devices section, Chelsio adapter should be listed as Ethernet
Controller. If the adapter is not listed, right-click on the system name or click
on the Actions menu and select Scan for hardware changes.
All 4-ports of T6425-CR adapter will be functional only if PCIe x8 -> 2x PCIe x4 slot
bifurcation is supported by the system and enabled in BIOS. Otherwise, only 2-
ports will be functional.
Note

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 10
For Chelsio adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
xiv.Based on the operating system, install the appropriate network driver. Install and load cxgb4
for Linux systems, VBD and NDIS for Windows systems, and cxl for ESXi systems.
xv. Finally, verify if the card is discovered:
•For Linux systems, examine the output of dmesg and you should see a similar
output:
The above output indicates the hardware configuration of the adapters as well as the
Serial numbers.
•For Windows systems, open Device Manager again. Expand Network adapters
section and now Chelsio adapter should be listed.
•For ESXi systems, examine the output of dmesg and you should see a similar
output:
Network device names for Chelsio’s physical ports are assigned using the following
convention: the port farthest from the motherboard will appear as the first network
interface. However, for T5 40G adapters, the association of physical Ethernet ports
and their corresponding network device names is opposite. For these adapters, the
port nearest to the motherboard will appear as the first network interface.
Note

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 11
3.Flashing Firmware and Option ROM
Depending on the boot mode selected, Chelsio Unified Boot provides the following methods to
flash Firmware and Option ROM onto Chelsio adapters:
•Legacy mode: cfut4
•uEFI mode:
oHII
odrvcfg
oFirmware Manager Protocol (FMP)
These methods also provide the functionality to update/erase Hardware configuration and Phy
Firmware files.
3.1. Preparing USB flash drive
This document assumes that you are using a USB flash drive as a storage media for the
necessary files. Follow the steps below to prepare the drive:
i. Create a DOS bootable USB flash drive. (Click here for instructions)
ii. Create a directory CHELSIO on the USB flash drive.
iii. If you haven’t done already, download Chelsio-Uboot-x.x.x.xx.zip from Chelsio Download
Center
iv. Unzip the downloaded package and change your working directory to OptionROM directory.
[root@host~]# unzip Chelsio-Uboot-x.x.x.xx.zip
[root@host~]# cd Chelsio-Uboot-x.x.x.xx/OptionROM
v. Copy all the files and place them in the CHELSIO directory created on the USB flash drive.
vi. Plug-in the USB flash drive in the system on which the Chelsio adapter is installed.
vii. Reboot the system.

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 12
3.2. Legacy
i. In BIOS, configure the system having Chelsio adapter to boot in Legacy mode.
ii. Boot the system from the plugged in USB flash drive and change your working directory to
CHELSIO directory.
C:\>cd CHELSIO
iii. Run the following command to list all Chelsio adapters present on the system. The list
displays a unique index for each adapter found.
C:\CHELSIO>cfut4 –l
iv. Delete any previous version of Option ROM flashed on the adapter.
C:\CHELSIO>cfut4 –d <idx> -xb
Here, idx is the adapter index found in step iii (0 in this case).

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 13
v. Delete any previous firmware using the following command.
C:\CHELSIO>cfut4 -d <idx> -xh –xf
vi. Delete any previous Option ROM settings.
C:\CHELSIO>cfut4 -d <idx> -xc
vii. Run the following command to flash the appropriate firmware.
C:\CHELSIO>cfut4 -d <idx> -uf <firmware_file>.bin
Here, firmware_file is the firmware image file present in the CHELSIO directory.

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 14
viii. Flash the unified Option ROM onto the Chelsio adapter using the following command.
C:\CHELSIO>cfut4 -d <idx> -ub cubt4.bin
Here, cubt4.bin is the unified option ROM image file present in the CHELSIO directory.
ix. To configure the base MAC address (optional), use the below command.
C:\CHELSIO>cfut4 -d <idx> -um <Hex MAC Address>
Example:
C:\CHELSIO>cfut4 -d 0 -um 000743000123
x. Reboot the system for changes to take effect.

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 15
3.3. uEFI
3.3.1. Loading uEFI driver
i. In BIOS, configure the system having Chelsio adapter to boot in uEFI mode.
ii. Boot to EFI Shell.
For Supermicro systems, enable Network Stack as well before proceeding.
Note

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 16
iii. Issue command drivers to determine if Chelsio uEFI driver is already loaded. The below
image shows that the driver is loaded.
If the driver is not loaded, continue to step (v)
iv. Note the handle and unload the driver.
fs0:\CHELSIO\> unload -n <driver_handle>
Example:
v. Load the uEFI driver (ChelsioUD.efi) present in the CHELSIO directory.

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 17
3.3.2. drvcfg
i. Please ensure that Chelsio uEFI driver is loaded correctly as mentioned in Loading uEFI
driver section.
ii. Run the following command to launch the Unified Boot Setup utility.
iii. Choose the Chelsio adapter which needs to be configured.
iv. Highlight Enter flash utility and press [Enter].

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 18
v. Highlight Option ROM and press [Enter].
vi. Highlight Update and press [Enter].

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 19
vii. Enter the path to the Option ROM file and press [Enter].
viii.Similarly, you can use the above method to update Firmware present in the CHELSIO
directory.
ix. Reboot the machine for changes to take effect.
3.3.3. HII
i. Go into the BIOS setup.
ii. Chelsio HII should be listed as Chelsio T5/T6 as shown below. Highlight it and press
[Enter].
If Chelsio T5/T6 is not listed,
•Load the Chelsio uEFI driver as mentioned in Loading uEFI driver section.
•Flash the Option ROM and Firmware as mentioned in drvcfg section.

Chapter I. Unified Boot Option ROM
Chelsio Unified Boot 20
iii. Highlight the Chelsio adapter to be configured and press [Enter].