AMD XILINX T2 Telco User manual

T2 Telco Accelerator Card
Installaon Guide
UG1527 (v1.0) June 15, 2022
Xilinx is creating an environment where employees, customers, and
partners feel welcome and included. To that end, we’re removing non-
inclusive language from our products and related collateral. We’ve
launched an internal initiative to remove language that could exclude
people or reinforce historical biases, including terms embedded in our
software and IPs. You may still find examples of non-inclusive
language in our older products as we work to make these changes and
align with evolving industry standards. Follow this link for more
information.

Table of Contents
Chapter 1: Introduction.............................................................................................. 4
Minimum System Requirements and Setup............................................................................ 5
Configuring the PCIe Slot Bifurcation.......................................................................................5
Chapter 2: Card Information and Installation............................................. 10
Safety Instructions.................................................................................................................... 10
Electrostatic Discharge............................................................................................................. 10
Before You Begin.......................................................................................................................10
Installing the Card.....................................................................................................................11
Chapter 3: Installing Additional Software......................................................12
Chapter 4: T2 Factory Installed Image..............................................................13
Flashing the Images in QSPI Using flash_app....................................................................... 13
Chapter 5: T2 Base Software Design.................................................................. 15
T2 Base Software Design on the ZU48 Zynq UltraScale+ RFSoC......................................... 15
T2 Skeleton Design Package.................................................................................................... 16
Chapter 6: Running the Tests.................................................................................18
ZU48 Zynq UltraScale+ RFSoC PCIe Loopback Test...............................................................19
DDR Read/Write Tests...............................................................................................................21
Test Application Guide and Address Map.............................................................................. 24
Chapter 7: Next Steps.................................................................................................25
Chapter 8: Dependencies/Known Issues.........................................................26
Chapter 9: Programming the Devices Using JTAG......................................27
Flashing the Images to ZU48 Zynq UltraScale+ RFSoC QSPI Using the Vitis
Environment..........................................................................................................................30
Programming the Bitstreams Directly ...................................................................................31
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 2
Send Feedback

Appendix A: Regulatory Compliance Statements...................................... 33
FCC Class A Products.................................................................................................................33
Safety.......................................................................................................................................... 33
EMC Compliance........................................................................................................................34
FCC Class A User Information..................................................................................................34
VCCI Class A Statement............................................................................................................ 35
Appendix B: Additional Resources and Legal Notices............................. 36
Xilinx Resources.........................................................................................................................36
Documentation Navigator and Design Hubs.........................................................................36
References..................................................................................................................................36
Revision History.........................................................................................................................37
Please Read: Important Legal Notices................................................................................... 37
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 3
Send Feedback

Chapter 1
Introduction
This document provides hardware and soware installaon procedures for the T2 Telco
accelerator card along with a guide to the T2 skeleton design. The skeleton design is created
specically for the 16 nm Zynq® UltraScale+™ RFSoC device on the T2 card, and provides
connecons and soware to validate the main interfaces of the board.
Figure 1: T2 Telco Accelerator Card
The Xilinx® T2 Telco accelerator card is a PCI Express® (PCIe) Gen4 x16 compliant card featuring
the 16 nm Zynq UltraScale+ RFSoC device. The T2 form factor is half height, half length (HHHL)
and single slot, with a PCIe Gen4 x16 interface that is x8x8 bifurcated providing x8 links from the
host to the Zynq UltraScale+ RFSoC device. Target applicaons for the T2 card include the
following:
• 4G LTE and 5G NR high-PHY lookaside acceleraon (supporng 3GPP split opon 7–2x)
• 5G layer 1 (L1) high-PHY lookaside acceleraon
• 4G LTE and 5G NR inline acceleraon of L1 funcons (supporng 3GPP split opon 7–2x) for
up to 4TRX
Chapter 1: Introduction
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 4
Send Feedback

The T2 card turns a standard server into a virtual baseband unit with the performance, low
latency, and power eciency needed for O-RAN 5G deployments. The turnkey soluon enables
operators, system integrators, and OEMs to get to market quickly and to simplify the deployment
of services at the edge.
Minimum System Requirements and Setup
The minimum system requirements for running a T2 Telco accelerator card are shown below:
Table 1: Minimum System Requirements
Component Requirement
Motherboard PCI Express 4.0 compatible with one dual-width x16 slot.
System power supply 75W through a PCI Express slot connection
Operating system Linux, 64-bit:
CentOS 7 and 8
Ubuntu 16, 18, and 20
System memory and CPU cores For deployment installations, a minimum of 16 GB plus
application memory requirements is required. For
development installations, a minimum of 64 GB of device
memory is required, but 80 GB is recommended.
A minimum of 16 logical cores is required.
Internet connection Required for downloading drivers and utilities.
Hard disk space Satisfy the minimum system requirements for your
operating system.
Auxiliary power supply No auxiliary power supply is required because the T2 card
cannot boot standalone.
Vitis™ application acceleration development flow The Vitis™ application acceleration development flow is not
currently supported.
The T2 Telco accelerator card is intended for deployment on most servers. Thermal tesng has
been conducted on a Dell R640. Funconal tesng has been conducted on a number of
addional servers including Dell R730 and R740.
Configuring the PCIe Slot Bifurcation
Perform the following steps to enable bifurcaon in the slot where the T2 card has been
inserted. This is required for the T2 card to work properly.
Note: These steps are for a single T2 card deployment.
1. Power on the server. The output on the monitor is shown below.
Chapter 1: Introduction
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 5
Send Feedback

2. Wait unl the monitor lists the shortcut keys, so you can navigate into the setup. The screen
with the shortcut keys looks like the following gure.
3. Press F2 to open the system setup sengs.
4. In the System Setup screen, select the System BIOS opon as shown below.
Chapter 1: Introduction
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 6
Send Feedback

5. Select Integrated Devices.
6. In the menu that is displayed, scroll down unl you get to Slot Bifurcaon. It is on the second
half of the page.
Chapter 1: Introduction
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 7
Send Feedback

7. Click the required slot to change bifurcaon.
8. Click Default Bifurcaon and select x8x8 or x8 Bifurcaon. This enables the T2 card in the
slot with the correct communicaon over the PCI. Click the Back buon in the lower right
hand side of the screen.
Chapter 1: Introduction
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 8
Send Feedback

9. Return to the System BIOS screen and click the Back buon again.
10. In the System BIOS Sengs main menu screen, click Finish.
11. A popup window will show a warning, as shown below. Click Yes to save the changes you
made.
12. Click the Finish buon again.
13. When the system has rebooted, check to see if the T2 card has been detected using the
lspci | grep Xilinx command.
Chapter 1: Introduction
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 9
Send Feedback

Chapter 2
Card Information and Installation
Safety Instructions
To ensure your personal safety and the safety of your equipment:
• Keep your work area and the computer/server clean and clear of debris.
• Before opening the computer/system cover, unplug the power cord.
Electrostatic Discharge
Electrostac discharge (ESD) can damage electronic components when they are improperly
handled, and can result in total or intermient failures. Always follow ESD prevenon procedures
when removing and replacing components. To prevent ESD damage:
• Use an ESD wrist or ankle strap and ensure that it makes skin contact. Connect the equipment
end of the strap to an unpainted metal surface on the chassis.
• Avoid touching the card with your clothing. The wrist strap protects components from ESD on
the body only.
• Handle the card by its bracket or edges only. Avoid touching the printed circuit board or the
connectors.
• Put the card down only on an anstac surface such as the bag supplied in your kit.
• If you are returning the card to Xilinx product support, place it back in its anstac bag
immediately.
Before You Begin
IMPORTANT! The T2 Telco accelerator card is a delicate and sensive electronic device and should be
installed by a qualied technician only. This equipment is intended for installaon in a restricted access
locaon.
Chapter 2: Card Information and Installation
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 10
Send Feedback

• Verify that the minimum card space is available to install your card.
• Check for card compability with your system. Check system requirements such as power, bus
type, and physical dimensions to support the card.
Installing the Card
The following procedure is a guide to T2 Telco accelerator card PCI x16 installaon. Consult your
computer documentaon for addional informaon.
Note: This procedure is for use with UL Listed Servers or ITE.
1. Ensure that the host power supply is disconnected.
2. If you are working with an enclosed computer, open your computer by removing the casing.
3. If necessary, remove the adjacent PCIe slot cover corresponding to the PCIe slot in which you
are installing the card.
4. Plug the T2 card into the bifurcated PCIe x16 slot on the motherboard.
Note: There is no auxiliary power connector on the T2 card.
Note: There is also a USB JTAG connector that can be connected from the server to the T2 card. It can
be used to program the ash. This is for development purposes and is not intended for use in a
producon seng.
Chapter 2: Card Information and Installation
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 11
Send Feedback

Chapter 3
Installing Additional Software
To run the T2 card skeleton design or other demos provided by Xilinx partners, install the
following addional soware on the server:
• DPDK package. The supported versions are 18.11, 19.11, and 20.11.
• DPDK-based pktgen applicaon. The following versions are supported:
○DPDK 18.11 – pktgen-3.6.1
○DPDK 19.11 – pktgen-19.12.0
○DPDK 20.11 – pktgen-21.05.0
Note: For DPDK version 20.11, the DPDG kernel modules are no longer included in the DPDK
package. The dpdk-kmods package is therefore included in the T2 card skeleton design package.
•Applicaon scripts
This soware is included in the T2 skeleton design package along with a script to ensure that the
soware is installed correctly. The package and the scripts are described in the T2 Skeleton
Design Package secon of this document.
The demos provided by Xilinx partners also contain the addional soware. For each design or
demo, there is a local directory for the addional soware as well as instrucons on building it
for the parcular design. There is no central locaon on the server for these demos.
The soware has been tested on the following Linux distribuons: Ubuntu 18, Ubuntu 20,
CentOS 7, and CentOS 8.
To run the Xilinx demos, the requirements are as follows:
• Root access is required.
• HugePages is enabled.
Chapter 3: Installing Additional Software
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 12
Send Feedback

Chapter 4
T2 Factory Installed Image
The T2 Telco accelerator card is preprogrammed with an image in the QSPIs that consists of two
separate designs that support the Zynq UltraScale+ RFSoC (ZU48). Each of the designs provides
simple connecvity from the host server to the card and the two devices.
The latest T2 factory installed image includes the latest T2 skeleton design.
Flashing the Images in QSPI Using flash_app
As part of the manufacturing design, an applicaon called ash_app runs on processors on the
two devices. This applicaon allows you to program 5G acceleraon soluons on the T2 card. An
alternave method using the JTAG/USB cable and Xilinx tools is described in Chapter 9:
Programming the Devices Using JTAG.
The following secons describe how to set up the ash_app code to program the QSPI that is
connected to the ZU48 Zynq UltraScale+ RFSoC device.
Flashing the Images to ZU48 Zynq UltraScale+ RFSoC
QSPI Using flash_app
1. Ensure that the PCIe PFs for ZU48 Zynq UltraScale+ RFSoC are visible in the lspci
command.
2. Execute ./rfsoc_flash_qspi.sh <RFSOC BOOT.BIN absolute file path>
[<QSPI flash offset> [<PCIe BDF>]] inside the flash_app directory.
3. Wait unl the message “Programmed boot image” appears.
Flashing the SC Image Using the sc_flash Script
1. Verify the SC state. Execute ./sc_flash.sh get_sc_state. The card should be in
normal mode before wring the SC image, as shown below:
SC is in NORMAL mode
Chapter 4: T2 Factory Installed Image
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 13
Send Feedback

2. Verify the SC rmware version. Execute ./sc_flash.sh get_sc_fw_version. A
possible successful output is shown below:
SC fw version: 01.02.00
3. Bring the card in FW_AVAILABLE mode. For the rst me, execute ./sc_flash.sh
<image_name> <bsl_password_file>. The expected output is as follows:
"Please reboot the Host and re-run the command"
4. Reboot the host.
5. Program the SC image. For the second me, execute ./sc_flash.sh <image_name>
<bsl_password_file>.
Running host Application
Fw upgrade started
Fw image Data > @0[16]
Fw image Data > @200[16]
Fw image Data > @1AE20[16]
Fw image Data > @3F000[16]
[flash_mem_sections] : 4
BslPasswordSize:56
Programming SC firmware image... (100%)
Programmed SC firmware image done successfully
6. During the upgrade, the SC state should be FW_UPDATE, as shown below.
SC is in FW_UPDATE mode
Execute ./sc_flash.sh get_sc_state.
Chapter 4: T2 Factory Installed Image
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 14
Send Feedback

Chapter 5
T2 Base Software Design
The T2 card skeleton design supports the ZU48 Zynq® UltraScale+™ RFSoC on the card. The
design provides connecvity with the ports on the board from the ZU48 device. The design
allows you to test the connecvity from the host to the card to demonstrate that the board is
fully funconal.
T2 Base Software Design on the ZU48
Zynq UltraScale+ RFSoC
The design on the ZU48DR Zynq UltraScale+ RFSoC has Linux running on the PS Cortex-A53
processors. It also has connecons from the PCIe Gen4 x8 interface from the host to the
dierent parts of the ZU48 device. The following connecons are accomplished through AXI
interconnect:
• The connecon between the PS and the PS DDR memory controller
• The connecon to the PL DDR through a MIG DDR memory controller
The following block diagram shows the connecons.
Chapter 5: T2 Base Software Design
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 15
Send Feedback

Figure 2: ZU48 Skeleton Design Connections
X26411-031722
Text
QDMA
AXI
Interconnect
AXI4-Stream
Interconnect
Config
PS
DDR4 MIG
PCIe Gen4 x8
PS DDR4
PL DDR4
AXI4-Lite
AXI MM
H2C AXIS
C2H AXIS
T2 Skeleton Design Package
The T2 card skeleton design package comes as a Git repository. The code is stored in dierent
branches, with one separate branch for each supported DPDK version (18.11, 19.11, and 20.11).
Chapter 5: T2 Base Software Design
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 16
Send Feedback

Figure 3: Git Branch Structure
The following les are included:
•build.sh: This script downloads all packages from the Internet and applies Xilinx patches.
•compile.sh: This script is used to compile the addional host soware required to run the
design.
•test.sh: This script is used to run all the tests described in Chapter 6: Running the Tests.
When you have downloaded the package, select the branch for the correct DPDK version, then
run build.sh followed by compile.sh at the root directory level.
Chapter 5: T2 Base Software Design
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 17
Send Feedback

Chapter 6
Running the Tests
The setup of the skeleton design to test the T2 card is shown in the following gure.
Figure 4: Test Setup
The following secons describe each test, including the setup, running, and expected results. The
primary objecve of tesng is to verify the following main interfaces, and then to verify data
transfer by comparing the output with expectaons:
Chapter 6: Running the Tests
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 18
Send Feedback

• ZU48 Zynq UltraScale+ RFSoC PCIe x8 interface
• ZU48 Zynq UltraScale+ RFSoC PS/PL DDR4
ZU48 Zynq UltraScale+ RFSoC PCIe Loopback
Test
The QDMA IP and DPDK driver are used to perform the PCIe loopback test. The host generates
L2 frames with the help of the DPDK pktgen applicaon. The QDMA poll mod driver helps to
control the QDMA IP inside the programmable logic from the host. The DPDK pktgen applicaon
sends the packets to QDMA and the same packets are redirected to QDMA, so the packets sent
by the pktgen applicaon are received by pktgen itself.
Test Procedure
1. Set up the system as shown in the following gure.
2. Ensure the condions in the Chapter 2: Card Informaon and Installaon secon have been
met.
3. Run the test.sh in root, with init as argument, to inialize the DMA and DPDK drivers.
4. Execute test.sh script, with rpci as argument, to launch the pktgen applicaon with one
port.
5. Set the packet size to 1514 with the command set all size 1514 inside the pktgen
applicaon.
6. Start the trac with the command start all and analyze the throughput.
7. Observe the RX and TX rate and check whether there is any loss.
8. When the test is complete, stop trac with the command stop all and then quit the test.
The expected results from running the Zynq UltraScale+ RFSoC PCIe loopback test are
shown below.
Chapter 6: Running the Tests
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 19
Send Feedback

Test Requirements
• The throughput should be near to 45 Gb/s in TX and RX of the pktgen applicaon.
• There should not be any packet loss; RX and TX counts should be idencal.
• There should not be any mismatch in TX and RX packets.
Chapter 6: Running the Tests
UG1527 (v1.0) June 15, 2022 www.xilinx.com
T2 Telco Accelerator Card Installation Guide 20
Send Feedback