Xilinx KCU105 User manual

KCU105 PCI Express
Streaming Data Plane
TRD User Guide
KUCon-TRD03
Vivado Design Suite
UG920 (v2017.1) June 01, 2017

PCIe Streaming Data Plane TRD www.xilinx.com 2
UG920 (v2017.1) June 01, 2017
Revision History
The following table shows the revision history for this document.
Date Version Revision
06/01/2017 2017.1 Released with Vivado Design Suite 2017.1. Updated Figure 3-2 through Figure 3-5
and Figure 4-1 through Figure 4-7 (no changes to text).
01/20/2017 2016.4 Released with Vivado Design Suite 2016.4 without changes from the previous
version.
10/05/2016 2016.3 Released with Vivado Design Suite 2016.3 without changes from the previous
version.
06/08/2016 2016.2 Released with Vivado Design Suite 2016.2 without changes from the previous
version.
04/14/2016 2016.1 Released with Vivado Design Suite 2016.1 without changes from the previous
version.
11/24/2015 2015.4 Released with Vivado Design Suite 2015.4 without changes from the previous
version.
10/05/2015 2015.3 Released with Vivado Design Suite 2015.3 with minor textual edits.
06/22/2015 2015.2 Updated for Vivado Design Suite 2015.2. Figure 3-1 and Figure 3-2 were replaced.
Figure 3-13 through Figure 3-15 were updated. In Hardware SGL Prepare Block,
page 49, element buffer size changed from 512 bytes to 4096 bytes. Figure 5-14 and
Figure 5-25 were updated.
05/13/2015 2015.1 Updated for Vivado Design Suite 2015.1. The TRD ZIP file changed to
rdf0307-kcu105-trd03-2015-1.zip. Updated Information about resource
utilization for the base design and the user extension design in Table 1-1 and
Table 1-2. Added details about Windows 7 driver support, setup, and test of the
reference design, updating these sections: Features, Computers, and Appendix A,
Directory Structure. Added section Install TRD Drivers on the Host Computer
(Windows 7) to Chapter 2, Setup and added Appendix B, Recommended Practices
and Troubleshooting in Windows. Removed QuestaSim/ModelSim Simulator
information, because QuestaSim simulation is not supported in Vivado tool release
2015.1. Updated many figures and replaced Figure 1-1, Figure 1-2, and Figure 5-12.
Updated XDMA Driver Stack and Design in Chapter 5. In Table C-3, traffic generator
was changed to traffic checker. Added Appendix E, APIs Provided by the XDMA Driver
in Windows.
02/26/2015 2014.4.1 Initial Xilinx release.
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 3
UG920 (v2017.1) June 01, 2017
Table of Contents
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Chapter 1: Introduction
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Resource Utilization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Chapter 2: Setup
Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Preliminary Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Chapter 3: Bringing Up the Design
Set the Host System to Boot from the LiveDVD (Linux) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Configure the FPGA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Run the Design on the Host Computer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Test the Reference Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Remove Drivers from the Host Computer
(Windows Only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Chapter 4: Implementing and Simulating the Design
Implementing the Base Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Implementing the User Extension Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Simulating the Designs Using Vivado Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Chapter 5: Targeted Reference Design Details and Modifications
Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Data Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Reference Design Modifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 4
UG920 (v2017.1) June 01, 2017
Appendix A: Directory Structure
Appendix B: Recommended Practices and Troubleshooting in Windows
Recommended Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Appendix C: Register Space
Generator and Checker Configuration Registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Ethernet MAC Registers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Appendix D: APIs Provided by the XDMA Driver in Linux
Appendix E: APIs Provided by the XDMA Driver in Windows
Appendix F: Network Performance
Private Network Setup and Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Appendix G: Additional Resources and Legal Notices
Xilinx Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Solution Centers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Please Read: Important Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 5
UG920 (v2017.1) June 01, 2017
Chapter 1
Introduction
This document describes the features and functions of the PCI Express® Streaming Data
Plane targeted reference design (TRD). The TRD comprises a base design and a user
extension design. The user extension design adds custom logic on top of the base design.
The pre-built user extension design in this TRD adds an Ethernet application.
Overview
The TRD targets the Kintex® UltraScale™ XCKU040-2FFVA1156E FPGA running on the
KCU105 evaluation board and provides a platform for data transfer between the host
machine and the FPGA. The top-level block diagram of the TRD base design is shown in
Figure 1-1.
The TRD uses an integrated Endpoint block for PCI Express (PCIe®) in a x8 Gen2
configuration along with an Expresso DMA Bridge Core from Northwest Logic [Ref 1] for
high performance data transfers between host system memory and the Endpoint (the FPGA
on the KCU105 board).
The DMA bridge core (DMA block) provides protocol conversion between PCIe transaction
layer packets (TLPs) and AXI transactions. The core’s hardware scatter gather list (SGL) DMA
interface provides buffers management at the Endpoint to enable the streaming interface.
X-Ref T arget - Fig ure 1-1
Figure 1-1: KCU105 PCI Express Streaming Data Plane Base Design
3URFHVVRU
5RRW
&RPSOH[
'0$
'ULYHU
3&,H
[*HQ
/LQN
ELWVDW
0+]
User
App 3&,H'0$
XCKU040-2FFVA1156E FPGA
KCU105 Evaluation Board
+RVW&RPSXWHU
+:6*/3UHSDUH
PPVVPP
$;,/LWH6ODYHV
$;,
,QWHUFRQQHFW
+DUGZDUHPRGXOHV 6RIWZDUHUXQQLQJRQKRVWFRPSXWHU &RPSRQHQWRQKRVWFRPSXWHU
*(1
&+.
00ELWV
DW0+]
6WUHDPELWV
DW0+]
UG920_c1_01_040615
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 6
UG920 (v2017.1) June 01, 2017
Chapter 1: Introduction
The downstream AXI4-Lite slaves include a power monitor module, user space registers,
and an AXI performance monitor.
In the system to card direction (S2C), the DMA block moves data from host memory to the
FPGA through the integrated Endpoint block and then makes the data available at the AXI
streaming interface. In the card to system (C2S) direction, theDMA block uses the data
available on the AXI streaming interface and writes to host system memory through the
integrated Endpoint block.
The base design uses a simple generator/checker (GEN/CHK) block as a data
provider/consumer on the AXI streaming interface. This base platform can be extended to
a variety of applications such as Ethernet, and Crypto Engine.
The user extension design (shown in Figure 1-2) provides an example of a dual 10GbE
network interface card. The Ethernet frames received from the host system are directed to
respective Ethernet ports for transmission, and incoming frames from Ethernet are directed
to the host after performing an address filtering operation.
The designs delivered as part of this TRD use Vivado® IP integrator to build the system. IP
Integrator provides intelligent IP integration in a graphical, Tcl-based,
correct-by-construction IP and system-centric design development flow. For further details
on IPI, see Vivado Design Suite Tutorial: Designing IP Subsystems Using IP Integrator (UG995)
[Ref 2].
X-Ref Target - Figure 1-2
Figure 1-2: KCU105 PCI Express Streaming Data Plane User Extension Design
UG920_c1_02_040615
352&(6625
Root
Complex
'0$
'ULYHU
3&,H
[*HQ
/LQN
128 bits at
250 MHz
User
App
3&,H
'0$
;&.8))9$()3*$
.&8(YDOXDWLRQ%RDUG
+RVW&RPSXWHU
$;,/LWH
6ODYHV
$;,
,QWHUFRQQHFW
+DUGZDUHPRGXOHV 6RIWZDUHUXQQLQJRQKRVWFRPSXWHU &RPSRQHQWRQKRVWFRPSXWHU
MM - 128 bits
at 250 MHz
Stream - 64 bits
at 156.25 MHz
*
7
User Extension logic
*
0$&
*
7
*
0$&
Depending on the user application, the two modules (red colored arrows) need to be changed.
All other blocks in the hardware design and software components can be reused.
+:6*/3UHSDUH
PPVVPP
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 7
UG920 (v2017.1) June 01, 2017
Chapter 1: Introduction
Features
The TRD includes these features:
•Hardware
°Integrated Endpoint block for PCI Express
- 8 lanes, each operating at 5 GT/s (gigatransfers per second) per lane per
direction
- 128-bit at 250 MHz
°DMA bridge core
- Scatter gather enabled
- 4 channels: 2 channels are used as S2C and 2 as C2S
- Support for an AXI3 interface
- Two ingress and two egress translation regions supported
Note: The IP can support a higher number of translation regions. The netlist used here is
built to support only two such regions. Contact Northwest Logic for further customization of
IP [Ref 1].
°SGL DMA interface block
- Queue management of channels: Destination queue for S2C and source queue
for C2S
- AXI memory map (MM) to AXI-stream interface conversion and AXI-stream to
MM conversion
- 128-bit at 250 MHz rate operation
- SGL submission block interface between the DMA bridge core and the SGL
preparation block for SGL element submission to DMA channels on a
round-robin basis across channels
- Traffic generator (packets on AXI4-stream interface) and checker block
operating at a 128-bit, 250 MHz rate
- AXI performance monitor capturing AXI4-stream interface throughput numbers
- Two 10G Ethernet MAC, PCS PMA blocks operating at 64-bit, 156.25 MHz
•Software
°64-bit Linux kernel space drivers for DMA and a raw data driver
°64-bit Windows 7 drivers for DMA and a raw data driver
°User space application
°Control and monitoring graphical user interface (GUI)
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 8
UG920 (v2017.1) June 01, 2017
Chapter 1: Introduction
Resource Utilization
Table 1-1 and Table 1-2 list the resources used by the TRD base and user extension designs
after synthesis. Place and route can alter these numbers based on placements and routing
paths. These numbers are to be used as a rough estimate of resource utilization. These
numbers might vary based on the version of the TRD and the tools used to regenerate the
design.
Table 1-1: Base Design Resource Utilization
Resource Type Available Used Usage (%)
CLB registers 484,800 79,865 16.63
CLB LUT 242,400 51,404 21.21
Block RAM 600 39.5 6.58
MMCME3_ADV 10 1 10
Global Clock Buffers 240 3 1.25
BUFG_GT 120 5 4.17
IOB 520 18 3.46
SYSMONE1 1 1 100
PCIE_3_1 3 1 33.33
GTHE3_CHANNEL 20 8 40
Table 1-2: User Extension Design Resource Utilization
Resource Type Available Used Usage (%)
CLB registers 484,800 101,313 20.90
CLB LUT 242,400 64,884 26.77
Block RAM 600 80 13.33
MMCME3_ADV 10 1 10
Global Clock Buffers 240 3 1.25
BUFG_GT 120 12 10.00
IOB 520 23 4.42
SYSMONE1 1 1 100
PCIE_3_1 3 1 33.33
GTHE3_CHANNEL201050
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 9
UG920 (v2017.1) June 01, 2017
Chapter 2
Setup
This chapter identifies the hardware and software requirements, and the preliminary setup
procedures required prior to bringing up the targeted reference design.
Requirements
Hardware
Board and Peripherals
• KCU105 board with the Kintex® UltraScale™ XCKU040-2FFVA1156E FPGA
• USB cable, standard-A plug to micro-B plug (Digilent cable)
• Power supply: 100 VAC–240 VAC input, 12 VDC 5.0A output
• ATX power supply
• ATX power supply adapter
Computers
A control computer is required to run the Vivado® Design Suite and configure the on-board
FPGA. This can be a laptop or desktop computer with any operating system supported by
Vivado tools, such as Redhat Linux or Microsoft® Windows 7.
The reference design test configuration requires a host computer comprised of a chassis
containing a motherboard with a PCI Express slot, monitor, keyboard, and mouse. A DVD
drive is also required if a Linux operating system is used. If a Windows 7 operating system
is used, the 64-bit Windows 7 OS and the Java SE Development Kit 7 must be installed.
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 10
UG920 (v2017.1) June 01, 2017
Chapter 2: Setup
Software
Vivado Design Suite 2017.1 is required. The Fedora 20 LiveDVD, on which the TRD software
and GUI run, is only required if a Linux operating system is used.
Preliminary Setup
Complete these tasks before bringing up the design.
Install the Vivado Design Suite
Install Vivado Design Suite 2017.1 on the control computer. Follow the installation
instructions provided in the Vivado Design Suite User Guide Release Notes, Installation, and
Licensing (UG973) [Ref 3].
Download the Targeted Reference Design Files
1. Download rdf0307-kcu105-trd03-2017-1.zip from the Xilinx Kintex UltraScale
FPGA KCU105 Evaluation Kit - Documentation & Designs website. This ZIP file contains
the hardware design, software drivers, and application GUI executables.
2. Extract the contents of the file to a working directory.
3. The extracted contents are located at <working_dir>/kcu105_axis_dataplane.
The TRD directory structure is described in Appendix A, Directory Structure.
Install TRD Drivers on the Host Computer (Windows 7)
Note: This section provides steps to install KUCon-TRD drivers and is only applicable to a host
computer running Windows 7 64-bit OS. If running Linux, proceed to Set DIP Switches, page 12.
Disable Driver Signature Enforcement
Note: Windows only allows drivers with valid signatures obtained from trusted certificate
authorities to load in a Windows 7 64-bit OS computer. Windows drivers provided for this reference
design do not have a valid signature. Therefore, you have to disable Driver Signature Enforcement on
the host computer as follows:
1. Power up the host system.
2. Press F8 to go to the Advanced Boot Options menu.
3. Select the Disable Driver Signature Enforcement option shown in Figure 2-1, and
press Enter.
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 11
UG920 (v2017.1) June 01, 2017
Chapter 2: Setup
X-Ref Target - Figure 2-1
Figure 2-1: Disable Driver Signature Enforcement
UG920_c2_01_040915
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 12
UG920 (v2017.1) June 01, 2017
Chapter 2: Setup
Install Drivers
1. From Windows Explorer, navigate to the folder in which the reference design is
downloaded (<dir>\kcu105_axis_dataplane\software\windows) and run the
setup file with Administrator privileges as shown in Figure 2-2.
2. Click Next after the InstallShield Wizard opens.
3. Click Next to install to the default folder; or click Change to install to a different folder.
4. Click Install to begin driver installation.
5. A warning screen is displayed as the drivers are installed, because the drivers are not
signed by a trusted certificate authority yet. To install the drivers, ignore the warning
message and click Install this driver software anyway. This warning message pops up
two times. Repeat this step.
6. After installation is complete, click Finish to exit the InstallShield Wizard.
Set DIP Switches
Ensure that the DIP switches and jumpers on the KCU105 board are set to the factory
default settings as identified in the Kintex UltraScale FPGA KCU105 Evaluation Board User
Guide (UG917) [Ref 4].
X-Ref Target - Figure 2-2
Figure 2-2: Run the Setup File with Administrator Privileges
UG920_c2_02_040915
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 13
UG920 (v2017.1) June 01, 2017
Chapter 2: Setup
Install the KCU105 Board
1. Remove all rubber feet and standoffs from the KCU105 board.
2. Power down the host chassis and disconnect the power cord.
CAUTION! Remove the power cord to prevent electrical shock or damage to the KCU105 board or other
components.
3. Ensure that the host computer is powered off.
4. Open the chassis. Select a vacant PCIe Gen2-capable expansion slot and remove the
expansion cover at the back of the chassis.
5. Plug the KCU105 board into the PCIe connector slot, as shown in Figure 2-3.
X-Ref Target - Figure 2-3
Figure 2-3: PCIe Connector Slot
UG920_c3_01_061915
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 14
UG920 (v2017.1) June 01, 2017
Chapter 2: Setup
6. Connect the ATX power supply to the KCU105 board using the ATX power supply
adapter cable as shown in Figure 2-4.
Note: A 100 VAC–240 VAC input, 12 VDC 5.0A output external power supply can be substituted
for the ATX power supply.
7. Slide the KCU105 board power switch SW1 to the ON position (ON/OFF is marked on the
board).
X-Ref Target - Figure 2-4
Figure 2-4: Power Supply Connection to the KCU105 Board
UG920_c3_02_061915
%RDUG3RZHU
6ZLWFK6
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 15
UG920 (v2017.1) June 01, 2017
Chapter 3
Bringing Up the Design
This chapter describes how to bring up and test the targeted reference design.
Set the Host System to Boot from the LiveDVD
(Linux)
Note: This section is only applicable to host computers running Linux. If running Windows 7,
proceed to Configure the FPGA, page 16.
1. Power on the host system. Stop it during BIOS to select options to boot from a DVD
drive. BIOS options are entered by pressing DEL, F12, or F2 keys on most systems.
Note: If an external power supply is used instead of the ATX power, the FPGA can be configured
first. Then power on the host system.
2. Place the Fedora 20 LiveDVD into the DVD drive.
3. Select the option to boot from DVD.
Complete the Configure the FPGA procedures before exiting the BIOS setup to boot from
the DVD.
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 16
UG920 (v2017.1) June 01, 2017
Chapter 3: Bringing Up the Design
Configure the FPGA
While in BIOS, program the FPGA with the BIT file.
1. Connect the standard-A plug to micro-B plug USB cable to the JTAG port on the KCU105
board and to the control computer laptop as shown in Figure 3-1.
Note: The host system can remain powered on.
Note: Figure 3-1 shows a Rev C board. The USB JTAG connector is on the PCIe panel for
production boards.
X-Ref Target - Figure 3-1
Figure 3-1: Connect the USB Cable to the KCU105 Board and the Control Computer
UG920_c3_03_021215
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 17
UG920 (v2017.1) June 01, 2017
Chapter 3: Bringing Up the Design
2. Launch the Vivado® Integrated Design Environment (IDE) on the control computer:
a. Select Start > All Programs > Xilinx Design Tools > Vivado 2017.1 > Vivado
2017.1.
b. On the getting started page, click Open Hardware Manager (Figure 3-2).
X-Ref Target - Figure 3-2
Figure 3-2: Vivado IDE Getting Started Page, Open Hardware Manager
8*BFBB
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 18
UG920 (v2017.1) June 01, 2017
Chapter 3: Bringing Up the Design
3. Open the connection wizard to initiate a connection to the KCU105 board:
a. Click Open target > Auto connect (Figure 3-3).
X-Ref Target - Figure 3-3
Figure 3-3: Using the User Assistance Bar to Open a Hardware Target
8*BFBB
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 19
UG920 (v2017.1) June 01, 2017
Chapter 3: Bringing Up the Design
4. Configure the wizard to establish connection with the KCU105 board by selecting the
default value on each wizard page. Click Next > Next > Next > Finish.
a. In the hardware view, right-click xcku040 and click Program Device (Figure 3-4).
X-Ref Target - Figure 3-4
Figure 3-4: Select Device to Program
8*BFBB
Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 20
UG920 (v2017.1) June 01, 2017
Chapter 3: Bringing Up the Design
b. In the Bitstream file field, browse to the location of the BIT file
<working_dir>/kcu105_axis_dataplane/ready_to_test/trd03_base_
top.bit and click Program (see Figure 3-5).
5. After the FPGA is programmed, the DONE LED status should illuminate as shown in
Figure 3-6.
X-Ref Target - Figure 3-5
Figure 3-5: Program Device Window
X-Ref Target - Figure 3-6
Figure 3-6: GPIO LED Indicators
8*BFBB
UG920_c3_08_092414
Send Feedback