Xilinx Versal acap cpm4 User manual

Versal ACAP CPM Mode for

PCI Express

Product Guide

Vivado Design Suite

PG346 (v3.3) November 16, 2022

Xilinx is creating an environment where employees, customers, and

partners feel welcome and included. To that end, we’re removing non-

inclusive language from our products and related collateral. We’ve

launched an internal initiative to remove language that could exclude

people or reinforce historical biases, including terms embedded in our

software and IPs. You may still find examples of non-inclusive language

in our older products as we work to make these changes and align

with evolving industry standards. Follow this link for more

information.

Table of Contents

Chapter 1: Overview......................................................................................................4

Navigating Content by Design Process.................................................................................... 4

Introduction to the CPM4...........................................................................................................5

Introduction to the CPM5.........................................................................................................11

Use Modes..................................................................................................................................16

Licensing and Ordering............................................................................................................22

Chapter 2: Tandem Configuration.......................................................................23

Overview.....................................................................................................................................23

Enable the Tandem Configuration Solution...........................................................................24

Deliver Programming Images to Silicon................................................................................ 30

Tandem Configuration Performance......................................................................................31

Design Operation...................................................................................................................... 31

Loading Tandem PCIe for Stage 2...........................................................................................32

Known Issues and Limitations.................................................................................................36

Chapter 3: Product Specification......................................................................... 38

Performance.............................................................................................................................. 38

Minimum Device Requirements..............................................................................................39

Port Descriptions.......................................................................................................................40

Register Space......................................................................................................................... 122

Chapter 4: Designing with the Core................................................................. 124

Clocking.................................................................................................................................... 124

Resets........................................................................................................................................126

AXI4-Stream Interface Description....................................................................................... 126

Chapter 5: Design Flow Steps...............................................................................188

Customizing and Generating the CIPS IP Core for CPM4.................................................. 188

Customizing and Generating the CIPS IP Core for CPM5.................................................. 211

Appendix A: GT Selection and Pin Planning for CPM4............................237

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 2

Send Feedback

CPM4 GT Selection.................................................................................................................. 238

CPM4 Additional Considerations...........................................................................................240

CPM4 GTY Locations............................................................................................................... 242

Appendix B: GT Selection and Pin Planning for CPM5............................243

General Guidance for CPM5 ..................................................................................................244

Guidance for CPM5 in Specifically Identified Engineering Sample Devices.....................247

Guidance for CPM5 Migration from Specifically Identified Engineering Sample

Devices.................................................................................................................................251

CPM5 GTYP Locations............................................................................................................. 251

Appendix C: Debugging...........................................................................................253

Finding Help on Xilinx.com.................................................................................................... 253

PCIe Link Debug Enablement................................................................................................ 254

Appendix D: Using the High Speed Debug Port Over PCIe for

Design Debug........................................................................................................... 261

Overview...................................................................................................................................261

Implementing the HSDP-over-PCIe Example Design......................................................... 268

Appendix E: Limitations for CPM4 and CPM5..............................................273

Appendix F: Migrating............................................................................................. 275

Migrating to CPM4.................................................................................................................. 275

Migrating to CPM5.................................................................................................................. 279

Appendix G: Additional Resources and Legal Notices........................... 285

Xilinx Resources.......................................................................................................................285

Documentation Navigator and Design Hubs...................................................................... 285

References................................................................................................................................285

Revision History.......................................................................................................................286

Please Read: Important Legal Notices................................................................................. 287

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 3

Send Feedback

Chapter 1

Overview

Navigating Content by Design Process

Xilinx® documentaon is organized around a set of standard design processes to help you nd

relevant content for your current development task. All Versal® ACAP design process Design

Hubs and the Design Flow Assistant materials can be found on the Xilinx.com website. This

document covers the following design processes:

•System and Soluon Planning: Idenfying the components, performance, I/O, and data

transfer requirements at a system level. Includes applicaon mapping for the soluon to PS,

PL, and AI Engine. Topics in this document that apply to this design process include:

•Introducon to the CPM4

•Introducon to the CPM5

•Use Modes

•Embedded Soware Development: Creang the soware plaorm from the hardware

plaorm and developing the applicaon code using the embedded CPU. Also covers XRT and

Graph APIs. The topic in this document that applies to this design process include:

•Register Space

•Host Soware Development: Developing the applicaon code, accelerator development,

including library, XRT, and Graph API use. The topic in this document that applies to this

design process include:

•Register Space

•Hardware, IP, and Plaorm Development: Creang the PL IP blocks for the hardware

plaorm, creang PL kernels, funconal simulaon, and evaluang the Vivado® ming,

resource use, and power closure. Also involves developing the hardware plaorm for system

integraon. Topics in this document that apply to this design process include:

•Chapter 5: Design Flow Steps

•Appendix A: GT Selecon and Pin Planning for CPM4

•Appendix B: GT Selecon and Pin Planning for CPM5

•PCIe Link Debug Enablement

Chapter 1: Overview

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 4

Send Feedback

Introduction to the CPM4

The integrated block for PCIe® Rev. 4.0 with DMA and CCIX Rev. 1.0 (CPM4) consists of two

PCIe® controllers, DMA features, CCIX features, and network on chip (NoC) integraon. The

Versal® ACAP CPM Mode for PCI Express enables direct access to the two high-performance,

independently customizable PCIe controllers. The CPM4 uses up to 16 Versal device GTY

channels over the XPIPE. Applicaon designs can also interface to the CPM4 with so logic and

clocking resources in the programmable logic. All feature references are applicable to both

instances of CPM4 PCIe controllers, with the following excepons:

• CPM4 PCIe Controller 0 supports up to x16 operaon, and CPM4 PCIe Controller 1 supports

up to x8 operaon.

• CPM4 PCIe Controller 1 with up to x8 support is available only when CPM4 PCIe Controller 0

is congured with 8 lanes or fewer.

• The CPM4 DMA features are supported only with CPM4 PCIe Controller 0. For more

informaon about CPM4 DMA features, see the Versal ACAP CPM DMA and Bridge Mode for

PCI Express Product Guide (PG347).

Chapter 1: Overview

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 5

Send Feedback

Figure 1: CPM4 Sub-Block for PCIe Function (CPM4 PCIE)

CPM4 PCIe Controller #1

CPM4 PCIe Controller #0

ARM(*) CoreSight I/F Module

AXI4-ST

Data

Link

Layer

Mux/Demux

Physical

Layer

Cfg

Reg

Space

Integrated PCIe RAM

Clock &

Reset

cfg

XPIPE Hard I/F

Up to 2 Quads

Gen1 (2.5 GT/s)

16b@125 MHz

Gen2 (5.0 GT/s)

16b@250 MHz

Gen3 (8.0 GT/s)

16b@500 MHz

Gen4 (16.0 GT/s)

32b@500 MHz

Transaction

Layer

(VC0,

CCIX VC1)

APB

Block

Program-

ming

Hard I/F

XPIPEIM

Init

Ctrl

AXI4-ST

Data

Link

Layer

Mux/Demux

CPM4 DMA

DMA Core

Physical

Layer

Cfg

Reg

Space

Integrated PCIe RAM

Clock &

Reset

cfg

512b

cfg

AXI4-ST TX

Integrated DMA RAM

AXI4-ST RX

512b

cfg

XPIPE Hard I/F

Up to 4 Quads

Gen1 (2.5 GT/s)

16b@125 MHz

Gen2 (5.0 GT/s)

16b@250 MHz

Gen3 (8.0 GT/s)

16b@500 MHz

Gen4 (16.0 GT/s)

32b@500 MHz

Transaction

Layer

(VC0,

CCIX VC1)

PCIe Core Clock In

PCIe Reset In

Global Event Inputs

XPIPEIM

Misc Port

512b

AXI4-

Bridge

DMA

AXI4-MM

Switch

Init

Ctrl

XPIPE

Static

Switch

Internal

Hard I/F

Programming Register Space

512b

cfg

512b

128b

512b 512b

32b

attr_*0

dbg_0_0 dbg_0_1

attr_dma_*

AXI4-MM

Master1

AXI4-MM

Master0

AXI4-MM

Slave0

Enhanced

AXI4-ST + CFG

Fabric I/F

64/128/256/

512b

CCIX TL Hard

I/F #1

256b

500/625/

781.25 MHz

Enhanced

AXI4-ST +

CFG + Misc

DMA I/Os

Fabric I/F

64/128/256/

512b

On Chip NOC

Hard I/F

62.5/125

CCIX TL

Hard I/F #0

256b

500/625/

781.25 MHz

To On Chip NOC

Hard I/F

32b AXI4-MM

Lite (MCAP)

To On Chip

NOC Hard I/F

32b AXI4-MM

Lite (MCAP)

attr_*1

dbg_1_0 dbg_1_1

32b

X22665-072320

Chapter 1: Overview

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 6

Send Feedback

The CPM4 PCIe controllers are designed to the PCI Express Base Specicaon Revision 4.0 and

support the Gen4 data rate (16 GT/s per lane). They also support the Gen1 (2.5 GT/s per lane),

Gen2 (5 GT/s per lane) and Gen3 (8 GT/s per lane) data rates, and can interoperate with

components that are compliant with all versions of the PCI Express Base Specicaon.

The CPM4 PCIe controllers are available through the Vivado IP catalog in the Vivado Integrated

Design Environment (IDE). The combinaon of the CPM4 PCIe controllers, the GTY, and clocking

implement all layers of the PCI Express protocol, and the conguraon space and controller.

Protocol Layers

The layers of the protocol are the AXI4-Stream layer, the transacon layer, the data link layer and

the physical layer, and they are described in subsequent secons.

AXI4-Stream Layer

The AXI4-Stream layer implements Xilinx-specic requirements. In the transmit or outbound

direcon, the AXI4 layer interfaces the transacon layer with two AXI4-Stream interfaces. In the

receive or inbound direcon, the transacon layer output is forwarded to two AXI4-Stream

interfaces. Applicaon designs can aach to the AXI4-Stream interfaces, exchange informaon

with the Versal® ACAP CPM Mode for PCI Express encoded as a Xilinx-specic streaming

protocol implementaon, and run on top of the industry standard AXI4-Stream interface. The

CPM4 PCIe controllers support management of up to 256 (extended tag) or 768 (10 bit Tag)

outstanding customer iniated read requests, as part of the streaming protocol. The AXI4-Stream

layer supports:

•Recepon and transmission of address translaon services (ATS) invalid requests, ATS invalid

compleons, ATS page requests and ATS PRG response message TLPs, which enable ATS to

be implemented in the fabric logic.

• AXI4-Stream interface widths of 64 bits, 128 bits, 256 bits and 512 bits.

Transaction Layer

The transacon layer is the upper layer of the PCI Express architecture, and its primary funcon

is to accept, buer, and forward transacon layer packets (TLPs). TLPs communicate informaon

with the use of memory, I/O, conguraon, and message transacons. To maximize the eciency

of communicaon between devices, the transacon layer enforces PCI-compliant transacon

ordering rules and supports relaxed ordering (RO) of received transacons. The transacon layer

also manages TLP buer space through credit-based ow control. The transacon layer

implements built-in tag management for transmied non-posted transacons. It also implements

cut-through forwarding of transacons in the transmit (or outbound) direcon.

Chapter 1: Overview

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 7

Send Feedback

CCIX Transaction Layer

The Cache Coherent Interconnect for Accelerators (CCIX) transacon layer requirements are

implemented by the oponal virtual channel 1 (VC1) in the design. Note that VC1 storage is in

addion to the PCI Express-compliant virtual channel 0 (VC0) storage. The CCIX transacon layer

interfaces with the CCIX protocol layer is implemented externally to the PCIe ports over the

CCIX transacon layer (ARM CXS) hard interface. For more informaon, see the Versal ACAP CPM

CCIX Architecture Manual (AM016).

Data Link Layer

The data link layer acts as an intermediate stage between the transacon layer and the physical

layer. Its primary responsibility is to provide a reliable mechanism for the exchange of informaon

between two components on a link. This includes data exchange (TLPs), error detecon and

recovery, inializaon services and the generaon and consumpon of data link layer packets

(DLLPs). DLLPs are used to transfer informaon between data link layers of two directly

connected components on the link. DLLPs convey informaon, such as power management, ow

control, and TLP acknowledgments. The data link layer supports 32 kilobyte replay buers and

the feature DLLP.

Physical Layer

The physical layer interfaces the data link layer with signaling technology for link data

interchange, and is subdivided into the logical sub-block and the electrical sub-block.

• The logical sub-block frames and de-frames TLPs and DLLPs. It also implements the link

training and status state machine (LTSSM), which handles link inializaon, training, and

maintenance. Scrambling and descrambling of data (for Gen1/Gen2/Gen3/Gen4 operaon) is

also performed in this sub-block.

• The electrical sub-block denes the input and output buer characteriscs that interface the

device to the PCIe link. The physical layer also supports lane reversal (for mul-lane designs)

and lane polarity inversion, as required by the PCI Express Base Specicaon 4.0 (hps://

www.pcisig.com/specicaons).

Data exchange with the other components on the link occurs over the serial lines of one or more

gigabit transceivers (GTs), which expose parallel interfaces at lower clock frequencies to the PCIe

controller. For Gen1, Gen2, Gen3 and Gen4 operaon, the physical layer is up-conguraon

capable in the downstream port mode only.

Standards

The CPM4 block adheres to the following standards:

•PCI Express Base Specicaon 4.0 Version 1.0, and Errata updates (available at hp://

pcisig.com/specicaons).

Chapter 1: Overview

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 8

Send Feedback

• Cache Coherent Interconnect for Accelerators (CCIX) Transport Specicaon 1.0 (available at

hp://www.ccixconsorum.com).

Features

• Support for the following PCI Express architecture components:

○PCI Express Endpoint, Legacy Endpoint

○Root Port

○Switch Upstream and Downstream Ports

• 2.5 GT/s, 5.0 GT/s and 8.0 GT/s line rates with x1, x2, x4, x8, and x16 lane operaon.

• 16.0 GT/s line rate with x1, x2, x4, x8 lane operaon.

• CCIX support in PCI Express and EDR PHY Modes

○PCI Express support for Gen4x4, and Gen4x8

• Advanced Error Reporng (AER) and End-to-End CRC (ECRC)

• Two PCI Express virtual channels

○One PCI Express compliant virtual channel, eight trac classes

○One CCIX compliant virtual channel

• Support for mulple funcons and Single-Root IO Virtualizaon (SR-IOV)

○Up to 4 physical funcons

○Up to 252 virtual funcons

• PASID Prex capability supported

• Built-in lane reversal and receiver lane-lane de-skew

• 3 x 64-bit or 6 x 32-bit Base Address Registers (BARs) that are fully congurable

○Expansion ROM BAR supported

• All Interrupt types are supported:

○INTx

○32 mul-vector MSI capability

○MSI-X capability with up to 2048 vectors with oponal built-in vector tables

• Features that enable high-performance applicaons include:

○AXI4-Stream TLP Straddle on Requester Compleon Interface

○Address Translaon Services (ATS) and Page Request Interface (PRI) Messaging

Chapter 1: Overview

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 9

Send Feedback

○Atomic Operaon Transacons Support

○Transacon Tag Scaling as Completer

○Flow Control Scaling

• Several ease of use and congurable features are supported:

○BAR and ID based ltering of received transacons

○Oponal ASPM support for endpoint port types only; ASPM is not supported for other

port types.

○Conguraon extend interface

○AXI4-Stream interfaces address align mode

○Debug and diagnoscs interface

Chapter 1: Overview

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 10

Send Feedback

Introduction to the CPM5

Figure 2: CPM5 Sub-Block for PCIe Function (CPM5 PCIE)

CPM5 PCIe Controller #1

CPM5 PCIe Controller #0

ARM(*) CoreSight I/F Module

AXI4-ST

Data

Link

Layer

Mux/Demux

Physical

Layer

Cfg

Reg

Space

Integrated PCIe RAM

Clock &

Reset

cfg

XPIPE Hard I/F

Up to 2 Quads

Gen1 (2.5 GT/s)

16b@125 MHz

Gen2 (5.0 GT/s)

16b@250 MHz

Gen3 (8.0 GT/s)

16b@500 MHz

Gen4 (16.0 GT/s)

32b@500 MHz

Transaction

Layer

(VC0,

CCIX VC1)

APB

Block

Program-

ming

Hard I/F

XPIPEIM

Init

Ctrl

AXI4-ST

Data

Link

Layer

Mux/Demux

Physical

Layer

Cfg

Reg

Space

Integrated PCIe RAM

Clock &

Reset

cfg

512b

cfg

XPIPE Hard I/F

Up to 4 Quads

Gen1 (2.5 GT/s)

16b@125 MHz

Gen2 (5.0 GT/s)

16b@250 MHz

Gen3 (8.0 GT/s)

16b@500 MHz

Gen4 (16.0 GT/s)

32b@500 MHz

Transaction

Layer

(VC0,

CCIX VC1)

PCIe Core Clock In

PCIe Reset In

Global Event Inputs

XPIPEIM

512b

Init

Ctrl

XPIPE

Static

Switch

Internal

Hard I/F

Programming Register Space

512b

cfg

512b

32b

attr_*0

dbg_0_0 dbg_0_1

attr_dma_*

Enhanced

AXI4-ST + CFG

Fabric I/F

CCIX TL Hard

I/F #1

256b

Enhanced

AXI4-ST +

CFG + Misc

DMA I/Os

Fabric I/F

64/128/256/

512b

On Chip NOC

Hard I/F

CCIX TL

Hard I/F #0

256b

To On Chip NOC

Hard I/F

32b AXI4-MM

Lite (MCAP)

To On Chip

NOC Hard I/F

32b AXI4-MM

Lite (MCAP)

attr_*1

dbg_1_0 dbg_1_1

32b

CPM5 DMA

DMA Core

512b

cfg

AXI4-ST TX AXI4-ST RX Misc Port

512b

AXI4-

Bridge

DMA

AXI4-MM

Switch

128b

512b 512b

AXI4-MM

Master1

AXI4-MM

Master0

AXI4-MM

Slave0

CPM5 DMA

DMA Core

512b

cfg

AXI4-ST TX AXI4-ST RX Misc Port

512b

AXI4-

Bridge

DMA

AXI4-MM

Switch

128b

512b 512b

AXI4-MM

Master1

AXI4-MM

Master0

AXI4-MM

Slave0

On Chip NOC

Or Fabric IF

X22665-072320

Chapter 1: Overview

PG346 (v3.3) November 16, 2022 www.xilinx.com

CPM Mode for PCI Express 11

Send Feedback