Xilinx Versal ACAP CPM4 User manual

Versal ACAP CPM Mode for
PCI Express
Product Guide
Vivado Design Suite
PG346 (v3.3) November 16, 2022
Xilinx is creating an environment where employees, customers, and
partners feel welcome and included. To that end, we’re removing non-
inclusive language from our products and related collateral. We’ve
launched an internal initiative to remove language that could exclude
people or reinforce historical biases, including terms embedded in our
software and IPs. You may still find examples of non-inclusive language
in our older products as we work to make these changes and align
with evolving industry standards. Follow this link for more
information.

Table of Contents
Chapter 1: Overview......................................................................................................4
Navigating Content by Design Process.................................................................................... 4
Introduction to the CPM4...........................................................................................................5
Introduction to the CPM5.........................................................................................................11
Use Modes..................................................................................................................................16
Licensing and Ordering............................................................................................................22
Chapter 2: Tandem Configuration.......................................................................23
Overview.....................................................................................................................................23
Enable the Tandem Configuration Solution...........................................................................24
Deliver Programming Images to Silicon................................................................................ 30
Tandem Configuration Performance......................................................................................31
Design Operation...................................................................................................................... 31
Loading Tandem PCIe for Stage 2...........................................................................................32
Known Issues and Limitations.................................................................................................36
Chapter 3: Product Specification......................................................................... 38
Performance.............................................................................................................................. 38
Minimum Device Requirements..............................................................................................39
Port Descriptions.......................................................................................................................40
Register Space......................................................................................................................... 122
Chapter 4: Designing with the Core................................................................. 124
Clocking.................................................................................................................................... 124
Resets........................................................................................................................................126
AXI4-Stream Interface Description....................................................................................... 126
Chapter 5: Design Flow Steps...............................................................................188
Customizing and Generating the CIPS IP Core for CPM4.................................................. 188
Customizing and Generating the CIPS IP Core for CPM5.................................................. 211
Appendix A: GT Selection and Pin Planning for CPM4............................237
PG346 (v3.3) November 16, 2022 www.xilinx.com
CPM Mode for PCI Express 2
Send Feedback

CPM4 GT Selection.................................................................................................................. 238
CPM4 Additional Considerations...........................................................................................240
CPM4 GTY Locations............................................................................................................... 242
Appendix B: GT Selection and Pin Planning for CPM5............................243
General Guidance for CPM5 ..................................................................................................244
Guidance for CPM5 in Specifically Identified Engineering Sample Devices.....................247
Guidance for CPM5 Migration from Specifically Identified Engineering Sample
Devices.................................................................................................................................251
CPM5 GTYP Locations............................................................................................................. 251
Appendix C: Debugging...........................................................................................253
Finding Help on Xilinx.com.................................................................................................... 253
PCIe Link Debug Enablement................................................................................................ 254
Appendix D: Using the High Speed Debug Port Over PCIe for
Design Debug........................................................................................................... 261
Overview...................................................................................................................................261
Implementing the HSDP-over-PCIe Example Design......................................................... 268
Appendix E: Limitations for CPM4 and CPM5..............................................273
Appendix F: Migrating............................................................................................. 275
Migrating to CPM4.................................................................................................................. 275
Migrating to CPM5.................................................................................................................. 279
Appendix G: Additional Resources and Legal Notices........................... 285
Xilinx Resources.......................................................................................................................285
Documentation Navigator and Design Hubs...................................................................... 285
References................................................................................................................................285
Revision History.......................................................................................................................286
Please Read: Important Legal Notices................................................................................. 287
PG346 (v3.3) November 16, 2022 www.xilinx.com
CPM Mode for PCI Express 3
Send Feedback

Chapter 1
Overview
Navigating Content by Design Process
Xilinx® documentaon is organized around a set of standard design processes to help you nd
relevant content for your current development task. All Versal® ACAP design process Design
Hubs and the Design Flow Assistant materials can be found on the Xilinx.com website. This
document covers the following design processes:
•System and Soluon Planning: Idenfying the components, performance, I/O, and data
transfer requirements at a system level. Includes applicaon mapping for the soluon to PS,
PL, and AI Engine. Topics in this document that apply to this design process include:
•Introducon to the CPM4
•Introducon to the CPM5
•Use Modes
•Embedded Soware Development: Creang the soware plaorm from the hardware
plaorm and developing the applicaon code using the embedded CPU. Also covers XRT and
Graph APIs. The topic in this document that applies to this design process include:
•Register Space
•Host Soware Development: Developing the applicaon code, accelerator development,
including library, XRT, and Graph API use. The topic in this document that applies to this
design process include:
•Register Space
•Hardware, IP, and Plaorm Development: Creang the PL IP blocks for the hardware
plaorm, creang PL kernels, funconal simulaon, and evaluang the Vivado® ming,
resource use, and power closure. Also involves developing the hardware plaorm for system
integraon. Topics in this document that apply to this design process include:
•Chapter 5: Design Flow Steps
•Appendix A: GT Selecon and Pin Planning for CPM4
•Appendix B: GT Selecon and Pin Planning for CPM5
•PCIe Link Debug Enablement
Chapter 1: Overview
PG346 (v3.3) November 16, 2022 www.xilinx.com
CPM Mode for PCI Express 4
Send Feedback

Introduction to the CPM4
The integrated block for PCIe® Rev. 4.0 with DMA and CCIX Rev. 1.0 (CPM4) consists of two
PCIe® controllers, DMA features, CCIX features, and network on chip (NoC) integraon. The
Versal® ACAP CPM Mode for PCI Express enables direct access to the two high-performance,
independently customizable PCIe controllers. The CPM4 uses up to 16 Versal device GTY
channels over the XPIPE. Applicaon designs can also interface to the CPM4 with so logic and
clocking resources in the programmable logic. All feature references are applicable to both
instances of CPM4 PCIe controllers, with the following excepons:
• CPM4 PCIe Controller 0 supports up to x16 operaon, and CPM4 PCIe Controller 1 supports
up to x8 operaon.
• CPM4 PCIe Controller 1 with up to x8 support is available only when CPM4 PCIe Controller 0
is congured with 8 lanes or fewer.
• The CPM4 DMA features are supported only with CPM4 PCIe Controller 0. For more
informaon about CPM4 DMA features, see the Versal ACAP CPM DMA and Bridge Mode for
PCI Express Product Guide (PG347).
Chapter 1: Overview
PG346 (v3.3) November 16, 2022 www.xilinx.com
CPM Mode for PCI Express 5
Send Feedback

Figure 1: CPM4 Sub-Block for PCIe Function (CPM4 PCIE)
CPM4 PCIe Controller #1
CPM4 PCIe Controller #0
ARM(*) CoreSight I/F Module
AXI4-ST
Data
Link
Layer
Mux/Demux
Physical
Layer
Cfg
Reg
Space
Integrated PCIe RAM
Clock &
Reset
cfg
XPIPE Hard I/F
Up to 2 Quads
Gen1 (2.5 GT/s)
16b@125 MHz
Gen2 (5.0 GT/s)
16b@250 MHz
Gen3 (8.0 GT/s)
16b@500 MHz
Gen4 (16.0 GT/s)
32b@500 MHz
Transaction
Layer
(VC0,
CCIX VC1)
APB
Block
Program-
ming
Hard I/F
XPIPEIM
Init
Ctrl
AXI4-ST
Data
Link
Layer
Mux/Demux
CPM4 DMA
DMA Core
Physical
Layer
Cfg
Reg
Space
Integrated PCIe RAM
Clock &
Reset
cfg
512b
512b
512b
cfg
AXI4-ST TX
Integrated DMA RAM
AXI4-ST RX
512b
512b
cfg
XPIPE Hard I/F
Up to 4 Quads
Gen1 (2.5 GT/s)
16b@125 MHz
Gen2 (5.0 GT/s)
16b@250 MHz
Gen3 (8.0 GT/s)
16b@500 MHz
Gen4 (16.0 GT/s)
32b@500 MHz
Transaction
Layer
(VC0,
CCIX VC1)
PCIe Core Clock In
PCIe Reset In
Global Event Inputs
XPIPEIM
Misc Port
512b
512b
AXI4-
MM
Bridge
DMA
AXI4-MM
Switch
Init
Ctrl
XPIPE
Static
Switch
PS
Internal
Hard I/F
Programming Register Space
512b
512b
cfg
512b
128b
128b
128b
512b 512b
32b
attr_*0
dbg_0_0 dbg_0_1
attr_dma_*
AXI4-MM
Master1
AXI4-MM
Master0
AXI4-MM
Slave0
Enhanced
AXI4-ST + CFG
+
Fabric I/F
64/128/256/
512b
CCIX TL Hard
I/F #1
256b
500/625/
781.25 MHz
Enhanced
AXI4-ST +
CFG + Misc
DMA I/Os
Fabric I/F
64/128/256/
512b
To
On Chip NOC
Hard I/F
62.5/125
CCIX TL
Hard I/F #0
256b
500/625/
781.25 MHz
To On Chip NOC
Hard I/F
32b AXI4-MM
Lite (MCAP)
To On Chip
NOC Hard I/F
32b AXI4-MM
Lite (MCAP)
attr_*1
dbg_1_0 dbg_1_1
32b
RX
TX
RX
TX
X22665-072320
Chapter 1: Overview
PG346 (v3.3) November 16, 2022 www.xilinx.com
CPM Mode for PCI Express 6
Send Feedback

The CPM4 PCIe controllers are designed to the PCI Express Base Specicaon Revision 4.0 and
support the Gen4 data rate (16 GT/s per lane). They also support the Gen1 (2.5 GT/s per lane),
Gen2 (5 GT/s per lane) and Gen3 (8 GT/s per lane) data rates, and can interoperate with
components that are compliant with all versions of the PCI Express Base Specicaon.
The CPM4 PCIe controllers are available through the Vivado IP catalog in the Vivado Integrated
Design Environment (IDE). The combinaon of the CPM4 PCIe controllers, the GTY, and clocking
implement all layers of the PCI Express protocol, and the conguraon space and controller.
Protocol Layers
The layers of the protocol are the AXI4-Stream layer, the transacon layer, the data link layer and
the physical layer, and they are described in subsequent secons.
AXI4-Stream Layer
The AXI4-Stream layer implements Xilinx-specic requirements. In the transmit or outbound
direcon, the AXI4 layer interfaces the transacon layer with two AXI4-Stream interfaces. In the
receive or inbound direcon, the transacon layer output is forwarded to two AXI4-Stream
interfaces. Applicaon designs can aach to the AXI4-Stream interfaces, exchange informaon
with the Versal® ACAP CPM Mode for PCI Express encoded as a Xilinx-specic streaming
protocol implementaon, and run on top of the industry standard AXI4-Stream interface. The
CPM4 PCIe controllers support management of up to 256 (extended tag) or 768 (10 bit Tag)
outstanding customer iniated read requests, as part of the streaming protocol. The AXI4-Stream
layer supports:
•Recepon and transmission of address translaon services (ATS) invalid requests, ATS invalid
compleons, ATS page requests and ATS PRG response message TLPs, which enable ATS to
be implemented in the fabric logic.
• AXI4-Stream interface widths of 64 bits, 128 bits, 256 bits and 512 bits.
Transaction Layer
The transacon layer is the upper layer of the PCI Express architecture, and its primary funcon
is to accept, buer, and forward transacon layer packets (TLPs). TLPs communicate informaon
with the use of memory, I/O, conguraon, and message transacons. To maximize the eciency
of communicaon between devices, the transacon layer enforces PCI-compliant transacon
ordering rules and supports relaxed ordering (RO) of received transacons. The transacon layer
also manages TLP buer space through credit-based ow control. The transacon layer
implements built-in tag management for transmied non-posted transacons. It also implements
cut-through forwarding of transacons in the transmit (or outbound) direcon.
Chapter 1: Overview
PG346 (v3.3) November 16, 2022 www.xilinx.com
CPM Mode for PCI Express 7
Send Feedback

CCIX Transaction Layer
The Cache Coherent Interconnect for Accelerators (CCIX) transacon layer requirements are
implemented by the oponal virtual channel 1 (VC1) in the design. Note that VC1 storage is in
addion to the PCI Express-compliant virtual channel 0 (VC0) storage. The CCIX transacon layer
interfaces with the CCIX protocol layer is implemented externally to the PCIe ports over the
CCIX transacon layer (ARM CXS) hard interface. For more informaon, see the Versal ACAP CPM
CCIX Architecture Manual (AM016).
Data Link Layer
The data link layer acts as an intermediate stage between the transacon layer and the physical
layer. Its primary responsibility is to provide a reliable mechanism for the exchange of informaon
between two components on a link. This includes data exchange (TLPs), error detecon and
recovery, inializaon services and the generaon and consumpon of data link layer packets
(DLLPs). DLLPs are used to transfer informaon between data link layers of two directly
connected components on the link. DLLPs convey informaon, such as power management, ow
control, and TLP acknowledgments. The data link layer supports 32 kilobyte replay buers and
the feature DLLP.
Physical Layer
The physical layer interfaces the data link layer with signaling technology for link data
interchange, and is subdivided into the logical sub-block and the electrical sub-block.
• The logical sub-block frames and de-frames TLPs and DLLPs. It also implements the link
training and status state machine (LTSSM), which handles link inializaon, training, and
maintenance. Scrambling and descrambling of data (for Gen1/Gen2/Gen3/Gen4 operaon) is
also performed in this sub-block.
• The electrical sub-block denes the input and output buer characteriscs that interface the
device to the PCIe link. The physical layer also supports lane reversal (for mul-lane designs)
and lane polarity inversion, as required by the PCI Express Base Specicaon 4.0 (hps://
www.pcisig.com/specicaons).
Data exchange with the other components on the link occurs over the serial lines of one or more
gigabit transceivers (GTs), which expose parallel interfaces at lower clock frequencies to the PCIe
controller. For Gen1, Gen2, Gen3 and Gen4 operaon, the physical layer is up-conguraon
capable in the downstream port mode only.
Standards
The CPM4 block adheres to the following standards:
•PCI Express Base Specicaon 4.0 Version 1.0, and Errata updates (available at hp://
pcisig.com/specicaons).
Chapter 1: Overview
PG346 (v3.3) November 16, 2022 www.xilinx.com
CPM Mode for PCI Express 8
Send Feedback

• Cache Coherent Interconnect for Accelerators (CCIX) Transport Specicaon 1.0 (available at
hp://www.ccixconsorum.com).
Features
• Support for the following PCI Express architecture components:
○PCI Express Endpoint, Legacy Endpoint
○Root Port
○Switch Upstream and Downstream Ports
• 2.5 GT/s, 5.0 GT/s and 8.0 GT/s line rates with x1, x2, x4, x8, and x16 lane operaon.
• 16.0 GT/s line rate with x1, x2, x4, x8 lane operaon.
• CCIX support in PCI Express and EDR PHY Modes
○PCI Express support for Gen4x4, and Gen4x8
• Advanced Error Reporng (AER) and End-to-End CRC (ECRC)
• Two PCI Express virtual channels
○One PCI Express compliant virtual channel, eight trac classes
○One CCIX compliant virtual channel
• Support for mulple funcons and Single-Root IO Virtualizaon (SR-IOV)
○Up to 4 physical funcons
○Up to 252 virtual funcons
• PASID Prex capability supported
• Built-in lane reversal and receiver lane-lane de-skew
• 3 x 64-bit or 6 x 32-bit Base Address Registers (BARs) that are fully congurable
○Expansion ROM BAR supported
• All Interrupt types are supported:
○INTx
○32 mul-vector MSI capability
○MSI-X capability with up to 2048 vectors with oponal built-in vector tables
• Features that enable high-performance applicaons include:
○AXI4-Stream TLP Straddle on Requester Compleon Interface
○Address Translaon Services (ATS) and Page Request Interface (PRI) Messaging
Chapter 1: Overview
PG346 (v3.3) November 16, 2022 www.xilinx.com
CPM Mode for PCI Express 9
Send Feedback

○Atomic Operaon Transacons Support
○Transacon Tag Scaling as Completer
○Flow Control Scaling
• Several ease of use and congurable features are supported:
○BAR and ID based ltering of received transacons
○Oponal ASPM support for endpoint port types only; ASPM is not supported for other
port types.
○Conguraon extend interface
○AXI4-Stream interfaces address align mode
○Debug and diagnoscs interface
Chapter 1: Overview
PG346 (v3.3) November 16, 2022 www.xilinx.com
CPM Mode for PCI Express 10
Send Feedback