Caen ELS DAMC-FMC2ZUP User manual

DAMC-FMC2ZUP User’s Manual
1
DAMC-FMC2ZUP
Zynq Ultrascale+ MPSoC based
Dual FMC/FMC+ Carrier Board with
MicroTCA.4 D1.1 support
User’s Manual
All Rights Reserved
© CAEN ELS s.r.l.
Rev. 1.0 –April 2021
MTCA.4 –MicroTCA for Physics

DAMC-FMC2ZUP User’s Manual
2
This product is licensed by
CAEN ELS s.r.l.
in AREA Science Park
S.S. 14 km 163,5 –34149 Basovizza (TS)
Italy
Mail: [email protected]
Web: www.caenels.com

DAMC-FMC2ZUP User’s Manual
3
Table of Contents
1. INTRODUCTION................................................................................................6
1.1 DAMC-FMC2ZUP DESCRIPTION ..................................................................6
1.2 DAMC-FMC2ZUP FEATURES.......................................................................7
1.3 ORDERING OPTIONS ........................................................................................9
1.4 DAMC-FMC2ZUP COMPONENTS OVERVIEW .............................................10
2. DAMC-FMC2ZUP ARCHITECTURE...........................................................12
2.1 FUNCTIONAL BLOCK DIAGRAM ....................................................................12
2.2 CLOCK TREE ARCHITECTURE........................................................................18
2.3 POWER SUPPLY ARCHITECTURE ...................................................................20
2.4 JTAG CHAIN ................................................................................................21
2.5 FLASH PROGRAMMING................................................................................22
2.6 WHITE RABBIT SUPPORT ..............................................................................22
3. MODULE MANAGEMENT CONTROLLER...............................................23
3.1 SERIAL CONSOLE ..........................................................................................23
3.2 JTAG MULTIPLEXING ...................................................................................23
3.3 HPM UPDATE................................................................................................24
3.4 XMODEM UPDATE......................................................................................24
3.5 FPGA BOOT MODE SELECTION......................................................................24
3.6 INTERLOCK MULTIPLEXING...........................................................................24
3.7 AUTO-DETECTION OF VARIABLE FMC VOLTAGE (VADJ) ..............................25
3.8 RTM E-KEYING.............................................................................................25
3.9 RTM TEMPERATURE SENSORS ......................................................................25
4. FRONT PANEL AVAILABLE FEATURES..................................................26
4.1 MICROUSB FRONT-PANEL CONNECTOR........................................................27
5. APPENDIX.........................................................................................................28
5.1 EXTENSION POWER CONNECTORS ................................................................28
5.2 HIGH DENSITY MICROHDMI TYPE-D ...........................................................29
5.3 MMC COMMANDS ........................................................................................32
5.4 FMC/FMC+ CONNECTIONS ..........................................................................32

DAMC-FMC2ZUP User’s Manual
4
Document Revisions
Document Revision
Date
Comment
Preliminary Release
October, 2020
Preliminary Release
1.0
April, 2021
First Release

DAMC-FMC2ZUP User’s Manual
5
Safety Information
The following table shows the general environmental requirements for a correct
operation of referred board in this User’s Manual:
Environmental Conditions
Requirements
Environment
Indoor use
Operating Temperature
0°C to 50°C
Operating Humidity
20% to 80% RH (non-condensing)
Altitude
Up to 2000 m
Pollution degree
2
Storage Temperature
-10°C to 60°C
Storage Humidity
5% to 90% RH (non-condensing)
CAEN ELS will repair or replace any product within the guarantee period if
the Guarantor declares that the product is defective due to workmanship or materials
and has not been caused by mishandling, negligence on behalf of the user, accident or
any abnormal conditions or operations.
Please read carefully the manual before operating any part of the board.

DAMC-FMC2ZUP User’s Manual Introduction
6
1.Introduction
This chapter presents the overall characteristics and features of the DAMC-
FMC2ZUP FMC carrier board.
1.1 DAMC-FMC2ZUP Description
The DAMC-FMC2ZUP is a high-end FMC/FMC+ carrier in MicroTCA.4 form
factor based on the new family of ZYNQ Ultrascale+ MPSoCs. The host FPGA is the
Xilinx ZU11EG-L2 or the ZU19EG-L2. The PS (Processing System) section offers a
quad-core ARM Cortex-A53 operating up to 1.333 GHz, a dual core ARM-R5 real-
time processor running up to 533 MHz and Mali-400 MP2 graphics up to 600 Mhz.
The board features extensive MicroTCA.4 backplane connectivity. PCI express
Gen.3 x4 link is supported on port 4-7 and can be expanded to x8 using ports 8-11 in
systems that support this mode of operation.
The card supports full multi-gigabit point-to-point link connectivity (low-
latency links) on ports 12-15. This can eventually be expanded to include ports 8-11
(PCIe is restricted to x4 configuration in this use-case) to accommodate non-
conventional backplane topologies.
Gigabit Ethernet port 0 is directly connected to the PS subsection of the FPGA
while port 1 connects to the PL (Programmable Logic) MGTs to allow
implementation of a SGMII interface over backplane.
Two transceivers from PS are connected to AMC port 2 and 3, allowing
attachment of up to two standard off-the-shelf MicroTCA SATA cards with hard
drives or SSDs.
All eight M-LVDS Timing/Trigger signals on ports 17-20 are accessible.
The four TCLK lines and the Fabric FCLKA are all connected to the internal
clock distribution network. A flexible clock scheme leveraging a 16-channel bi-
directional cross point switch allows to receive and/or drive any clock from/to the
backplane (TCLKA, TLCKB, TCLKC, TCLKD) and from/to the FMCs (bi-
directional clocks).
The board features two DDR4 interfaces, a 64-bits wide 4GB memory directly
attached to the PS and a 16-bits wide 1GB memory connected to the PL, both
memories operate by default at 2400 MT/s, with the option to increase the data-rate to
2666 MT/s for the PL when the board does not operate on low-power core voltage.

DAMC-FMC2ZUP User’s Manual Introduction
7
The card provides two FMC sockets, one FMC+ (VITA 57.4 compliant) and one
FMC High Pin Count (VITA 57.1 compliant). Both connectors populate all LA and
HA differential pair lines and can operate on VADJ level in the range from 1.2V to
1.8V. A total of 24 transceivers are routed to the FMC+ socket and 8 to the FMC HPC
connector. Front panel connectivity comprises a Micro HDMI type-D connector with
a proprietary pin assignment offering a Gigabit Ethernet interface (over external SFP),
clock input and two trigger in/out signals. The front panel Gigabit Ethernet connection
can be used in conjunction with the available HW infrastructure provided on the board
to implement a White Rabbit endpoint.
A fully fledged USB Type-C connector implementing Alternate Mode Display
Port is connected to the PS, this allows to provide USB HS, USB SS 3.0, DisplayPort
and power delivery capabilities on a single compact interface. This offers the
possibility to omit a CPU boards in the MicroTCA systems.
The RTM interface is designed according to class D1.1 and implements the
full set of 42 LVDS lines and 2 MGT links. The board supports all existing digital
RTMs from DESY (i.e. DRTM-AD84, DRTM-VM2, DRTM-PZT4).
The ARM Cortex-A53 processor in the MPSoC can run GNU/Linux from a
MicroSD card on the front panel or from the embedded 8GB eMMC flash memory.
The Zynq Ultrascale+ family of FPGAs is supported by all modern development
tools, such as Vivado, HLS, Yocto, Petalinux, SDSoC and SDAccel, this makes
DAMC-FMC2ZUP capable of addressing the ever-growing need of processing power
while also reducing development time.
1.2 DAMC-FMC2ZUP Features
•Two FPGA Low-Power Xilinx MPSoC Ultrascale+ options are available:
MPSoC FPGA
ZU11EG-L2
ZU19EG-L2
Logic Cells
653 k
1.143 k
DSP Slices
2.928
1.968
BlockRAM
21.1 MiB
34.6 MiB
UltraRAM
22.5 MiB
36.0 MiB
PCIe
Gen. 3 x4 or x8
Ethernet MAC
100G
ARMs
1.333 GHz Quad-Core ARM Cortex-A53,
MALI-400 GPU, 600 Mhz Dual-core ARM
Cortex-R5
GTY
transceivers
16 GTY transceivers (28.21 Gbps) routed to
FMC+ transceivers
GTH
32 GTH Transceivers (16.375 Gbps/ 12.5 Gbps

DAMC-FMC2ZUP User’s Manual Introduction
8
Transceivers
low-power) routed to FMC HPC, backplane,
Front Panel, Zone3 etc.
GTR
Transceivers
4 GTR Transceivers (6.0 Gbps) routed to SATA
links and FMC connector
•FPGA core voltage can be set to low-power (0.72V) or standard power
(0.85V) operation via MMC user commands, this allows to switch between
low heat dissipation and higher performance
•FMC+ Slot providing 24 transceivers (16 GTY + 8 GTH) and full sets of LA
and HA differential pairs
•Standard FMC HPC socket with 8 transceivers and full LA and HA
connectivity
•USB Type-C connector providing DisplayPort and USB 3.0 interfaces. This
allows to set-up a stand-alone system with display and USB host capability
(may save a CPU module)
•Zone 3 Class D1.1 compliance with full interlock support
•64-bit 4 GiB DDR4 with 2400 MT/s connected to PS (accessible from PL via
AXI Bridge)
•16-bit 1 GiB DDR4 with 2666 MT/s (2400 MT/s in low-power mode)
connected to PL
•Flexible clock architecture provides access (receive/transmit) to all MicroTCA
backplane clocks (TCLKA, TCLKB, TCLKC, TCLKD, FMC bidirectional
clocks). 3 PLLs generate clocks for PS, Zone3 and FMCs
•White Rabbit Support (Input from front panel, output to backplane via M-
LVDS and TCLKs)
•SD-Card slot (connected to PS) accessible from the front panel
•8GB eMMC Memory
•Full MLVDS and interlock receive/transmit capability
•JTAG interface available from front panel accessible microUSB connector or
AMC backplane (requires JSM module). FPGAs, FMCs and RTM can be set
as target devices in the JTAG chain
•Full HPM update functionality

DAMC-FMC2ZUP User’s Manual Introduction
9
1.3 Ordering options
There are two ordering option for the two versions of the DAMC-FMC2ZUP
carrier board:
Description
Ordering Code
MicroTCA.4 Zynq UltraScale+ FMC+ Carrier
with XCZU11EG-L2FFVC1760E
DAMCFMC2ZUP1
MicroTCA.4 Zynq UltraScale+ FMC+ Carrier
with XCZU19EG-L2FFVC1760E
DAMCFMC2ZUP2

DAMC-FMC2ZUP User’s Manual Introduction
10
1.4 DAMC-FMC2ZUP Components Overview
This section details the type and location of components on the board.
Components on side #1 (top-side):
1. Main FPGA - Zynq Ultrascale+
2. FMC+ connector
3. FMC HPC connector
4. Secondary FPGA Spartan 7
5. DDR4 memory modules, PS (a), PL (b)
6. Power Section
7. Ethernet IC
8. Main JTAG Connector
9. FPGA PS - ARM JTAG for PS Connector
10. Extension Power Connectors
Figure 1-1: DAMC-FMC2ZUP - Top side
1
2
3
4
5a
5b
6
7
8
9
10

DAMC-FMC2ZUP User’s Manual Introduction
11
Components on side #2 (bottom-side):
1. MMC
2. MLVDS transceivers
3. Flash memories for FPGA configuration:
a. Zynq Ultrascale+
b. Spartan 7
4. Power Management
5. Clock Section
6. eMMC memory
Figure 1-2: DAMC-FMC2ZUP - Bottom side
1
3b
3a
2
4
5
6

DAMC-FMC2ZUP User’s Manual DAMC-FMC2ZUP Architecture
12
2.DAMC-FMC2ZUP Architecture
This chapter offers a technical overview of the many features implemented on
this board.
2.1 Functional Block Diagram
This section presents the block diagram of the connectivity implemented on the
board, excluding the clock network which is described in the next section. In the
following paragraphs are described the connections with the backplane, with
FMC/FMC+ connectors and with Zone3.
2.1.1 MicroTCA backplane
The following interconnections are available over the MicroTCA backplane:
Gigabit Ethernet (ports 0 and 1):
GbE interface on Port 0 is provided through a specialized RGMII to Ethernet
1000-baseX IC to guarantee fully compliance to the MicroTCA standard. The RGMII
communication is under control of the PS, thus allowing an easy implementation of
the required protocol stacks in software. The redundant GbE interface on Port 1 is
connected to Multi-Gigabit Transceiver (MGT) on the PL and can be used as a SGMII
interface to communicate with the redundant MCH.
SATA (ports 2 and 3):
Serial-ATA interfaces are directly connected to the PS MGTs. This gives the
user flexibility to store data from an Operating System running on the ARM Cores to
a Hard Disk Drive (HDD) or Solid-State Disk (SSD) residing on another AMC card.
For further details on this interface please refer to the backplane connectivity manual.

DAMC-FMC2ZUP User’s Manual DAMC-FMC2ZUP Architecture
13
PCIe over Fat Pipe (ports 4 to 7):
PCIe x4 Gen.3 (Gen.4 can be enabled but is not officially supported)
connectivity is achieved using the Hard IP blocks located in the PL area of the FPGA.
Connecting these ports to the PL MGTs instead of the PS ones offer many benefits
among which a higher data throughput (PS MGTs are limited to PCIe Gen.2). This
interface can be expanded to provide a x8 lanes link when ports 4 to 11 (Fat Pipe and
Extended Fat Pipe) are used in systems that support this configuration
1
.
Extended Fat Pipe (ports 8 to 11):
Ports 8 to 11 are routed to a dedicated MGT Quad on the PL section of the
FPGA. These lanes can be used to extend the PCIe interface or to implement any
other supported protocol (restrictions may apply depending on MicroTCA system
configuration).
Low Latency Links (ports 12 to 15):
As for the previous link these ports are also routed to a dedicated MGT Quad
of the PL. These lanes are usually implemented on the MicroTCA backplane as
board-to-board links and their usage is user’s application dependent.
M-LVDS (ports 17 to 20):
Ports 17 to 20 are implemented using specialized ICs and are accessible from
the PL section of the Main FPGA. The enable signal of the transmitters is under
control of the FPGA itself (optional factory configuration where RX19, TX19 and
RX20 transmitters are always disabled). Read-back of the MLVDS lanes is always
enabled and available to the FPGA logic.
1
x8 links are not possible on systems with redundant MCHs due to the non-transparent nature of the
PCIe switches in the bus enumeration process

DAMC-FMC2ZUP User’s Manual DAMC-FMC2ZUP Architecture
14
Figure 2-1: Block Diagram

DAMC-FMC2ZUP User’s Manual DAMC-FMC2ZUP Architecture
15
2.1.2 FMC+ (VITA 57.4)
The following interconnections are available over the FMC+ connector:
•LA (00-33) LVDS Differential pairs:
All the LA signals of the FMC interface are routed to the PL section of the FPGA.
Two FPGA banks are associated to this interface, LA-00 to LA-16 are connected to
one bank while LA-17 to LA-33 are routed to the other. LA-00, LA-01, LA-17 and
LA-18 are clock capable pins routed to dedicated inputs on the FPGA, these signals
have access to the clock resources on the PL.
•HA (00-23) LVDS Differential pairs:
All the HA signals of the FMC interface are routed to the PL section of the FPGA. All
the signals reside on the same I/O bank, HA-00, HA-01 and HA-17 are clock capable
pins and are routed to dedicated inputs on the FPGA allowing them to access the
clock resources of the PL.
•MGT interfaces:
A total of 24 MGT (6 full MGT Quads) are provided to the FMC connector. 16 GTY
Transceivers up to 28.21 Gb/s and 8 GTH Transceivers up to 16.375 Gb/s or 12.5
Gb/s (low power mode).
•CLK0-M2C and CLK1-M2C:
Both signals are available and connected on dedicated clock input resources on the PL
section of the FPGA. CLK0 resides in the same I/O bank as LA-00 to LA-16 and
CLK1 resides in the same I/O bank as LA-17 to LA-33.
•GBT-CLK0, GBT-CLK1 and GBT-CLKHS 2 to 5:
All signals are available and connected to the dedicated reference clock input
resources of the associated PL MGT Quads. CLK0 resides in the same Quad as MGT
lanes 0 to 3 and CLK1 resides in the same Quad as MGT lanes 4 to 7, CLKHS2
resides in the same Quad as MGT lanes 8 to 11 and so on.

DAMC-FMC2ZUP User’s Manual DAMC-FMC2ZUP Architecture
16
2.1.3 FMC HPC (VITA 57.1)
The following interconnections are available over the FMC connector:
•LA (00-33) LVDS Differential pairs:
All the LA signals of the FMC interface are routed to the PL section of the FPGA.
Two FPGA banks are associated to this interface, LA-00 to LA-16 are connected to
one bank while LA-17 to LA-33 are routed to the other. LA-00 and LA-17 are clock
capable pins routed to dedicated inputs on the FPGA, this gives them access to the
clock resources on the PL.
•HA (00-23) LVDS Differential pairs:
All the HA signals of the FMC interface are routed to the PL section of the FPGA. All
the signals reside on the same I/O bank, HA-00, HA-01 and HA-17 are clock capable
pins and are routed to dedicated inputs on the FPGA, giving them access to the clock
resources of the PL.
•MGT interfaces:
A total of 8 GTH MGTs (two full MGT Quads) are available on the FMC connector.
•CLK0-M2C and CLK1-M2C:
Both signals are available and connected on dedicated clock input resources on the PL
section of the FPGA. CLK0 resides in the same I/O bank as LA-00 to LA-16 and
CLK1 resides in the same I/O bank as LA-17 to LA-33.
•GBT-CLK0 and GBT-CLK1:
Both signals are available and connected to the dedicated reference clock input
resources on the PL MGT Quads. CLK0 resides in the same Quad as MGT lanes 0 to
3 and CLK1 resides in the same Quad as MGT lanes 4 to 7.

DAMC-FMC2ZUP User’s Manual DAMC-FMC2ZUP Architecture
17
2.1.4 Zone3 Connectivity
Zone3 connectivity in accordance to recommended pinout of Class D1.1 is
achieved as follows:
•Fixed LVDS outputs OUT0 to OUT2:
These links use specialized LVCMOS to LVDS drivers under full control of MMC
Stamp SoM (System on Module). The driving signals are provided by the CPLD
embedded on the MMC module.
•AMC TCLK and CLK1:
These signals are driven by the clock network as will be described in Section 2.2. To
comply the unpowered/unconnected interface requirements described in the
recommendation document, a LVDS buffer is used on both signal paths. The MMC
has control over the enable signal of these buffers as required by the eKeying process.
•RTM CLK1:
This signal is driven by the RTM module and fed to the clock network as will be
described in Section 2.2.
•GTP0-1_CLK_IN and GTP0-1_CLK_OUT:
Transceiver reference clocks from/to the RTM are part of the clock distribution
network and described in Section 2.2.
•GTP0 and GTP1 TX and RX pairs:
High speed transceiver lanes are routed to dedicated TX and RX interfaces on a PL
MGT Quad. This Quad is shared by the MGTs routed to the port 1 of the MicroTCA
backplane and to the Micro HDMI type D connector on the front panel.
•LVDS I/O differential pairs:
A total of 24 LVDS pairs are available for user defined application on the Zone3
connector, these signals are routed to the Spartan 7 FPGA and from there, with a
proprietary interface, to the main FPGA. This arrangement allows to interface to a
RTM card whose logic levels are 2.5V.
•Zone3 Management and Debug:
Presence, I2C bus and JTAG signals are routed to the MMC Stamp SoM.

DAMC-FMC2ZUP User’s Manual DAMC-FMC2ZUP Architecture
18
2.2 Clock Tree Architecture
The clock architecture is based on a bidirectional cross-point switch and 3
independent PLLs.
The cross-point switch is connected to:
•4 Telecommunication Clock signals (TCLKA, TCLKB, TCLKC and TCLKD)
provided on standard MicroTCA backplanes
•3 digital clocks to and from the Zone3 connector
•the 2 bidirectional clocks of each FMC modules (CLKBIDIR2/3)
•one of the output clocks of the White Rabbit section
•FPGA I/O pins with dedicated access to internal clock tree resources
•the main PLL located on the board.
The board is capable of driving and receiving clock signals from any of the 4
TCLK dedicated paths of the MicroTCA backplane and from the bidirectional clocks
of FMCs. The main PLL IC receives one input clock from the cross-point switch and
provides one of its outputs back to it. The second input of this component can be
externally provided through the Micro HDMI type D connector. Two of main PLL
output are routed to the inputs of the 2 secondary PLL, used to fed the PS on the Main
FPGA, and to the PLL towards the Zone3 connector. The other outputs are used as
reference for MGTs and to synchronize the PL section of the Main FPGA.
The secondary PLL listed as PS PLL in the previous block diagram is
responsible to provide the needed reference clocks to PS section of the Main FPGA.
Any modification in the register values of this component might impact negatively the
performances of the PS and in some cases prevent the ARM cores from booting up.
The Zone3 PLL provides the RTM related clocks and synchronization
between the secondary FPGA (Spartan 7) and Main FPGA to ensure the proper
functionality of the interconnection bus. The Fabric Clock (FCLKA) distributed by
the MCH in a MicroTCA system is connected, through a PCIe compliant jitter cleaner
IC, to a reference clock input of the MGT Quad used to implement the PCIe
connectivity on ports 4-7. Fixed frequency clock resources are connected to various
components to ensure proper operation and availability of a reference signal to the
programmable logic even before the local PLLs are accessed and configured by the
user code.
The PLLs are accessible over an I2C bus by both the MMC Stamp SoM and
Main FPGA PS.
The block diagram of the clock network is presented in the following figure:

DAMC-FMC2ZUP User’s Manual DAMC-FMC2ZUP Architecture
19
Figure 2-2: Clock Network Diagram

DAMC-FMC2ZUP User’s Manual DAMC-FMC2ZUP Architecture
20
2.3 Power Supply Architecture
The power supply section is implemented by a cascade of DC/DC converters and
LDOs, all of which are controlled by a PMBus manager IC, this ensures that the
proper sequencing, as required by the FPGA and IC specifications, is maintained both
at start-up and power-down. The PMBus manager offers full telemetry and
monitoring of the power rails (voltage and current readouts) and is responsible to
bring the board to a safe condition whenever a supply voltage is out of specifications.
This component also allows trimming the output voltages and it’s responsible to
set the VADJ rail to the voltage level required by the FMC modules. The MMC
Stamp SoM has full control over the power conversion stages through the PMBus
manager. By interfacing to the MCH the user can access the readout values for the
following power rails:
•0.72/0.85 V Main FPGA Core
•1.0 V Secondary FPGA Core
•TBD
A general overview of the power section is presented in Figure 2.3.
Figure 2-3: Power-Supply Section