Terasic TR10a-HL User manual

1

2
CONTENTS
Chapter 1 Overview................................................................................... 4
1.1 General Description............................................................................... 4
1.2 Key Features......................................................................................... 5
1.3 Block Diagram....................................................................................... 6
Chapter 2 Board Components ................................................................. 9
2.1 Board Overview..................................................................................... 9
2.2 Configuration, Status and Setup.......................................................... 10
2.3 General User Input/Output .................................................................. 13
2.4 Temperature Sensor and Fan Control.................................................. 15
2.5 Power Monitor..................................................................................... 17
2.6 Clock Circuit........................................................................................ 18
2.7 FLASH Memory................................................................................... 20
2.8 QDRII+ SRAM..................................................................................... 23
2.9 QSPF+ Ports....................................................................................... 36
2.10 PCI Express...................................................................................... 40
2.11 QSPF+ Ports..................................................................................... 43
Chapter 3 System Builder....................................................................... 45
3.1 Introduction......................................................................................... 45
3.2 General Design Flow........................................................................... 46
3.3 Using System Builder.......................................................................... 47
Chapter 4 Flash Programming................................................................. 53
4.1 CFI Flash Memory Map....................................................................... 53
4.2 FPGA Configure Operation.................................................................. 54
4.3 Flash Programming with Users Design................................................ 55
4.4 Restore Factory Settings..................................................................... 56
Chapter 5 Peripheral Reference Design................................................ 58
5.1 Configure Si5340A/B in RTL ............................................................... 58
5.2 Nios II control for SI5340/Temperature/Power..................................... 67
5.3 Fan Speed Control.............................................................................. 72
Chapter 6 Memory Reference Design.................................................... 75
6.1 QDRII+ SRAM Test ............................................................................. 75
6.2 QDRII+ SRAM Test by Nios II.............................................................. 79
Chapter 7 PCI Express Reference Design .............................................. 83
7.1 PCI Express System Infrastructure...................................................... 83
7.2 PC PCI Express Software SDK........................................................... 84
7.3 PCI Express Software Stack ............................................................... 85
7.4 PCIe Reference Design - Fundamental............................................... 93
7.5 PCIe Reference Design –QDRII+..................................................... 100
Chapter 8 Transceiver Verification ...................................................... 108
8.1 Function of the Transceiver Test Code .............................................. 108
8.2 Loopback Fixture............................................................................... 108
8.3 Testing................................................................................................110
Chapter 9 Additional Information......................................................... 112
Getting Help.............................................................................................112

3

4
Chapter 1
Overview
his chapter provides an overview of the TR10a-HL Development Board and
installation guide.
1.1 General Description
The Terasic TR10a-HLArria 10 GX FPGADevelopment Kit provides the ideal hardware
solutionfor designsthatdemandhigh capacityandbandwidthmemory interfacing,ultra-
low latency communication, and power efficiency. With a full-height, half length form-
factor package, the TR10a-HL is designed for the most demanding high-end
applications, empowered with the top-of-the-lineAltera Arria 10 GX, delivering the best
system-level integration and flexibility in the industry.
The Arria® 10 GX FPGAfeatures integrated transceivers that transfer at a maximum of
12.5 Gbps, allowing the TR10a-HL to be fully compliant with version 3.0 of the PCI
Express standard, as well as allowing an ultra low-latency, straight connections to four
external 40G QSFP+ modules. Not relying on an external PHY will accelerate
mainstream development of network applicationsenabling customers to deploy designs
for a broad range of high-speed connectivity applications. For designs that demand high
capacity and high speed for memory and storage, the TR10a-HL delivers with six
independent banks of QDRII+ SRAM, high-speed parallel flash memory. The feature-
set of the TR10a-HL fully supports all high-intensity applications such as low-latency
trading, cloud computing, high-performance computing, data acquisition, network
processing, and signal processing.
T

5
1.2 Key Features
The following hardware is implemented on the TR10a-HL board:
FPGA
Altera Arria® 10 GX FPGA (10AX115N2F45E1SG)
FPGA Configuration
On-Board USB Blaster II or JTAG header for FPGAprogramming
Fast passive parallel (FPPx32) configuration via MAX II CPLD and flash memory
General user input/output:
8 LEDs
4 push-buttons
2 dip switches
Clock System
50MHz Oscillator
Programmable clock generators Si5340A and Si5340B
Memory
QDRII+ SRAM
FLASH
Communication Ports
Four QSFP+ connectors
PCI Express (PCIe) x8 edge connector
One RS422 transceiver with RJ45 connector
System Monitor and Control
Temperature sensor
Fan control
Power monitor
Power
PCI Express 6-pin power connector, 12V DC Input
PCI Express edge connector power

6
Mechanical Specification
PCI Express full-height and 1/2-length
1.3 Block Diagram
Figure 1-1 shows the block diagram of the TR10a-HL board. To provide maximum
flexibility for the users, all key components are connected with the Arria 10 GX FPGA
device. Thus, users can configure the FPGA to implement any system design.
Figure 1-1 Block diagram of the TR10a-HLboard
Below is more detailed information regarding the blocks in Figure 1-1.
Arria 10 GX FPGA
10AX115N2F45E1SG

7
1,150K logic elements (LEs)
67-Mbits embedded memory
48 transceivers (12.5Gbps)
3,036 18-bit x 19-bit multipliers
1,518 Variable-precision DSP blocks
4 PCI Express hard IP blocks
768 user I/Os
384 LVDS channels
32 phase locked loops (PLLs)
FPGA Configuration
On-board USB Blaster II for use with the Quartus II Programmer
MAXII CPLD 5M2210 System Controller and Fast Passive Parallel (FPP x32)
configuration
Memory devices
48MB QDRII+ SRAM
256MB FLASH
General user I/O
8 user controllable LEDs
4 user push buttons
2 user dip switches
On-Board Clock
50MHz oscillator
Programming PLL providing clock for 40G QSFP+ transceiver
Programming PLL providing clock for PCIe transceiver
Programming PLL providing clocks for QDRII+ SRAM
Four QSFP+ ports

8
Four QSFP+ connector (40 Gbps+)
PCI Express x8 edge connector
Support for PCIe x8 Gen1/2/3
Edge connector for PC motherboard with x8 or x16 PCI Express slot
Power Source
PCI Express 6-pin DC 12V power
PCI Express edge connector power

9
Chapter 2
Board Components
his chapter introduces all the important components on the TR10a-HL.
2.1 Board Overview
Figure 2-1 is the top and bottom view of the TR10a-HL development board. It depicts
the layout ofthe board and indicates the location of the connectors and key components.
Users can refer to this figure for relative location of the connectors and key components.
Figure 2-1 FPGA Board (Top)
T

10
Figure 2-2 FPGABoard (Bottom)
2.2 Configuration, Status and Setup
Configure
The FPGA board supports two configuration methods for the Arria 10 FPGA:
Configure the FPGA using the on-board USB-Blaster II.
Flash memory configuration of the FPGA using stored images from the flash
memory on power-up.
For programming by on-board USB-Blaster II, the following procedures show how to
download a configuration bit stream into the Arria 10 GX FPGA:
Make sure that power is provided to the FPGA board
Connect your PC to the FPGAboard using a micro-USB cable and make sure
the USB-Blaster II driver is installed on PC.
Launch Quartus II programmer and make sure the USB-Blaster II is detected.
In Quartus II Programmer, add the configuration bit stream file (.sof), check
the associated “Program/Configure” item, and click “Start” to start FPGA
programming.

11
Status LED
The FPGA Board development board includes board-specific status LEDs to indicate
board status. Please refer to Table 2-1 for the description of the LED indicator.
Table 2-1 Status LED
Board
Reference
LED Name
Description
D2
12-V Power
Illuminates when 12-V power is active.
D3
3.3-V Power
Illuminates when 3.3-V power is active.
D7
CONF DONE
Illuminates when the FPGA is successfully
configured. Driven by the MAX II CPLD 5M2210
System Controller.
D10
Loading
Illuminates when the MAX II CPLD 5M2210 System
Controller is actively configuring the FPGA. Driven
by the MAX II CPLD 5M2210 System Controller
with the Embedded Blaster CPLD.
D8
Error
Illuminates when the MAX II CPLD EPM2210
System Controller fails to configure the FPGA.
Driven by the MAX II CPLD EPM2210 System
Controller.
D9
PAGE
Illuminates when FPGA is configured by the factory
configuration bit stream.
Setup PCI Express Control DIP switch
The PCI Express Control DIP switch (SW2) is provided to enable or disable different
configurations of the PCIe Connector. Table 2-2 lists the switch controls and description.
Table 2-2 SW2 PCIe Control DIP Switch
Board
Reference
Signal Name
Description
Default
SW2.1
PCIE_PRSNT2n_x1
On : Enable x1 presence detect
Off: Disable x1 presence detect
Off
SW2.2
PCIE_PRSNT2n_x4
On : Enable x4 presence detect
Off: Disable x4 presence detect
Off
SW2.3
PCIE_PRSNT2n_x8
On : Enable x8 presence detect
Off: Disable x8 presence detect
On

12
Setup Configure Mode
The position 1~3 of DIP switch SW1 are used to specify the configuration mode of the
FPGA. As currently only one mode is supported, please set all positions as shown in
Figure 2-3.
Figure 2-3 Position of DIP switch SW1 for Configure Mode
Select Flash Image for Configuration
The position 4 of DIP switch SW1 is used to specify the image for configuration of the
FPGA. Setting Position 4 of SW1 to “1” (down position) specifies the default factory
image to beloaded, asshown in Figure 2-4. Setting Position 4 of SW1 to “0” (upposition)
specifies the TR10a-HL to load a user-defined image, as shown in Figure 2-5.

13
Figure 2-4 Position of DIP switch SW1 for Image Select –Factory Image Load
Figure 2-5 Position of DIP switch SW1 for Image Select –User Image Load
2.3 General User Input/Output
This section describes the user I/O interface to the FPGA.
User Defined Push-buttons
The FPGA board includes four user defined push-buttons that allow users to interact
with the Arria 10 GX device. Each push-button provides a high logic level or a low logic
level when it is not pressed or pressed, respectively. Table 2-3 liststheboard references,
signal names and their corresponding Arria 10 GX device pin numbers.
Table 2-3 Push-button Pin Assignments, Schematic Signal Names, and
Functions
Board
Reference
Schematic
Signal
Name
Description
I/O
Standard
Arria 10 GX
Pin Number
PB0
BUTTON0
High Logic Level when the button
is not pressed
1.8-V
PIN_AC11
PB1
BUTTON1
1.8-V
PIN_AC12
PB2
BUTTON2
1.8-V
PIN_AC12
PB3
BUTTON3
1.8-V
PIN_AP8

14
User-Defined Dip Switch
There are two dip switches on the FPGAboard to provide additionalFPGAinput control.
When a dip switch is in the DOWN position or the UPPER position, it provides a high
logic level or a low logic level to the Arria 10 GX FPGA, respectively, as shown in Figure
2-6.
Figure 2-6 2 Dip switches
Table 2-4 lists the signal names and their corresponding Arria 10 GX device pin
numbers.
Table 2-4 Dip Switch Pin Assignments, Schematic Signal Names, and Functions
Board
Reference
Schematic
Signal Name
Description
I/O
Standard
Arria 10 GX
Pin Number
SW0
SW0
High logic level when SW in the
UPPER position.
1.8-V
PIN_ BD28
SW1
SW1
1.8-V
PIN_AM27
User-Defined LEDs
The FPGA board consists of 8 user-controllable LEDs to allow status and debugging
signals to be driven to the LEDs from the designs loaded into the Arria 10 GX device.
Each LED is driven directly by the Arria 10 GX FPGA. The LED is turned on or off when
the associated pins are driven to a low or high logic level, respectively. A list of the pin
names on the FPGA that are connected to the LEDs is given in Table 2-5.

15
Table 2-5 User LEDs Pin Assignments, Schematic Signal Names, and Functions
Board
Reference
Schematic
Signal Name
Description
I/O
Standard
Arria 10 GX
Pin Number
LED0
LED0
Driving a logic 0on the I/O port
turns the LED ON.
Driving a logic 1on the I/O port
turns the LED OFF.
1.8-V
PIN_T11
LED1
LED1
1.8-V
PIN_R11
LED2
LED2
1.8-V
PIN_N15
LED3
LED3
1.8-V
PIN_M15
D6-1
LED_BRACKET0
1.8-V
PIN_BB32
D6-3
LED_BRACKET1
1.8-V
PIN_AW30
D6-5
LED_BRACKET2
1.8-V
PIN_AV30
D6-7
LED_BRACKET3
1.8-V
PIN_AM30
2.4 Temperature Sensor and Fan Control
The FPGA board is equipped with a temperature sensor, TMP441, which provides
temperature sensing. These functions are accomplished by connecting the temperature
sensor to the internal temperature sensing diode of the Arria 10 GX device. The
temperature status and holding configuration information registers of the temperature
sensor can be programmed by a two-wire SMBus, which is connected to the Arria 10
GX FPGA. In addition, the 7-bit POR slave address for this sensor is set to
‘0011100b’.Figure 2-7 shows the connection between the temperature sensor and the
Arria 10 GX FPGA.
Figure 2-7 Connections between the temperature sensor and the Arria 10 GX

16
FPGA
An optional 3-pin +12V fan located on J15 of the FPGA board is intended to reduce the
temperature of the FPGA. The board is equipped with a Fan-Speed regulator and
monitor, MAX6650, through an I2C interface, Users regulate and monitor the speed of
fan depending on the measured system temperature. Figure 2-8 shows the connection
between the Fan-Speed Regulator and Monitor and the Arria 10 GX FPGA.
Figure 2-8 Connections between the Fan-Speed Regulator/ Monitor and the Arria
10 GX FPGA
The pin assignments for the associated interface are listed in109H109H Table 2-6.
Table 2-6 Temperature Sensor and Fan Speed Control Pin Assignments,
Schematic Signal Names, and Functions
Schematic
Signal Name
Description
I/O Standard
Arria 10 GX Pin
Number
TEMPDIODEp
Positive pin of temperature
diode in Arria 10
-
PIN_N21
TEMPDIODEn
Negative pin of temperature
diode in Arria 10
-
PIN_P21
TEMP_I2C_SCL
SMBus clock
1.8-V
PIN_AU12
TEMP_I2C_SDA
SMBus data
1.8-V
PIN_AV12
FAN_I2C_SCL
2-Wire Serial Clock
1.8-V
PIN_AJ33
FAN_I2C_SDA
2-Wire Serial-Data
1.8-V
PIN_AK33
FAN_ALERT_n
Active-low AL
ERT input
1.8-V
PIN_AL32

17
2.5 Power Monitor
The TR10a-HLhas implemented a power monitor chip to monitor the board input power
voltage and current. Figure 2-9 shows the connection between the power monitor chip
and the Arria 10 GX FPGA. The power monitor chip monitors both shunt voltage drops
and board input power voltage allows user tomonitor thetotal boardpower consumption.
Programmable calibration value, conversion times, and averaging, combined with an
internal multiplier, enable direct readouts of current in amperes and power in watts.
Table 2-7 shows the pin assignment of power monitor I2C bus.
Figure 2-9 Connections between the Power Monitor chip and the Arria 10 GX
FPGA
Table 2-7 Pin Assignment of Power Monitor I2C bus
Schematic
Signal Name
Description
I/O
Standard
Arria 10 GX
Pin Number
POWER_MONITOR_I2C_SCL
Power Monitor SCL
1.8V
PIN_AT26
POWER_MONITOR_I2C_SDA
Power Monitor SDA
1.8V
PIN_AP25
POWER_MONITOR_ALERT_N
Power Monitor ALERT
1.8V
PIN_BD23

18
2.6 Clock Circuit
The development board includes four 50 MHz oscillators and two programmable clock
generators. Figure 2-10 shows the default frequencies of on-board all external clocks
going to the Arria 10 GX FPGA.
Figure 2-10 Clock circuit of the FPGA Board
A clock buffer is used to duplicate the 50 MHz oscillator, so there are six 50MHz clocks
fed into different five FPGAbanks. The two programming clock generators are low-jitter
oscillators which are used to provide special and high quality clock signals for high-
speed transceivers and high bandwidth memory. Through I2C serial interface, the clock
generator controllers in the Arria 10 GX FPGAcan be used to program the Si5340Aand
Si5340B to generate 40G Ethernet QSFP+ and high bandwidth memory reference
clocks respectively.
Table 2-8 lists the clock source, signal names, default frequency and their
correspondingArria 10 GX device pin numbers.
Table 2-8 Clock Source, Signal Name, Default Frequency, Pin Assignments and

19
Functions
Source
Schematic
Signal Name
Default
Frequency
I/O
Standard
Arria 10 GX
Pin Number
Application
Y8
CLK_50_B2H
50.0 MHz
1.8V
PIN_AP34
Y9
CLK_50_B2G
1.8V
PIN_AW35
Y10
CLK_50_B2F
1.8V
PIN_AY31
Y1
CLK_50_B3D
1.8V
PIN_AN7
CLK_50_B3F
1.8V
PIN_G12
CLK_50_B3H
1.8V
PIN_D21
Y5
CLK_100_B3D
100.0MHz
1.8V
PIN_AJ11
Y7
OSC_100_CLKUSR
100.0MHz
1.8V
PIN_AV26
User-supplied
configuration
clock
U3
QSFPA_REFCLK_p
644.53125
MHz
LVDS
PIN_AH5
40G QSFP+ A
port
QSFPB_REFCLK_p
644.53125
MHz
LVDS
PIN_AD5
40G QSFP+ B
port
QSFPC_REFCLK_p
644.53125
MHz
LVDS
PIN_Y5
40G QSFP+ C
port
QSFPD_REFCLK_p
644.53125
MHz
LVDS
PIN_T5
40G QSFP+ D
port
U20
QDRIIA_REFCLK_p
275 MHz
LVDS
PIN_L9
QDRII+ reference
clock for A port
QDRIIB_REFCLK_p
275 MHz
LVDS
PIN_N18
QDRII+ reference
clock for B port
QDRIIC_REFCLK_p
275 MHz
LVDS
PIN_G24
QDRII+ reference
clock for C port
QDRIID_REFCLK_p
275 MHz
LVDS
PIN_M34
QDRII+ reference
clock for D port
QDRIIE_REFCLK_p
275 MHz
LVDS
PIN_AP14
QDRII+ reference
clock for E port
QDRIIF_REFCLK_p
275 MHz
LVDS
PIN_AT7
QDRII+ reference
clock for F port
Table 2-9 lists the programmable oscillator control pins, signal names, I/O standard and
their corresponding Arria 10 GX device pin numbers.

20
Table 2-9 Programmable oscillator control pin, Signal Name, I/O standard, Pin
Assignments and Descriptions
Programmable
Oscillator
Schematic
Signal Name
I/O
Standard
Arria 10 GX
Pin Number
Description
Si5340A
(U3)
Si5340A_I2C_SCL
1.8-V
PIN_AU27
I2C bus, connected
with Si5340A
Si5340A_I2C_SDA
1.8-V
PIN_AT27
Si5340A
(U3)
Si5340A_RST
1.8-V
PIN_AW28
Si5340A reset
signal
Si5340A_INTR
1.8-V
PIN_AW29
Si5340A interrupt
signal
Si5340A_OE_n
1.8-V
PIN_AV28
Si5340A output
enable signal
Si5340B
(U20)
Si5340B_I2C_SCL
1.8-V
PIN_G37
I2C bus, connected
with Si5340B
Si5340B_I2C_SDA
1.8-V
PIN_H31
Si5340B_RST
1.8-V
PIN_G38
Si5340B reset
signal
Si5340B_INTR
1.8-V
PIN_G32
Si5340B interrupt
signal
Si5340B_OE_n
1.8-V
PIN_AL31
Si5340B output
enable signal
2.7 FLASH Memory
The development board has two 1Gb CFI-compatible synchronous flash devices for
non-volatile storage of FPGA configuration data, user application data, and user code
space.
Each interface has a 16-bit data bus and the two devices combined allow for FPP x32
configuration.This device is part of the shared flash and MAX (FM) bus, which connects
to the flash memory and MAX V CPLD (5M2210) System Controller. Figure 2-11 shows
the connections between the Flash, MAX and Arria 10 GX FPGA.