Applistar DNPCIe 10G K7 LL QSFP User manual

DINI GROUP
LOGIC Emulation Source
UserManual
DNPCIe_10G_K7_LL(_QSFP)

L O G I C E M U L A T I O N S O UR CE
DNPCIe_10G_K7_LL (_QSFP) User Manual Version
1.9
DateofPrintDecember12,2017
Dini Group 2012-2017
7469 Draper Ave.
La Jolla, CA92037
Phone [phone] • Fax [phone]
[e-mail]
www.dinigroup.com

Copyright Notice and Proprietary Information
Copyright © 2012-2017 Dini Group. All rights reserved. No part of this copyrighted work may be
reproduced, modified or distributed in any form or by any means, without the prior written permission of
the Dini Group.
Right to Copy Documentation
Dini Group permits licensee to make copies of the documentation for its internal use only. Each copy shall
include all copyrights, trademarks, disclaimers and proprietary rights notices.
Disclaimer
Dini Group has made reasonable efforts to ensure that the information in this document is accurate and
complete. However, the Dini Group assumes no liability for errors, or for any incidental, consequential,
indirect, or special damages, including, without limitation, loss of use, loss or alteration of data, delays, or
lost profits or savings, arising from the use of this document or the product which it accompanies.

Table of Contents
INTRODUCTION ............................................................................................................................................................................................................................1
1DNPCIE_10G_K7_LL (_QSFP) ETHERNET PACKET ANALYSIS ENGINE .........................................................................................................1
1.1 Overview.........................................................................................................................................................................................................................1
1.2 FPGA –Xilinx, Kintex-7 ...............................................................................................................................................................................................1
1.3 Two Channels of 10 GbE or Four Channels of 10 GbE for the _QSFP version.......................................................................................................2
1.4 QDR II+ SSRAM - Memory with the Lowest Latency.................................................................................................................................................2
1.5 DDR3 DRAM - Bulk Memory .......................................................................................................................................................................................2
1.6 PCI Express –Customizable 4-lane, GEN2 PCI Express...........................................................................................................................................3
1.7 Time Synchronization ....................................................................................................................................................................................................3
1.8 How Everything Works … .............................................................................................................................................................................................3
2DNPCIE_10G_K7_LL (_QSFP) ETHERNET PACKET ANALYSIS ENGINE FEATURES.......................................................................................4
3PACKAGE CONTENTS:............................................................................................................................................................................................6
4INSPECT THE BOARD..............................................................................................................................................................................................6
5ADDITIONAL INFORMATION..................................................................................................................................................................................7
GETTING STARTED .....................................................................................................................................................................................................................8
1BEFORE YOU BEGIN ..............................................................................................................................................................................................8
1.1 Configuring the Programmable Components..............................................................................................................................................................8
1.2 Warnings ........................................................................................................................................................................................................................8
2INSTALLING THE SOFTWARE .................................................................................................................................................................................8
2.1 Exploring the Customer Support Package...................................................................................................................................................................9
3BOARD SETUP ........................................................................................................................................................................................................9
3.1 Before Powering Up the Board ....................................................................................................................................................................................9
3.2 Cooling Requirements –IMPORTANT!! ...................................................................................................................................................................10
3.3 Powering Up the Board...............................................................................................................................................................................................10
4USING THE REFERENCE DESIGN (MAIN) ............................................................................................................................................................11
PROGRAMMING/CONFIGURING THE HARDWARE......................................................................................................................................................14
1INTRODUCTION ....................................................................................................................................................................................................14
2CONFIGURING THE FPGA USING JTAG .............................................................................................................................................................15
2.1 Setup - Configuring the FPGA using JTAG ...............................................................................................................................................................15
2.2 Powering Up the Board...............................................................................................................................................................................................15
2.3 Installing Digilent cable driver...................................................................................................................................................................................15
2.3.1 Windows ..............................................................................................................................................................................................................15
2.3.2 Linux ....................................................................................................................................................................................................................16
2.4 Configuring the FPGA ................................................................................................................................................................................................17
3CONFIGURING THE FPGA USING MASTER BPI..................................................................................................................................................18
3.1 Setup - Configuring the FPGA using Master BPI .....................................................................................................................................................19
3.2 Powering Up the Board...............................................................................................................................................................................................19
3.3 Configuring the FPGA ................................................................................................................................................................................................19
3.4 Using multiple FPGA “boot” images for configuration fallback ............................................................................................................................21
4USING CHIPSCOPE PRO (VIA JTAG)...................................................................................................................................................................22
4.1 Setup –Using ChipScope Pro (via JTAG) .................................................................................................................................................................23
4.2 Powering Up the Board...............................................................................................................................................................................................23
4.3 Configuring the FPGA ................................................................................................................................................................................................23
HARDWARE DESCRIPTION ....................................................................................................................................................................................................25

1DESCRIPTION........................................................................................................................................................................................................25
1.1 Overview.......................................................................................................................................................................................................................25
1.2 FPGA –Xilinx, Kintex-7 .............................................................................................................................................................................................26
1.3 Two Channels of 10 GbE or Four Channels of 10 GbE for the _QSFP version.....................................................................................................26
1.4 QDR II+ SSRAM - Memory with the Lowest Latency...............................................................................................................................................27
1.5 DDR3 DRAM - Bulk Memory .....................................................................................................................................................................................27
1.6 PCI Express –Customizable 4-lane, GEN2 PCI Express.........................................................................................................................................27
1.7 Time Synchronization ..................................................................................................................................................................................................28
2FPGA (KINTEX-7) ...............................................................................................................................................................................................28
2.1 FPGA Configuration ...................................................................................................................................................................................................28
2.2 USB Port (RS232/JTAG) .............................................................................................................................................................................................28
2.2.1 RS232/JTAG Circuit Diagram ...........................................................................................................................................................................28
2.2.2 Connections between FPGA and the RS232 Port .............................................................................................................................................29
2.3 QDR II+ SRAM Memory.............................................................................................................................................................................................29
2.3.1 QDRII+ SRAM Memory Architecture ..............................................................................................................................................................30
2.3.2 Design Guidelines –QDR II+ SRAM IO Standards.........................................................................................................................................30
2.3.3 Connections between FPGA and QDR II+ SRAM Devices (4M x 18)...........................................................................................................31
2.4 DDR3 Memory (VLP MINIUDIMM) .........................................................................................................................................................................33
2.4.1 DDR3 SDRAM Memory Interface Solution .....................................................................................................................................................33
2.4.2 Design Guidelines - DDR3 Termination ...........................................................................................................................................................34
2.4.3 Design Guidelines –DDR3 IO Standards .........................................................................................................................................................35
2.4.4 Serial Presence-Detect EEPROM Operation.....................................................................................................................................................35
2.4.5 Clocking Connections between FPGA and MINIUDIMM ..............................................................................................................................36
2.4.6 Connections between FPGA and MINIUDIMM ..............................................................................................................................................36
2.5 EEPROM ......................................................................................................................................................................................................................41
2.5.1 EEPROM Circuit Diagram .................................................................................................................................................................................41
2.5.2 Connections between FPGA and the EEPROM................................................................................................................................................41
2.6 PCI Express Interface (x4)..........................................................................................................................................................................................41
2.6.1 System Requirements..........................................................................................................................................................................................42
2.6.2 Clocking - Jitter Attenuator ................................................................................................................................................................................42
2.6.3 PCI Express Circuit .............................................................................................................................................................................................42
2.6.4 Connections between FPGA and PCI Express Edge Connector ......................................................................................................................42
2.7 SFP+ Interface (only for DNPCIe_10G_K7_LL) .....................................................................................................................................................43
2.7.1 SFP+ Circuit Diagram.........................................................................................................................................................................................43
2.7.2 LED indicators.....................................................................................................................................................................................................44
2.7.3 SFP+ Pin Assignments........................................................................................................................................................................................45
2.7.4 Connections between FPGA and the SFP+ Connectors ...................................................................................................................................45
2.8 QSFP+ Interface (only for the DNPCIe_10G_K7_LL_QSFP) ................................................................................................................................47
2.8.1 QSFP+ Circuit Diagram......................................................................................................................................................................................47
2.8.2 LED indicators.....................................................................................................................................................................................................48
2.8.3 QSFP+ Pin Assignments.....................................................................................................................................................................................48
2.8.4 Connections between FPGA and the QSFP+ Connectors ................................................................................................................................49
2.9 Time Synchronization ..................................................................................................................................................................................................51
2.9.1 Time Synchronization Circuit Diagram .............................................................................................................................................................51
2.9.2 Connections between the FPGA and Time Synchronization Circuitry ...........................................................................................................51
3CLOCK GENERATION ...........................................................................................................................................................................................52
3.1 System Clock –IDELAYCTRL ....................................................................................................................................................................................52
3.1.1 Connection between FPGA and the System Clock Oscillator..........................................................................................................................52
3.2 High-Speed (GTX) Clocks...........................................................................................................................................................................................52
4LED INDICATORS ................................................................................................................................................................................................52
4.1 FPGA Status LEDs ......................................................................................................................................................................................................52
4.2 Configuration DONE LEDs ........................................................................................................................................................................................53
5POWER DISTRIBUTION.........................................................................................................................................................................................53
5.1 In-System Operation ....................................................................................................................................................................................................53
6MECHANICAL .......................................................................................................................................................................................................53
6.1 Board Dimensions .......................................................................................................................................................................................................53
APPENDIX 55
1APPENDIX A: UCF FILE......................................................................................................................................................................................55
2ORDERING INFORMATION ...................................................................................................................................................................................55

List of Figures
Figure 1 - DNPCIe_10G_K7_LL (_QSFP) Ethernet Packet Analysis Engine. (upper picture is the DNPCIe_10G_K7_LL and lower picture is the
DNPCIe_10G_K7_LL_QSFP) ..................................................................................................................................................................................................................4
Figure 2 - USB Flash Drive Directory Structure...........................................................................................................................................................................................................9
Figure 3 - DNPCIe_10G_K7_LL (_QSFP) Block Diagram –Note the two SFP+ modules are replaced with one QSFP+ module in the _QSFP version ....... 26
Figure 4 –FPGA Serial Port............................................................................................................................................................................................................................................ 29
Figure 5 - QDR II+ Memory Architecture................................................................................................................................................................................................................. 30
Figure 6 –FPGA Serial Port............................................................................................................................................................................................................................................ 41
Figure 7 - SFP+ Channel 0 Interface ........................................................................................................................................................................................................................... 44
Figure 8 –SFP+ GTX Oscillator.................................................................................................................................................................................................................................. 44
Figure 9 - QSFP+ Channel 0 Interface........................................................................................................................................................................................................................ 47
Figure 10 –QSFP+ GTX Oscillator............................................................................................................................................................................................................................ 48

List of Tables
Table 1 –USB Flash Drive Directory Contents ...........................................................................................................................................................................................................9
Table 2 –Kintex-7 Uncompressed Bitstream Length .............................................................................................................................................................................................. 18
Table 3 - Connections between RS232 Port and the FPGA................................................................................................................................................................................... 29
Table 4 –QDR II+ SRAM IO Standards................................................................................................................................................................................................................... 30
Table 5 - Connections between FPGA and the QDR II+ SRAM Devices ........................................................................................................................................................ 31
Table 6 - Serial Presence-Detect EEPROM Connections....................................................................................................................................................................................... 35
Table 7 –Clocking Connections between FPGA and the UDIMM Connector ................................................................................................................................................ 36
Table 8 - Connections between FPGA and the UDIMM Connector .................................................................................................................................................................. 36
Table 9 - Connections between FPGA and the EEPROM .................................................................................................................................................................................... 41
Table 10 - Connections between FPGA and the PCI Express Edge Connector............................................................................................................................................... 43
Table 11 –SFP+ Pin Assignments ............................................................................................................................................................................................................................... 45
Table 12 - Connections between FPGA and the SFP+ Connectors .................................................................................................................................................................... 46
Table 13 –QSFP+ Pin Assignments............................................................................................................................................................................................................................ 48
Table 14 - Connections between FPGA and the QSFP+ Connectors................................................................................................................................................................. 49
Table 15 - Connection between the FPGA and the System Clock Oscillator .................................................................................................................................................... 52
Table 16 –FPGA Status LEDs ..................................................................................................................................................................................................................................... 52
Table 17 –FPGA DONE LED.................................................................................................................................................................................................................................... 53

I N T R O D U C T I O N
DNPCIe_10G_K7_LL (_QSFP) User Manual www.dinigroup.com 1
Introduction
This User Manual accompanies the DNPCIe_10G_K7_LL (_QSFP)
Ethernet Packet Analysis Engine. For specific information regarding the Xilinx
Kintex-7 parts, please referencethedatasheet onthe Xilinxwebsite.
1DNPCIe_10G_K7_LL (_QSFP) Ethernet Packet
Analysis Engine
1.1 Overview
The DNPCIe_10G_K7_LL (_QSFP) is a PCI Express based FPGA board designed
to minimize input to output processing latency on 10Gb Ethernet packets. The primary
application is for ultra-low latency, high throughput trading without CPU intervention.
Every possible variable that affects input to output latency has been analyzed and
minimized. Raw 10 GbE packets can be analyzed and acted upon without interrupts or
an operating system adding delay to the process. This configurable hardware computing
platform has the ability to achieve the theoretical minimum Ethernet packet processing
latency. This board also has a time code input to allow for precise message time
stamping and tracking.
1.2 FPGA –Xilinx, Kintex-7
The Xilinx, Kintex-7, in the FFG676 package is utilized for this product. This package
supports 400 IOs with the majority utilized. Most are dedicated to a variety of off chip
memory peripherals including QDR II+ for low-latency, high speed look-up, and
DDR3 for performance oriented bulk storage. The Kintex-7 FPGAs contain high-speed
transceiver PHYs. The GTX transceivers are capable of handling data rates of 500 Mb/s
to 12.5 Gb/s, making these applicable to 10 Gigabit Ethernet (10 GbE) and
GEN1/GEN2 PCI Express applications. Four of the GTX transceivers are used for
GEN2-capable PCIe. For the DNPCIe_10G_K7_LL version, two of the GTX
transceivers are connected to 10 GbE SFP+ sockets. For the
DNPCIe_10G_K7_LL_QSFP version, four of the GTX transceivers are connected to
the 40 GbE QSFP+ socket.
Chapter
1

I N T R O D U C T I O N
DNPCIe_10G_K7_LL (_QSFP) User Manual www.dinigroup.com 2
Either the XC7K325T or the XC7K410T FPGAs can be populated. Both come in three
speeds grades, with -3 being the fastest.
1.3 Two Channels of 10 GbE or Four Channels of 10 GbE for
the _QSFP version
The Kintex-7 FPGAs have transceivers capable of 10 GbE. The physical interface is
handled using SFP+ modules or a single QSFP+ module for the _QSFP verison. This
allows you to bypass a MAC if necessary and process raw Ethernet packets. The
DNPCIe_10G_K7_LL has two 10 GbE channels and the
DNPCIe_10G_K7_LL_QSFP has four 10 GbE channel, and can support
10GBASET-ER, 10GBASET-SR, 10GBASET-KR.
1.4 QDR II+ SSRAM - Memory with the Lowest Latency
One, quad data rate, static RAMs (QDR II+ SSRAM) is used in the 4M x 18 size. This
style of memory has separate input and output data paths, enabling maximum
read/write data bandwidth with minimum latency. Using -3 speed grade FPGA, this
interface is capable of running at the maximum I/O frequency of 500MHz. To
minimize processing latency, we suspect it will be best to clock these QDRII+ SSRAMs
at 312.50 MHz, exactly twice the internal Ethernet controller frequency of 156.25MHz.
The Kintex-7 FPGAs are capable of generating internal 2x clocks that are phase
synchronous, eliminating the latencies associated with the tricky re-synchronization of
data moving between different clock frequencies. The internal controller can be
optimized in any way you choose. Dini Group provides several Verilog examples. All
functions of the QDR II+ SSRAM can be exploited, including concurrent read and
write operations and four-tick bursts. The only real limitation is the amount of time and
effort spent in customizing the individual memory controllers.
1.5 DDR3 DRAM - Bulk Memory
A single 244-pin PC3-10600 DDR3 VLP MINIUDIMM socket enables up to 4GB of
memory for bulk storage and lookup. Using a -2 or -3 speed grade FPGA, this interface
is tested at the maximum FPGA I/O frequency: 666.5Hz (1333Mb/s with DDR). The
user can use this memory as 64-bits with 8 bits of error correction (ECC), or as a 72-bit
byte-memory without correction.
To minimize data synchronization across clock boundaries, it probably makes sense to
clock the DDR3 interface at a 3x multiple of the base Ethernet frequency of 156.25
MHz, which is 468.75MHz. A 3x phase synchronous clock can be easily generated
internal to the FPGA, allowing zero latency synchronous data transfers between the
Ethernet packet receiving logic and the DDR3 memory controller. The DDR3
controller can be optimized in any way you choose. We, of course, provide several
Verilog examples. All functions of the DDR3 DRAM can be exploited and optimized.
Timing variables such as CAS latency and precharge can be tailored to the minimum
given your operating frequency and the timing specification of the exact DDR3 memory

I N T R O D U C T I O N
DNPCIe_10G_K7_LL (_QSFP) User Manual www.dinigroup.com 3
utilized. As with the QDRII+ SRAM, the only real limitation is the amount of time and
effort spent customizing the DDR3 memory controller to your needs.
1.6 PCI Express –Customizable 4-lane, GEN2 PCI Express
PCI Express is connected directly to the FPGA via 4-lanes of GTX transceivers. The
interfaces are GEN2 capable, and the board is shipped with PCIe IP that is a full
function, fixed, 4-lane master/target. To gain access to the PCIe interface, this IP must
be integrated with the user application. Dini Group provides support with the IP,
including BAR sizes. Additionally we can optionally add or subtract DMA engines,
scratchpad memories, interrupts, and other host-related functions to maximize the
performance, while utilizing the minimum FPGA resources. 'C' source for drivers for
several operating systems are included no charge. Partial reconfiguration of the FPGA is
supported via the PCIe interface.
1.7 Time Synchronization
The time code input allows for precise message time stamping and tracking. This input
can receiver PPS, or IRIG-B000 (RS232, RS485, RS422, TLL).
1.8 How Everything Works …
With direct data feeds such as NASDAQ (ITCH/OUCH) or Financial Information
Exchange (FIX), the DNPCIe_10G_K7_LL (_QSFP) contains all of the basic
functions required to minimize the amount of time it takes to receive Ethernet packets,
process them, and respond deterministically. The MAC, operating system et al, can be
bypassed. There are no interrupts. No operating system. Not a single clock cycle is
wasted here, enabling a near theoretical minimum in-to-out response time. For
algorithms requiring processing, FPGA resources can be hard coded to perform the
task. This includes real-time Monte Carlo analysis and floating point, all operating 1000's
of times faster than possible in a processor-based approach.