N.A.T. NAT-MCH User manual

NAT-MCH Hub Module PCIe – Technical Reference Manual
NAT-MCH
μTCA Telecom MCH Module
Technical Reference Manual V 1.0
Hub Module PCIe HW Revision 1.4

NAT-MCH Hub Module PCIe – Technical Reference Manual
The NAT-MCH Hub Module PCIe has been designed by:
N.A.T. GmbH
Kamillenweg 22
D-53757 Sankt Augustin
Phone: ++49/2241/3989-0
Fax: ++49/2241/3989-10
E-Mail: [email protected]
Internet: http://www.nateurope.com
Version 2.0 © N.A.T. GmbH 2

NAT-MCH Hub Module PCIe – Technical Reference Manual
Disclaimer
The following documentation, compiled by N.A.T. GmbH (henceforth called N.A.T.), repre-
sents the current status of the product’s development. The documentation is updated on a
regular basis. Any changes which might ensue, including those necessitated by updated speci-
fications, are considered in the latest version of this documentation. N.A.T. is under no obli-
gation to notify any person, organization, or institution of such changes or to make these
changes public in any other way.
We must caution you, that this publication could include technical inaccuracies or typographi-
cal errors.
N.A.T. offers no warranty, either expressed or implied, for the contents of this documentation
or for the product described therein, including but not limited to the warranties of merchant-
ability or the fitness of the product for any specific purpose.
In no event will N.A.T. be liable for any loss of data or for errors in data utilization or
processing resulting from the use of this product or the documentation. In particular, N.A.T.
will not be responsible for any direct or indirect damages (including lost profits, lost savings,
delays or interruptions in the flow of business activities, including but not limited to, special,
incidental, consequential, or other similar damages) arising out of the use of or inability to use
this product or the associated documentation, even if N.A.T. or any authorized N.A.T.
representative has been advised of the possibility of such damages.
The use of registered names, trademarks, etc. in this publication does not imply, even in the
absence of a specific statement, that such names are exempt from the relevant protective laws
and regulations (patent laws, trade mark laws, etc.) and therefore free for general use. In no
case does N.A.T. guarantee that the information given in this documentation is free of such
third-party rights.
Neither this documentation nor any part thereof may be copied, translated, or reduced to any
electronic medium or machine form without the prior written consent from N.A.T. GmbH.
This product (and the associated documentation) is governed by the N.A.T. General
Conditions and Terms of Delivery and Payment.
Note:
The release of the Hardware Manual is related
to a certain HW board revision given in the
document title. For HW revisions earlier than
the one given in the document title please
contact N.A.T. for the corresponding older
Hardware Manual release.
Version 2.0 © N.A.T. GmbH 3

NAT-MCH Hub Module PCIe – Technical Reference Manual
Table of Contents
CONVENTIONS................................................................................................................................................... 7
1BOARD SPECIFICATION........................................................................................................................ 8
2STATEMENT ON ENVIRONMENTAL PROTECTION.................................................................... 10
2.1 COMPLIANCE TO ROHS DIRECTIVE .................................................................................................... 10
2.2 COMPLIANCE TO WEEE DIRECTIVE ................................................................................................... 10
2.3 COMPLIANCE TO CE DIRECTIVE ......................................................................................................... 11
3INSTALLATION ...................................................................................................................................... 12
3.1 SAFETY NOTE..................................................................................................................................... 12
3.2 INSTALLATION PREREQUISITES AND REQUIREMENTS ......................................................................... 12
3.2.1 Requirements................................................................................................................................. 12
3.2.2 Power supply................................................................................................................................. 12
3.2.3 Automatic Power Up..................................................................................................................... 12
4INTRODUCTION..................................................................................................................................... 13
5HUB MODULE PCIE BASICS................................................................................................................ 14
6HUB MODULE PCIE BLOCK DIAGRAM........................................................................................... 15
7BOARD FEATURES ................................................................................................................................ 16
8LOCATION OVERVIEW........................................................................................................................ 17
9FUNCTIONAL BLOCKS......................................................................................................................... 18
9.1 PCI EXPRESS SWITCHES ..................................................................................................................... 18
9.2 SPREAD SPECTRUM CLOCK................................................................................................................. 19
9.3 MICROCONTROLLER ........................................................................................................................... 19
10 NAT-MCH HUB MODULE PCIE PROGRAMMING NOTES........................................................... 20
10.1 SPI INTERFACE................................................................................................................................... 20
10.2 I²C INTERFACE.................................................................................................................................... 20
10.3 REGISTER............................................................................................................................................ 20
10.3.1 Board Identifier Register.......................................................................................................... 20
10.3.2 PCB Revision Register ............................................................................................................. 21
10.3.3 Atmel Version........................................................................................................................... 21
10.3.4 Hub Module PCIe Typ.............................................................................................................. 21
10.3.5 Hot Plug Present 1 Register.....................................................................................................22
10.3.6 Hot Plug Present 2 Register.....................................................................................................22
10.3.7 Hot Plug Button 1 Register....................................................................................................... 23
10.3.8 Hot Plug Button 2 Register....................................................................................................... 24
10.3.9 Hot Plug Power Enable 1 Register........................................................................................... 24
10.3.10 Hot Plug Power Enable 2 Register........................................................................................... 25
10.3.11 Miscellaneous Control Register ............................................................................................... 26
11 CONNECTORS......................................................................................................................................... 27
11.1 CONNECTOR OVERVIEW ..................................................................................................................... 27
11.2 NAT-MCH HUB MODULE PCIE CONNECTOR CON1......................................................................... 28
Version 2.0 © N.A.T. GmbH 4

NAT-MCH Hub Module PCIe – Technical Reference Manual
11.3 NAT-MCH HUB MODULE X48 EXTENDER CONNECTOR CON2......................................................... 31
11.4 CONNECTOR CON4: INTERFACE TO CLK-MODULE ........................................................................... 33
12 KNOWN BUGS / RESTRICTIONS........................................................................................................ 34
APPENDIX A: REFERENCE DOCUMENTATION...................................................................................... 35
APPENDIX B: DOCUMENT’S HISTORY...................................................................................................... 36
List of Figures
Figure 1: Block Diagram of the NAT-MCH Hub Module PCIe.........................................15
Figure 2: Location diagram of the NAT-MCH Hub Module PCIe (top-view) ...................17
Figure 3: Location diagram of the NAT-MCH Hub Module PCIe (bottom-view).............17
Figure 4: Connectors of the NAT-MCH Hub Module PCIe (top view)..............................27
Figure 5: Connectors of the NAT-MCH Hub Module PCIe (bottom view)........................27
List of Tables
Table 1: List of used Abbreviations .....................................................................................7
Table 2: NAT-MCH Hub Module PCIe Features ................................................................8
Table 3: NAT-MCH Hub Module PCIe Technical Key Features........................................ 9
Table 4: 1st Switch to Fabric Port Mapping........................................................................18
Table 5: 2nd Switch to Fabric Port Mapping.......................................................................19
Table 6: Board Identifier Register......................................................................................20
Table 7: PCB Revision Register.........................................................................................21
Table 8: PCB Revision Register.........................................................................................21
Table 9: PCB Revision Register.........................................................................................21
Table 10: HP_PRSNT1 Register..........................................................................................22
Table 11: HP_PRSNT1 - Register Bits................................................................................22
Table 12: HP_PRSNT2 Register..........................................................................................22
Table 13: HP_PRSNT2 - Register Bits................................................................................23
Table 14: HP_BUTTON1 Register......................................................................................23
Table 15: HP_BUTTON1 - Register Bits ............................................................................23
Table 16: HP_BUTTON2 Register......................................................................................24
Table 17: HP_BUTTON2 - Register Bits ............................................................................24
Table 18: HP_PWREN1 Register ........................................................................................24
Table 19: HP_PWREN1 - Register Bits...............................................................................25
Table 20: HP_PWREN2 Register ........................................................................................25
Version 2.0 © N.A.T. GmbH 5

NAT-MCH Hub Module PCIe – Technical Reference Manual
Table 21: HP_PWREN2 - Register Bits...............................................................................25
Table 22: Misc_CTRL Register ...........................................................................................26
Table 23: Misc_CTRL - Register Bits..................................................................................26
Table 24: Hub Module PCIe backplane connector CON1...................................................28
Table 25: Hub Module x48 Extender backplane connector CON2...................................... 31
Table 26: Connector to CLK-Module CON4.......................................................................33
Version 2.0 © N.A.T. GmbH 6

NAT-MCH Hub Module PCIe – Technical Reference Manual
Conventions
If not otherwise specified, addresses and memory maps are written in hexadecimal notation,
identified by 0x.
Table 1 gives a list of the abbreviations used in this document:
Table 1: List of used Abbreviations
Abbreviation
Description
AMC Advanced Mezzanine Card
b bit, binary
B byte
ColdFire MCF5470
CPU Central Processing Unit
CU Cooling Unit
DMA Direct Memory Access
E1 2.048 Mbit G.703 Interface
FLASH Programmable ROM
FRU Field Replaceable Unit
J1 1,544 Mbit G.703 Interface (Japan)
K kilo (factor 400 in hex, factor 1024 in decimal)
LIU Line Interface Unit
M mega (factor 10,0000 in hex, factor 1,048,576 in
decimal)
MCH µTCA Carrier Hub
MHz 1,000,000 Herz
µTCA Micro Telecommunications Computing Architecture
PCIe PCI Express
PCI Peripheral Component Interconnect
PM Power Manager
RAM Random Access Memory
ROM Read Only Memory
SDRAM Synchronous Dynamic RAM
SSC Spread Spectrum Clock
T1 1,544 Mbit G.703 Interface (USA)
Version 2.0 © N.A.T. GmbH 7

NAT-MCH Hub Module PCIe – Technical Reference Manual
1 Board Specification
Table 2: NAT-MCH Hub Module PCIe Features
Power Consumption 12V / 1.6A max. (only Hub Module PCIe x48)
Environmental
Conditions
Temperature (operating):
Temperature (storage):
Humidity:
0°C to +50°C with forced cooling
-40°C to +85°C
10 % to 90 % rh noncondensing
Standards Compliance PCI Express Base Specification Rev. 1.1
PICMG µTCA.0 Rev. 1.0
PICMG AMC.0 Rev. 2.0
PICMG AMC.1 Rev. 1.0
PICMG SFP.0 Rev. 1.0 (System Fabric Plane Format)
IPMI Specification v2.0 Rev. 1.0
Product Safety The board complies with EN60950 and UL1950
Version 2.0 © N.A.T. GmbH 8

NAT-MCH Hub Module PCIe – Technical Reference Manual
Table 3: NAT-MCH Hub Module PCIe Technical Key Features
•support for 6 (option ”x24”) or 12 (option “x48”)
•non-blocking switch fabric
•built of two PLX5832 PCIe switches
•high performance x8 interconnect between chips to
prevent performance bottleneck
•upstream port (and NT-upstream port) configurable to
any of the 12 AMC slots
•PCIe hot plug support for each AMC slot
•clustering support, two clusters of 6 AMC modules each
can be operated individually each having its on root
complex
•support x1 and x4 width ports to any AMC
•PCIe compliant spread spectrum clocking
Version 2.0 © N.A.T. GmbH 9

NAT-MCH Hub Module PCIe – Technical Reference Manual
2 Statement on Environmental Protection
2.1 Compliance to RoHS Directive
Directive 2002/95/EC of the European Commission on the "Restriction of the use of
certain Hazardous Substances in Electrical and Electronic Equipment" (RoHS)
predicts that all electrical and electronic equipment being put on the European market
after June 30th, 2006 must contain lead, mercury, hexavalent chromium,
polybrominated biphenyls (PBB) and polybrominated diphenyl ethers (PBDE) and
cadmium in maximum concentration values of 0.1% respective 0.01% by weight in
homogenous materials only.
As these hazardous substances are currently used with semiconductors, plastics (i.e.
semiconductor packages, connectors) and soldering tin any hardware product is
affected by the RoHS directive if it does not belong to one of the groups of products
exempted from the RoHS directive.
Although many of hardware products of N.A.T. are exempted from the RoHS
directive it is a declared policy of N.A.T. to provide all products fully compliant to the
RoHS directive as soon as possible. For this purpose since January 31st, 2005 N.A.T.
is requesting RoHS compliant deliveries from its suppliers. Special attention and care
has been paid to the production cycle, so that wherever and whenever possible RoHS
components are used with N.A.T. hardware products already.
2.2 Compliance to WEEE Directive
Directive 2002/95/EC of the European Commission on "Waste Electrical and
Electronic Equipment" (WEEE) predicts that every manufacturer of electrical and
electronical equipment which is put on the European market has to contribute to the
reuse, recycling and other forms of recovery of such waste so as to reduce disposal.
Moreover this directive refers to the Directive 2002/95/EC of the European
Commission on the "Restriction of the use of certain Hazardous Substances in
Electrical and Electronic Equipment" (RoHS).
Having its main focus on private persons and households using such electrical and
electronic equipment the directive also affects business-to-business relationships. The
directive is quite restrictive on how such waste of private persons and households has
to be handled by the supplier/manufacturer, however, it allows a greater flexibility in
business-to-business relationships. This pays tribute to the fact with industrial use
electrical and electronical products are commonly integrated into larger and more
complex environments or systems that cannot easily be split up again when it comes to
their disposal at the end of their life cycles.
Version 2.0 © N.A.T. GmbH 10

NAT-MCH Hub Module PCIe – Technical Reference Manual
As N.A.T. products are solely sold to industrial customers, by special arrangement at
time of purchase the customer agreed to take the responsibility for a WEEE compliant
disposal of the used N.A.T. product. Moreover, all N.A.T. products are marked
according to the directive with a crossed out bin to indicate that these products within
the European Community must not be disposed with regular waste.
If you have any questions on the policy of N.A.T. regarding the Directive 2002/95/EC
of the European Commission on the "Restriction of the use of certain Hazardous
Substances in Electrical and Electronic Equipment" (RoHS) or the Directive
2002/95/EC of the European Commission on "Waste Electrical and Electronic
Equipment" (WEEE) please contact N.A.T. by phone or e-mail.
2.3 Compliance to CE Directive
Compliance to the CE directive is declared. A ‘CE’ sign can be found on the PCB.
Version 2.0 © N.A.T. GmbH 11

NAT-MCH Hub Module PCIe – Technical Reference Manual
3 Installation
3.1Safety Note
To ensure proper functioning of the NAT-MCH Hub Module PCIe during its usual
lifetime take refer to the safety note section of the NAT-MCH BASIC-Module
Technical Reference Manual before handling the board.
3.2 Installation Prerequisites and Requirements
IMPORTANT
Before powering up check this section for installation prerequisites and requirements
3.2.1 Requirements
The installation requires a NAT-MCH Basic-Module and a CLK Module where the
Hub Module PCIe can be mechanically fixed on to. The Hub Module PCIe must be
completely connected and joint to the complete PCB stack (Basic-Module and CLK
Module), before the NAT-MCH can be stacked into a MicroTCA backplane (as one
device). For further requirements refer to the requirements section of the NAT-MCH
BASIC-Module Technical Reference Manual.
3.2.2 Power supply
The power supply for the NAT-MCH Hub Module PCIe must meet the following
specifications:
+12 V / 1.6 A max. (only Hub Module PCIe x48, in addition to other PCBs of the
NAT-MCH).
3.2.3 Automatic Power Up
Power ramping/monitoring and power up reset generation is done by the NAT-MCH
Basic-Module
In the following situations the NAT-MCH Basic-Module will automatically be reset
and proceed with a normal power up.
•The voltage sensor generates a reset, when +12 V voltage level drops below 8V.
Version 2.0 © N.A.T. GmbH 12

NAT-MCH Hub Module PCIe – Technical Reference Manual
4 Introduction
The NAT-MCH consists of a Basic-Module, which can be expanded with additional
PCBs. The Basic-Module satisfies the basic requirements of the MicroTCA Specification
for a MicroTCA Carrier Hub. The main capabilities of the Basic-Module are:
•management of up to 12 AMCs, two cooling units (CUs) and one or more power
modules (PMs)
•Gigabit Ethernet Hub Function for Fabric A ( up to 12 AMCs) and for the Update
Fabric A to a second (redundant) NAT-MCH
To meet also the optional requirements of the MicroTCA specification, a CLK-Module
and different HUB Modules are available. With the Clock-Module the following
functions can be enabled:
•generation and distribution of synchronized clock signals for up to 12 AMCs
Through the extension of the NAT-MCH with a HUB Module, hub functions for fabric
D to G can be enabled. With the different versions the customers have the opportunity to
choose a HUB Module that fits best to their applications. The versions differ in:
•max. number of supported AMCs ( up to 6 / up to 12)
•supported protocols:
oPCI Express
oSerial Rapid IO
o10Gigabit Ethernet
The features of the individual modules are described in more detail in the corresponding
Technical Reference Manuals.
A general arrangement of the different modules of a NAT-MCH is shown in Figure 1.
Figure 1: Arrangement of different NAT-MCH Modules
LED Module
Basic Module CLK Module
Hub Module
This Technical Reference Manual describes the Hub Module PCIe.
Version 2.0 © N.A.T. GmbH 13

NAT-MCH Hub Module PCIe – Technical Reference Manual
5 Hub Module PCIe Basics
The Hub Module PCIe is an expansion Module of the NAT-MCH. In addition to the CLK
Module it can be mounted on the NAT-MCH Basic-Module. The Hub Module PCIe is in a
6 Slot (x24) and in a 12 slot (x48) option available. With the Hub Module PCIe the 3rd
tongue of the NAT-MCH connector to the MicroTCA backplane is always installed. With the
x48 option, additional the 4th tongue is installed. The NAT-MCH Hub Module PCIe
implements the following major features:
•support of PCI Express x4 switching function for fabrics D to G of up to 6 AMCs
(Hub Module PCIe x24)
•support of PCI Express x4 switching function for fabrics D to G of up to 12 AMCs
(Hub Module PCIe x48)
•support of spread spectrum clocking (distribution of the SSC signal is done by the
CLK Module)
The Hub Module PCIe contains out of two PLX 8532 PCIe switches. The first PCIe switch
connects to AMC 1-6. This Switch is subsequent referred as 1st Switch or 1st PEX8532. The
second PCIe switch connects to AMC 7-12. This switch is subsequent referred as 2nd switch
or 2nd PEX8532.
Version 2.0 © N.A.T. GmbH 14

NAT-MCH Hub Module PCIe – Technical Reference Manual
6 Hub Module PCIe Block Diagram
Figure 1: Block Diagram of the NAT-MCH Hub Module PCIe
* The HUB-PCB x48_ext and the 2nd PCIe switch are only assembled in the x48 version.
Version 2.0 © N.A.T. GmbH 15

NAT-MCH Hub Module PCIe – Technical Reference Manual
7 BOARD FEATURES
•PCI Express Switch
The board is equipped with PLX PEX8532 PCI Express switches, which provide non-
blocking switching at full line rate. Quality of Service (QoS) is provided by the
PEX8532, supporting 2 virtual cannels and 8 traffic classes per port. One of all ports
can be configured as transparent upstream port, and one of all ports can be configured as
non-transparent upstream port.
•Spread Spectrum Clock
The board is equipped with an ICS9FG104, which generates reference clock for the two
switches.
•Microprocessor
To configure the two switches and the SSC the board is equipped with an Atmel 8-bit
microcontroller.
•Interfaces
The NAT-MCH Hub Module PCIe implements interfaces to connect fabrics D to G of
up to 12 AMCs.
•Interface to other NAT-MCH PCBs
Basic-Module: - The Microcontroller on the Hub Module PCIe can be updated
by the CPU on the Basic-Module via a SPI interface. Normal
communication between the Microprocessor and the CPU is
done by IPMI messages via the I²C interface.
- For each of the PCIe switches a configuration EEPROM resides
on the Hub Module PCIe. These two EEPROMs can be
programmed / updated by the CPU on the Basic-Module via the
SPI interface.
CLK-Module: - The Hub Module PCIe can provide the 100 MHz PCI Express
compliant clock signal to the CLK-Module.
Version 2.0 © N.A.T. GmbH 16

NAT-MCH Hub Module PCIe – Technical Reference Manual
8 Location Overview
Figure 2: "Location diagram of the NAT-MCH Hub Module PCIe”shows the position
of important components.
Figure 2: Location diagram of the NAT-MCH Hub Module PCIe (top-view)
Figure 3: Location diagram of the NAT-MCH Hub Module PCIe (bottom-view)
Version 2.0 © N.A.T. GmbH 17

NAT-MCH Hub Module PCIe – Technical Reference Manual
9 Functional Blocks
The NAT-MCH Hub Module PCIe is divided into a number of functional blocks,
which are described in the following paragraphs.
9.1 PCI Express Switches
Each of the two PEX8532 PCI Express Switches supports 6 ports, each with 4 lanes
(PCIe x4), in order to connect 6 AMCs. One port with 8 lanes is used to connect the two
Switches.
Both PEX8532 can be configured by strapping pins, by loading an EEPROM, or by PCI
Express messages from a host. A standard configuration is done by the microprocessor
and resistors by setting the strapping pins. The values of the strapping signals that are
connected to the microcontroller can be controlled by programming a register in the
microcontroller.
These standard settings can be changed by reading the EEPROM after a reset, or by
receiving PCI Express messages from a host.
The EEPROM contains basic configuration information for the PCIe switch as well as
user settings, e.g. upstream port settings. The user settings can be changed by the CPU
on the Basic-Module via the SPI interface.
The /PERST pin is also connected to the microcontroller. The value of this pin can also
be controlled by programming a register in the microcontroller.
The first port of the switch is not connected to the first port of fabric D-G, and so on. To
ease routing of the differential fabrics between the switches and the backplane
connectors the following allocation has been selected: Connection to 2nd PEX8532
Table 4: 1st Switch to Fabric Port Mapping
# AMC Slot
Fabric D-G
#Port
1st PEX8532
AMC1 2
AMC2 1
AMC3 8
AMC4 9
AMC5 10
AMC6 11
connection to
2nd PEX8532 0
Not used 3
Version 2.0 © N.A.T. GmbH 18

NAT-MCH Hub Module PCIe – Technical Reference Manual
Table 5: 2nd Switch to Fabric Port Mapping
# AMC Slot
Fabric D-G
#Port
2nd PEX8532
AMC7 2
AMC8 1
AMC9 8
AMC10 9
AMC11 10
AMC12 11
Connection to
1st PEX8532 0
Not used 3
9.2 Spread Spectrum Clock
An ICS9FG104 clocking device generates a PCI Express compliant clock signal with a
frequency of 100 MHz fixed or spread spectrum (0.5% down spread). The ICS9FG104
feeds the PEX8532 with this clock signal. Additional the CLK Module can be sourced
by the ICS9FG104 over CON4.
9.3 Microcontroller
An 8-bit Atmel microcontroller resides on the Hub Module PCIe. The microcontroller
can be updated by the CPU on the Base-Module via the SPI interface. Normal
communication between the CPU and the microcontroller is done by IPMI messages
over the I²C interface.
The strapping options and the reset signal of the switches can be controlled through
programming registers in the microcontroller. Also the spread spectrum function of the
ICF9FG104 can be controlled over these register.
Furthermore, three temperature sensors are connected to a second I²C bus of the
Microcontroller. The microcontroller makes these sensors accessible to the CPU on the
Base-Module via IPMI.
Version 2.0 © N.A.T. GmbH 19

NAT-MCH Hub Module PCIe – Technical Reference Manual
10 NAT-MCH Hub Module PCIe Programming Notes
10.1 SPI Interface
The SPI interface can be multiplexed between three devices on the Hub Module PCIe.
Default the SPI interface is connected to the Microcontroller. In this it is only used for
maintenance purposes, e.g. updating the microcontroller firmware.
Or the SPI Bus can be connected to one of the two configuration EEPROMs that reside
on the Hub Module PCIe. In this case the SPI interface is used to read or write data on
the EEPROM.
10.2 I²C Interface
The I²C interface is the main communication interface between the microcontroller and
the CPU of the Basic-Module. All communication is based on IPMI Messages.
10.3 Register
A register interface is implemented in the Atmel microcontroller. With the help of this
interface different functions can be controlled and various identification values can be
red.
10.3.1 Board Identifier Register
The Board Identifier Register contains the Board ID that identifies the board as NAT-
MCH Hub Module PCIe.
Table 6: Board Identifier Register
Board Identifier - Address 0x00
Default value 0xb7
Bit 7 6 5 4 3 2 1 0
Access R R R R R R R R
Func BOARD_ID
Version 2.0 © N.A.T. GmbH 20