IDT PCI Express 89HPES32NT24xG2 User manual

®
6024 Silver Creek Valley Road, San Jose,California 95138
Telephone: [phone] • [phone] • FAX: [phone]
Printed in U.S.A.
©2013 Integrated Device Technology,Inc.
IDT®89HPES32NT24xG2
PCI Express®Switch
User Manual
January 2013

GENERAL DISCLAIMER
Integrated Device Technology, Inc. reserves the right to make changes to its products or specifications at any time, without notice, in order to improve design or performance
and to supply the best possible product. IDT does not assume any responsibility for use of any circuitry described other than the circuitry embodied in an IDT product. The
Company makes no representations that circuitry described herein is free from patent infringement or other rights of third parties which may result from its use. No license is
granted by implication or otherwise under any patent, patentrights or other rights, of Integrated Device Technology, Inc.
CODE DISCLAIMER
Code examples provided by IDT are for illustrative purposes only and should not be relied upon for developing applications. Any use of the code examples below is completely
at yourown risk. IDT MAKESNO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE NONINFRINGEMENT, QUALITY,SAFETY OR SUITABILITY
OF THE CODE, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICU-
LAR PURPOSE, ORNON-INFRINGEMENT. FURTHER, IDTMAKES NO REPRESENTATIONS OR WARRANTIES AS TO THE TRUTH,ACCURACY ORCOMPLETENESS
OF ANY STATEMENTS, INFORMATION OR MATERIALS CONCERNING CODE EXAMPLES CONTAINED IN ANY IDT PUBLICATION OR PUBLIC DISCLOSURE OR
THAT IS CONTAINED ON ANY IDT INTERNET SITE. IN NO EVENT WILL IDT BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, INDIRECT, PUNITIVE OR
SPECIAL DAMAGES, HOWEVER THEY MAY ARISE, AND EVEN IF IDT HAS BEEN PREVIOUSLY ADVISED ABOUT THE POSSIBILITY OF SUCH DAMAGES. The code
examplesalsomaybe subjecttoUnitedStatesexportcontrollawsandmaybesubjecttotheexportorimportlawsof othercountriesandit isyourresponsibilitytocomplywith
any applicable laws or regulations.
LIFE SUPPORT POLICY
Integrated Device Technology's products are not authorized for use as critical components in life support devices or systems unless a specific writtenagreement pertaining to
such intended use is executed between the manufacturer and an officer of IDT.
1. Life support devices or systems are devices or systems which (a) are intended for surgical implant into the body or (b) support or sustain life and whose failure to perform,
when properly used in accordance with instructions for use providedin the labeling, can be reasonably expected toresult in a significant injury to the user.
2.A criticalcomponentisanycomponentsofalifesupportdeviceorsystemwhosefailuretoperformcanbereasonablyexpected to cause the failure of the life support device
or system, or to affect its safety or effectiveness.
IDT, theIDT logo, and Integrated Device Technology are trademarks or registered trademarks of Integrated Device Technology, Inc.

Notes
PES32NT24xG2 User Manual 1 January 30, 2013
®
About This Manual
Overview
This user manual includes hardware andsoftware information on the 89HPES32NT24xG2, a member of
IDT’s PRECISE™ family of PCI Express® switching solutions offering the next-generation I/O interconnect
standard. The part number PES32NT24xG2 which is used extensively throughout this manual in fact
covers two distinct switch devices: the PES32NT24AG2 and the PES32NT24BG2. The information in this
manual applies equally to both devices except where noted in occasional notes and footnotes in various
chapters.
Finding Additional Information
Information not included inthis manual such as mechanicals, packagepin-outs, and electrical character-
istics can be found in thedata sheet for this device, which is available from the IDT website (www.idt.com)
as well as through your local IDT sales representative.
Content Summary
Chapter 1, “PES32NT24xG2 Device Overview,” provides an introduction to the performance capabili-
ties of the 89HPES32NT24xG2 and a high level architectural overview of the device.
Chapter 2, “Clocking,” provides a description of the PES32NT24xG2 clocking architecture.
Chapter 3, “Reset and Initialization,” describes thePES32NT24xG2 reset operations and initialization
procedure.
Chapter 4, “Switch Core,” provides a description of the PES32NT24xG2 switch core.
Chapter 5, “Switch Partitions,” describes how the PES32NT24xG2 supports up to 16 active switch
partitions.
Chapter 6, “Failover,” provides a description of the flexible failover mechanism that allows the
construction of highly-available systems.
Chapter 7, “Link Operation,” describes the operation of the link feature including polarity inversion,
link width negotiation, and lane reversal.
Chapter 8, “SerDes,” describes basic functionality and controllability associated with the Serialiazer-
Deserializer (SerDes) block in PES32NT24xG2 ports.
Chapter 9, “Power Management,” describes the power management capability structure locatedin the
configuration space of each PCI-to-PCI bridge in the PES32NT24xG2.
Chapter 10, “Transparent Operation,” describes the device-specific architectural features for the
transparent switch associated with each PES32NT24xG2partition (i.e., the PCI-to-PCI bridge functions and
their interaction in the switch).
Chapter 11, “Hot-Plug and Hot-Swap,” describes the behavior of the hot-plug and hot-swap features
in the PES32NT24xG2.
Chapter 12, “SMBus Interfaces,” describes the operation of the 2 SMBus interfaces on the
PES32NT24xG2.
Chapter 13, “General Purpose I/O,” describes how the 9 General Purpose I/O (GPIO) pins may be
individually configured as general purpose inputs, general purpose outputs, or alternate functions.
Chapter 14, “Non-Transparent Operation,” describes how a non-transparent bridge in the
PES32NT24xG2 allows two roots or PCI Express trees (i.e., hierarchies) to be interconnected with one or
more shared address windows between them.

IDT
PES32NT24xG2 User Manual 2 January 30, 2013
Notes
Chapter 15, “DMA Controller,” describes how the PES32NT24xG2 supports two direct memory
access controller (DMA) functions.
Chapter 16, “Switch Events,” describes mechanisms provided by the PES32NT24xG2 to facilitate
communication between roots associated with different partitions as well as for communication between
these roots and a management agent.
Chapter 17, “Multicast,” describes how the multicast capability enables a single TLP to be forwarded
to multiple destinations.
Chapter 18, “Temperature Sensor,” provides a description of the on-chip temperature sensor with
three programmable temperature thresholds and a temperaturehistory capability.
Chapter 19, “Register Organization,” describes the organization of all the software visible registers in
the PES32NT24xG2 and provides the address space for those registers.
Chapter 20, “PCI to PCI Bridge and Proprietary Port Specific Registers,” lists the Type 1 configura-
tion header registers in the PES32NT24xG2 and provides a description of each bit in those registers.
Chapter 21, “Proprietary Registers,” lists the proprietary registers in the PES32NT24xG2 and
provides a description of each bit in those registers.
Chapter 22, “NT Endpoint Registers,” lists the NT Endpoint registers in the PES32NT24xG2 and
provides a description of each bit in those registers.
Chapter 23, “DMA Registers,” lists the DMA registers in the PES32NT24xG2 and provides a descrip-
tion of each bit in those registers.
Chapter 24, “Switch Control Registers,” lists the switch control and status registers in the
PES32NT24xG2 and provides a description of each bit in those registers.
Chapter 25, “JTAG Boundary Scan,” discusses an enhanced JTAG interface, including a system logic
TAP controller, signal definitions, a test data register, an instruction register, and usage considerations.
Chapter 26, “Usage Models,” describes possible configurations of the PES32NT24xG2 switch and
presents some important system usage models.
Signal Nomenclature
To avoid confusion when dealing with a mixture of “active-low” and “active-high” signals, the terms
assertion and negation are used. The term assert or assertion is used to indicate that a signal is active or
true, independent ofwhether that level is represented by ahigh or low voltage. The termnegate or negation
is used to indicate that a signal is inactive or false.
To define the active polarity of a signal, a suffix will be used. Signals ending with an ‘N’ should be inter-
preted as being active, or asserted, when at a logic zero (low) level. All other signals (including clocks,
buses and select lines) will be interpreted as being active, or asserted when at a logic one (high) level.
To define buses, the most significant bit (MSB) will be on the left and least significant bit (LSB) will be on
the right. No leading zeros will be included.
Throughout this manual, when describing signal transitions, the following terminology is used. Rising
edge indicates a low-to-high (0 to1) transition. Falling edge indicates ahigh-to-low (1 to 0) transition. These
terms are illustrated in Figure 1.

IDT
PES32NT24xG2 User Manual 3 January 30, 2013
Notes
Figure 1 Signal Transitions
Numeric Representations
To represent numerical values, either decimal, binary, or hexadecimal formats will be used. The binary
format is as follows: 0bDDD, where “D” represents either 0 or 1; the hexadecimal format is as follows:
0xDD, where “D” represents the hexadecimal digit(s); otherwise, it is decimal.
The compressed notation ABC[x|y|z]D refers to ABCxD, ABCyD, and ABCzD.
The compressed notation ABC[y:x]D refers to ABCxD, ABC(x+1)D, ABC(x+2)D,... ABCyD.
Data Units
The following data unit terminology is used in this document.
In quadwords, bit 63 is always the most significant bit and bit 0 is the least significant bit. In double-
words, bit 31 is always the most significant bit and bit 0 is the least significant bit. In words, bit 15 is always
the most significant bit and bit 0 is the least significant bit. In bytes, bit 7 is always the most significant bit
and bit 0 is the least significant bit.
The ordering of bytes within words is referred to as either “big endian” or “little endian.” Big endian
systems label byte zero as the most significant (leftmost) byte of a word. Little endian systems label byte
zero as the least significant (rightmost) byte of a word. See Figure 2.
Term Words Bytes Bits
Byte 1/2 1 8
Word 1 2 16
Doubleword (DWord) 2 4 32
Quadword (QWord) 4 8 64
Table 1 Data Unit Terminology
1 2 3 4
high-to-low
transition low-to-high
transition
single clock cycle

IDT
PES32NT24xG2 User Manual 4 January 30, 2013
Notes
Figure 2 Example of Byte Ordering for “Big Endian” or “Little Endian” System Definition
Register Terminology
Software in the context of this register terminology refers to modifications made by PCI Express root
configuration writes, writes to registers made through the slave SMBus interface, or serial EEPROM
register initialization. See Table 2.
Type Abbreviation Description
Hardware Initialized HWINIT Register bits are initialized by firmware or hardware mechanisms
such as pin strapping or serial EEPROM. (System firmware hard-
ware initialization is only allowed for system integrated devices.)
Bits are read-only after initialization and can only be reset (for
write-once by firmware) with reset.
Read Only and Clear RC Software can read the register/bits with this attribute. Reading the
value will automatically cause the register/bit to be reset to zero.
Writing to a RC location has no effect.
Read Clear and Write RCW Software can read the register/bits with this attribute. Reading the
value will automatically cause the register/bits to be reset to zero.
Writes cause the register/bits to be modified.
Reserved Reserved The value read from a reserved register/bit is undefined. Thus,
software must deal correctly with fields that are reserved. On
reads, softwaremustuseappropriatemaskstoextract the defined
bits and not rely on reserved bits being any particular value. On
writes, software must ensure that the values of reserved bit posi-
tions are preserved. That is, the values of reserved bit positions
must first be read, merged with the new values for other bit posi-
tions and then written back.
In addition to reserved registers, some valid register fields have
encodings markedas reserved.Suchregister fields must never be
written with a value corresponding to an encoding marked as
reserved. Violating this rule produces undefined operation in the
device.
Read Only RO Software can only read registers/bits with this attribute. Contents
are hardwired to a constant value or are status bits that may be
set and cleared by hardware. Writing to a RO location has no
effect.
Table 2 Register Terminology (Part 1 of 2)
0123
bit 0bit 31
Address of Bytes within Words: Big Endian
3210
bit 0bit 31
Address of Bytes within Words: Little Endian

IDT
PES32NT24xG2 User Manual 5 January 30, 2013
Notes
Use of Hypertext
In Chapter 19, Tables 19.2, 19.5, 19.6, 19.10, and 19.11 contain register names and page numbers
highlighted in blue under the Register Definition column. In pdf files, users can jump from this source table
directly to the registers by clicking on the register name in the source table. Each register name in the table
is linked directly to the appropriate register in Chapters 20 through 24. To return to the source table after
having jumped to the register section, click on the same register name (in blue) in the register section.
Reference Documents
PCI Express Base Specification Revision 2.1., March 4, 2009, PCI-SIG.
PCI Local Bus Specification Revision 3.0., February 3, 2004, PCI-SIG.
PCI-to-PCI Bridge Architecture Specification Revision 1.2., June 9, 2003, PCI-SIG.
Address Translation Services Specification, March 8, 2007, PCI-SIG.
PCI Bus Power Management Interface Specification, Revision 1.2., March 3, 2004, PCI-SIG
SMBus Specification, Version 2.0, August 3, 2000, SBS Implementers Forum.
Revision History
July 8, 2009: Initial publication of preliminary user manual.
Read and Write RW Software can both read and write bits with this attribute.
Read and Write Clear RW1C Software can read and write to registers/bits with this attribute.
However, writing a value of zero to a bit with this attribute has no
effect. A RW1C bit can only be set to a value of 1 by a hardware
event. To clear a RW1C bit (i.e., change its value to zero) a value
of one must be written to the location. An RW1C bit is never
cleared by hardware.
Read and Write when
Unlocked Software can read the register/bits with this attribute. Writing to register/bits with this attribute will only cause the value to be modi-
fied if the REGUNLOCK bit in the SWCTL register is set. When
the REGUNLOCK bit is cleared, writes are ignored and the regis-
ter/bits are effectively read-only.
Sticky Sticky Register/bits with this designation take on their initial value as a
resultofaswitch fundamental reset or partition fundamentalreset.
Other resets have no effect.
Switch Sticky SWSticky Register/bits with this designation take on their default value as a
result of a switch fundamental reset. Other resets have no effect.
Modified Switch Sticky MSWSticky A MSWSticky register is a Switch Sticky register that in addition to
taking on its default value as a result of a switch fundamental
reset, it takes on its default value when the event(s) defined in the
register description occur, unless the register has been written-to
by software/firmware before the occurrence of the event.
If the value of an MSWSticky register has been written by soft-
ware/firmware,itpreservesthevalueacrossalleventsuntilwritten
again or until a switch fundamental reset is applied to the device.
After a switch fundamental reset, the MSWSticky register will
return to taking on the value as defined in the register description.
Type Abbreviation Description
Table 2 Register Terminology (Part 2 of 2)

IDT
PES32NT24xG2 User Manual 6 January 30, 2013
Notes
July 14, 2009: Includes changes in several chapters based on recent updates in the functional specifi-
cation.
July 30, 2009: Includes changes in several chapters based on recent updates in the functional specifi-
cation.
August 28, 2009: Added Chapter 27, Usage Models.
September 18, 2009: In Chapter 2, added Table 2.5. In Chapter 4, added new sections Packet Routing
Classes and Proprietary Weighted Round Robin (WRR) Arbitration, and revised Figure 4.8. Made
numerous revisions in Chapter 8. In Chapter 10, made changes to the Action Taken column in Table 10.14.
In Chapter 12, updatedthe I/O Expandertables. In Chapter15, made changes to Table 15.7and added text
to DMA Multicast section. In Chapter 25, made numerous changes in SerDes x Transmitter Lane Control 0
and 1 registers
October 6, 2009: In Chapter 3, added text to section Switch Fundamental Rest and moved this section
in front of Boot Configuration Vector section, and added text to Switch Modes section. In Chapter 5, added
text to sections Switch Partitioning and Non-Transparent Operations. In Chapter 15, modified sections Data
Transfer and Addressing, Source Address Expired Error, and Destination Address Expired Error. In Chapter
16, added text to section Switch Signals. In Chapter 21, modified description for bit EIS in the PCIESSTS
register. In Chapter 24, modified description of RUN bit in the DMAC[1:0]CTL register.
October 14, 2009: In Chapter 22, added WRR Port Arbitration Counts registers, and in Chapter 20,
updated Figure 20.2 and Table 20.5 to show new registers. Corrected Table 26.2, Boundary Scan Chain.
November 4, 2009: In Chapter 2, added new section Support for Spread Spectrum Clocking (SCC) with
updated tables and modified Limitations column in Table 2.6, Clock Frequency Limitations. In Chapter 5,
added three new sections: Partition State Change Latency, Port Operating Mode Change Latency, and
Partition Reconfiguration Latency. In Chapter 8, deleted all references to Slew Rate. In Chapter 10, title for
Table 10.11 was changed to Unexpected Completions instead of Unsupported Requests, and a new bullet
was added at the topof section Address Routed TLPs.In Chapter 14,modified text inOverviewsectionand
in section Unsupported Request (UR) Error. In Chapter 15, modified text in section Reception of a Request
TLP That is Unsupported. In Chapter 19, added reference to junction temperature in the Overview section.
In Chapter 20, added new section Configuration Register Side-Effects under Overview. In Chapter 24, DMA
Channel Error Mask register, de-featured bits 2 and 17. In Chapter 25, SMBus Control register, de-featured
bit 16 (MSMBIOM).
November 23, 2009: In Chapter 4, corrected port numbers for Stack 1 in Figure 4.1.
December 4, 2009: In Chapter 10, modified text in section Error Emulation Control in the PCI-to-PCI
Bridge Function and added new section Error Emulation Usage and Limitations. In Chapter 14, modified
text in section Error Emulation Control in the NT Function and added new section Error Emulation Usage
and Limitations.
January 6, 2010: In Chapter 17, corrected references to NT Multicast Control register and NT Multicast
Transmit Enable bit in the following sections: NT Multicast TLP Routing, and Usage Restrictions. In Chapter
20, corrected Figure 20.5. In Chapter 23, NTIERRORMSK0 register, changed bits 29 and 30 to reserved. In
Chapter 24, DMAIERRORMSK0 register, changed bits 29 and 30 to reserved. In Chapter 25, changed
default values for several bits in the TMPADJA and TSSLOPE registers.
March 4, 2010: Revised manual to include references to the PES32NT24BG2 device in Chapters 1, 2,
3, 12, and 26 as appropriate and changed manual name to PES32NT24xG2.
March 8, 2010: Removed references to OUTDBELLCLR and OUTSBELLDBELLCLR registers in
Chapter 14, Non-Transparent Switch Operation.
March 17, 2010: In Chapter 8, updated Tables 8.6, 8.7, 8.8, 8.11, and 8.12. In Chapter 13, deleted refer-
ence to multiple GPIOAFSEL registers;there is only one register. In Chapter24, deleted “other”from ECRC
Error name for bit 31 in the DMAC[1:0]ERRSTS register.
May 10, 2010: In Chapter 21, PCI Bridge Registers, the ACSCAP register offset address was corrected
to 0x324. In Chapter 23, NT Endpoint Register, revised the Description for MODE field in BARSETUP0
register and LADDR field in BARLIMIT0 register.

IDT
PES32NT24xG2 User Manual 7 January 30, 2013
Notes
May 21, 2010: In Chapter 23, NT Endpoint Registers, revised Description for INDEX field in
LUTOFFSET register to read that if BAR4 is selected, the INDEX field must only be set to values 0 to 11
(instead of 12 to 23).
June 21, 2010: In Chapter 23, NT Endpoint Registers, revised Bit Field column in NTMTBLDATA
register.
August 27, 2010: In Chapter 4, revised textin sections Internal Errors and Reporting of Port AER Errors
as Internal Errors and updated Figures 4.7 and 4.8. In Chapter 5, revised text in Reset Mode Change
Behavior. In Chapter 7, revised text in Link Width Negotiation in the Presence of Bad Lanes section and
Crosslink section. In Chapter 11, corrected reference to DLLLASC in Hot Plug Events section. In Chapter
12, revised description for BYTECNT in Tables 12.19 and 12.21. In Chapter 14, added Note at end of
section NT Mapping Table. In Chapter 15, deleted section DMA Channel Errors and revised text in
Descriptor Prefetching, ECRC Errors, and Completion Timeout sections. In Chapter 16, revised text in
section Port AER Errors. In Chapter 17, changed reference from NTMTC to NTMCC in NT Multicast TLP
Routing section.
In Chapter 21, madethe following changes:revised description for MAXLNKSPD bitin PCI Express Link
Capabilities register (also applies to same bit in same register in Chapters 23 and 24), revised description
for bits in PCI Express Slot Control register.
In Chapter 22, made the following changes: added text under section Physical Layer Control and Status
Registers, revised description for bits in PCI Express Slot Control Initial Value register, deleted Port AER
Status register, revised Port AER Mask register, added bit 10 to bit 9 as Reserved and revised description
for ILSCC bit in Phy Link Configuration 0 register.
In Chapter 23, made the following changes: revised PCI Express Device Capabilities 2 and PCI Express
Device Control 2 registers, revised Description for REG and EREG bits in ECFGADDR register, added bits
31:16 row in AER Correctable Error Status register, added text in Description of NXTPTR in SNUMCAP
register, added text in Description of NXTPTR in PCIEVCECAP register, revised information for fields
PARBC and PATBLOFF in VCR0CAP register, revised information for fields LPAT and PARBSEL in
VCR0CTL register, revised Description for PATS in VCR0STS register, added text in Description of
NXTPTR in ACSECAPH register, added text in Description of NXTPTR in MCCAPH register, changed
default value for bits 30:29 from 0x1 to 0x3 in NTIERRORMSK0 register, changed bit 6 in the NTINTMSK
register to Reserved, revised description for bits SIZE and MODE in Bar 0 Setup register, revised informa-
tion in LADDR field in BARLIMIT0 register, revised text under register title for BAR 1 Limit Address and
changed Default Value for Reserved and LADDR and Description for LADDR, revised text under register
title for BARLIMIT3 and changed Default Value for Reserved and LADDR and Description for LADDR,
revised Default Value and Description for LADDR field in BARLIMIT4 register, revised text under register
title for BARLIMIT5 and changed Default Value and Description for LADDR, revised description for INDEX
in LUTOFFSET register.
In Chapter 24, made the following changes: revised bits 4 to 21 in PCIEDCAP2 register, revised bits 4 to
15 in PCIEDCTL2 register, revised bits 21 and 22 and added bits 24 and 25 in AERUES register, revised bit
22 and added bits 24 and 25 in AERUEM register, revised bit 22 and added bits 24 and 25 in AERUESV
register, changed Default Value for CIE bit in AERCES register, changed Type and Default Value for
ECRCGC and ECRCCC bits in AERCTL register, changed bit 24 (HEC) to reserved in DMAIERRORMSK1
register, de-featured bits 0 and 6 through 9 in the DMAC[1:0]ERRSTS register, de-featured bits 0 and 6
through 7 and changed name of bit 31 to ECRCE in DMAC[1:0]ERRMSK register.
In Chapter 25, made the following changes: revised SESTS register, revised description for COUNT
field in FCAP[3:0]TIMER register, added bits 20 and 21 and revised default value and/or description for bits
22 to 25 and changed name/value/description of bit 29 in SMBUSSTS register, removed default value for
TEMP field in TMPSTS register.
September 27, 2010: In Chapter 22, changed bit 16 in the IERRORSTS0 register from ULD to
Reserved.
October 22, 2010: In Chapter 15, added footnote to Table 15.7. In Chapter 25, re-arranged bits 24:28 in
TMPCTL register.

IDT
PES32NT24xG2 User Manual 8 January 30, 2013
Notes
December 21, 2010: In Chapter 2, revised header in Table 2.6 to read “Initial Port Clock Mode.” In
Chapter 5, added new footnote #1 in section Port Operating Mode Change. In Chapter 15, deleted refer-
ence to DATCT bit in Completion Timeout section. In Chapter 23, added text to SUBVID and SUBID regis-
ters. In Chapter 24, changed bit 20 (DATCT bit) in the DMAC[1:0]ERRSTS and DMAC[1:0]ERRMSK
registers to Reserved and added text to SUBVID and SUBID registers. In Chapter 26, deleted PERSTN,
GLK1, and SMODE from Table 26.1.
March 11, 2011: In Chapter 26, revised Usage Considerations section to remove reference to
JTAG_TCK being driven to a known value.
March 25, 2011: In Chapter 22, added PHYLSTATE0 register with FLRET bit description.
May 20, 2011: In Chapter 1, added ZC silicon to Table 1.2.
June 21, 2011: In Chapter 5, section Reset Mode Change Behavior, changed fourth bullet to read “The
port remains in a Reset state for at least 250 µs.”
June 24, 2011: In Chapter 25, added bit BDISCARD to the Switch Control register.
July 15, 2011: In Chapter 1, revised section Switch Events and removed “and Signals” from the section
title. In Chapter 5, revised the following sections: Downstream Switch Port, Port Operating Mode Change
Latency, and System Notification of Partition Reconfiguration. In Chapter 8, revised section Programmable
De-emphasis Adjustment. In Chapter 16, removed “and Signals” from title and revised section Global
Signals and deleted Signals section. In Chapter 21, MCBLKALLH register, changed lower 32 to upper 32 in
description of MCBLKALL bit. In Chapters 22 and 23, deleted references to SSIGNAL field. In Chapter 25,
added section Internal Switch Timers with 4 new registers and deleted SSIGNAL register. Updated Figure
20.5 and Table 20.11, Switch Configuration and Status, in Chapter 20 to account for new registers.
July 27, 2011: In Chapter 22, added bits 7:0 (RCVD_OVRD) in SERDESCFG register.
August 23, 2011: In Chapter 24, DMACxCFG register, changed 0x2 in DPREFETCH field to Reserved.
September 12, 2011: In Chapter 8, added additional reference in last paragraph of section Driver
Voltage Level and Amplitude Boost.
October 24, 2011: In Chapter 22, added Port Control Register. In Chapter 20, added reference to Port
Control register in Table 20.5.
November 7, 2011: In Chapter 2, section Local Port Clocked Mode, added recommendation to tie
unused port clock pins to ground.
December 4, 2011: In Chapter 25, revised Description for AFSEL0 field in the GPIOAFSEL register.
January 11, 2012: Removed Hardware Error Containment chapter. Deleted references to SWFRST bit.
February 8, 2012: In Chapter 12, added footnote for RERR and WERR bits in Table 12.20.
February 23, 2012: Added paragraph after Table 12.19 to explain use of DWord addresses.
March 14, 2012: In the Overview section of Chapter 2, changed “single” to ”two” differential global refer-
ence clock pairs.
May 1, 2012: In Chapter 2, Clocking, made text changes to state that unused port clock pins should be
connected to Vss on the board. In Chapter 12, SMBus Interfaces, added new section Setting Up I2C
Commands for Block Transactions.
June 27, 2012: In Chapter 12, changed BYTCNT=7 to BYTCNT=4 in Figure 12.14. In Chapter 24,
changed type and default values for bits 16 and 20 in Switch Control register.
January 30, 2013: In Figure 12.12, changed No-ack to Ack between DATALM and DATAUM.

Notes
PES32NT24xG2 User Manual i January 30, 2013
Table of Contents
®
About This Manual
Overview ........................................................................................................................................1
Content Summary ..........................................................................................................................1
Signal Nomenclature .....................................................................................................................2
Numeric Representations ..............................................................................................................3
Data Units ......................................................................................................................................3
Register Terminology .....................................................................................................................4
Use of Hypertext ............................................................................................................................5
Reference Documents ...................................................................................................................5
Revision History .............................................................................................................................5
PES32NT24xG2 Device Overview
Overview.........................................................................................................................................1-1
System Identification.......................................................................................................................1-1
Vendor ID................................................................................................................................1-1
Device ID................................................................................................................................1-1
Revision ID.............................................................................................................................1-1
JTAG ID..................................................................................................................................1-2
SSID/SSVID............................................................................................................................1-2
Device Serial Number Enhanced Capability...........................................................................1-2
Architectural Overview....................................................................................................................1-2
Port Operating Modes.....................................................................................................................1-3
Switch Partitioning..........................................................................................................................1-6
Non-Transparent Operation............................................................................................................1-8
DMA Operation.............................................................................................................................1-12
Dynamic Reconfiguration and Failover.........................................................................................1-15
Switch Events...............................................................................................................................1-16
Multicasting and Non-Transparent Multicasting............................................................................1-17
Clocking
Overview.........................................................................................................................................2-1
Port Clocking Modes.......................................................................................................................2-2
Global Clocked Mode.............................................................................................................2-3
Local Port Clocked Mode........................................................................................................2-4
Support for Spread Spectrum Clocking (SSC).......................................................................2-5
Port Clocking Mode Selection.................................................................................................2-6
System Clocking Configurations.....................................................................................................2-8
Reset and Initialization
Overview.........................................................................................................................................3-1
Switch Fundamental Reset.............................................................................................................3-2
Boot Configuration Vector...............................................................................................................3-4
Stack Configuration.........................................................................................................................3-5
Static Configuration of a Stack...............................................................................................3-9
Dynamic Reconfiguration of a Stack via EEPROM / SMBus..................................................3-9
Switch Modes.......................................................................................................................3-10
Partition Resets.............................................................................................................................3-11

IDT Table of Contents
PES32NT24xG2 User Manual ii January 30, 2013
Notes
Partition Fundamental Reset................................................................................................3-12
Partition Hot Reset ...............................................................................................................3-12
Partition Upstream Secondary Bus Reset............................................................................3-13
Partition Downstream Secondary Bus Reset .......................................................................3-14
Port Mode Change Reset.............................................................................................................3-14
Switch Core
Overview.........................................................................................................................................4-1
Switch Core Architecture................................................................................................................4-1
Ingress Buffer.........................................................................................................................4-2
Egress Buffer..........................................................................................................................4-3
Crossbar Interconnect............................................................................................................4-4
Virtual Channel Support..................................................................................................................4-5
Packet Routing Classes..................................................................................................................4-5
Packet Ordering..............................................................................................................................4-6
Arbitration .......................................................................................................................................4-6
Port Arbitration........................................................................................................................4-6
Cut-Through Routing......................................................................................................................4-9
Request Metering .........................................................................................................................4-11
Operation..............................................................................................................................4-13
Completion Size Estimation..................................................................................................4-14
Internal Errors...............................................................................................................................4-16
Switch Core Time-Outs ........................................................................................................4-17
Memory SECDED ECC Protection.......................................................................................4-18
End-to-End Data Path Parity Protection...............................................................................4-18
Reporting of Port AER Errors as Internal Errors...................................................................4-19
Switch Partition and Port Configuration
Overview.........................................................................................................................................5-1
Switch Partitions.............................................................................................................................5-1
Partition Configuration............................................................................................................5-2
Partition State.........................................................................................................................5-3
Partition State Change ...........................................................................................................5-4
Switch Ports....................................................................................................................................5-5
Switch Port Mode ...................................................................................................................5-5
Port Operating Mode Change.......................................................................................................5-13
Common Operating Mode Change Behavior .......................................................................5-15
No Action Mode Change Behavior.......................................................................................5-21
Reset Mode Change Behavior .............................................................................................5-21
Partition Reconfiguration and Failover..........................................................................................5-21
Partition Reconfiguration Latency.........................................................................................5-23
System Notification of Partition Reconfiguration ..................................................................5-23
Failover
Overview.........................................................................................................................................6-1
Failover Initiation.............................................................................................................................6-1
Software Initiated Failover......................................................................................................6-2
Signal Initiated Failover..........................................................................................................6-2
Watchdog Timer Initiated Failover..........................................................................................6-2

IDT Table of Contents
PES32NT24xG2 User Manual iii January 30, 2013
Notes
Link Operation
Overview.........................................................................................................................................7-1
Port Merging...................................................................................................................................7-1
Port Maximum Link Width...............................................................................................................7-1
Polarity Inversion............................................................................................................................7-2
Lane Reversal.................................................................................................................................7-2
Link Width Negotiation....................................................................................................................7-4
Link Width Negotiation in the Presence of Bad Lanes ...........................................................7-5
Dynamic Link Width Reconfiguration..............................................................................................7-5
Dynamic Link Width Reconfiguration in the PES32NT24xG2................................................7-6
Link Speed Negotiation...................................................................................................................7-6
Link Speed Negotiation in the PES32NT24xG2.....................................................................7-7
Software Management of Link Speed ....................................................................................7-8
Link Retraining................................................................................................................................7-9
Link States......................................................................................................................................7-9
Link Down Handling......................................................................................................................7-10
Slot Power Limit Support..............................................................................................................7-11
Upstream Port ......................................................................................................................7-11
Downstream Switch Port......................................................................................................7-12
Link Active State Power Management (ASPM)............................................................................7-12
L0s ASPM.............................................................................................................................7-12
L1 ASPM ..............................................................................................................................7-13
Link Status....................................................................................................................................7-16
De-emphasis Negotiation .............................................................................................................7-16
Crosslink.......................................................................................................................................7-17
Hot Reset Operation on a Crosslink.....................................................................................7-17
Link Disable Operation on a Crosslink .................................................................................7-18
Gen 1 Compatibility Mode ............................................................................................................7-18
SerDes
Overview.........................................................................................................................................8-1
SerDes Numbering and Port Association.......................................................................................8-1
SerDes Transmitter Controls..........................................................................................................8-3
Driver Voltage Level and Amplitude Boost.............................................................................8-3
De-emphasis ..........................................................................................................................8-4
PCI Express Low-Swing Mode...............................................................................................8-4
Receiver Equalization.....................................................................................................................8-4
Programming of SerDes Controls...................................................................................................8-4
Programmable Voltage Margining and De-Emphasis ............................................................8-5
SerDes Transmitter Control Registers....................................................................................8-6
Transmit Margining Using the PCI Express Link Control 2 Register....................................8-12
Low-Swing Transmitter Voltage Mode..........................................................................................8-12
Receiver Equalization Controls.....................................................................................................8-14
SerDes Power Management.........................................................................................................8-14
Power Management
Overview.........................................................................................................................................9-1
Power Management Event (PME) Messages.................................................................................9-4
PCI Express Power Management Fence Protocol .........................................................................9-4
Upstream Switch Port or Downstream Switch Port Mode......................................................9-4
NT Function Mode or NT with DMA Function Mode...............................................................9-5
Upstream Switch Port with NT and/or DMA Function Mode...................................................9-5

IDT Table of Contents
PES32NT24xG2 User Manual iv January 30, 2013
Notes
Transparent Switch Operation
Overview.......................................................................................................................................10-1
Transaction Routing......................................................................................................................10-1
Virtual Channel Support................................................................................................................10-2
Maximum Payload Size................................................................................................................10-2
Upstream Port Device Number.....................................................................................................10-2
Bus Locking..................................................................................................................................10-2
Interrupts.......................................................................................................................................10-4
Downstream Port Interrupts..................................................................................................10-4
Upstream Port Interrupts......................................................................................................10-4
Legacy Interrupt Aggregation...............................................................................................10-5
Access Control Services...............................................................................................................10-6
ECRC Support............................................................................................................................10-10
Error Detection and Handling by the PCI-to-PCI Bridge Function..............................................10-11
Physical Layer Errors .........................................................................................................10-11
Data Link Layer Errors........................................................................................................10-12
Transaction Layer Errors....................................................................................................10-13
Routing Errors ....................................................................................................................10-23
Error Emulation Control in the PCI-to-PCI Bridge Function................................................10-24
Hot-Plug and Hot-Swap
Overview.......................................................................................................................................11-1
Hot-Plug Signals...........................................................................................................................11-3
Port Reset Outputs.......................................................................................................................11-5
Power Enable Controlled Reset Output................................................................................11-5
Power Good Controlled Reset Output..................................................................................11-6
Hot-Plug Events............................................................................................................................11-7
Legacy System Hot-Plug Support.................................................................................................11-7
Hot-Swap......................................................................................................................................11-8
SMBus Interfaces
Overview.......................................................................................................................................12-1
Master SMBus Interface...............................................................................................................12-1
Initialization and I2C Reset...................................................................................................12-1
Serial EEPROM...................................................................................................................12-2
Initialization from Serial EEPROM........................................................................................12-3
Programming the Serial EEPROM.....................................................................................12-10
I/O Expanders.....................................................................................................................12-11
Slave SMBus Interface...............................................................................................................12-22
Initialization.........................................................................................................................12-22
SMBus Transactions ..........................................................................................................12-23
Setting Up I2C Commands for Block Transactions.....................................................................12-29
CSR Register Read or Write Operation..............................................................................12-29
SMBus Transactions ..........................................................................................................12-30
Examples of Setting Up the I2C CSR Byte Sequence for a CSR Register Read...............12-32
Examples of Setting Up the I2C CSR Byte Sequence for a CSR Register Write...............12-35
General Purpose I/O
Overview.......................................................................................................................................13-1
GPIO Configuration ......................................................................................................................13-1
Input......................................................................................................................................13-1
Output...................................................................................................................................13-1
Alternate Function ................................................................................................................13-1

IDT Table of Contents
PES32NT24xG2 User Manual v January 30, 2013
Notes
Non-Transparent Switch Operation
Overview.......................................................................................................................................14-1
Base Address Registers (BARs)...................................................................................................14-1
BAR Limit..............................................................................................................................14-2
Mapping NT Configuration Space to BAR 0.........................................................................14-4
TLP Translation ............................................................................................................................14-4
Direct Address Translation...................................................................................................14-4
Lookup Table Address Translation.......................................................................................14-5
ID Translation ...............................................................................................................................14-9
NT Mapping Table................................................................................................................14-9
Request ID Translation.......................................................................................................14-11
Completion ID Translation..................................................................................................14-13
Requester ID Capture Register..........................................................................................14-14
TLP Attribute Processing............................................................................................................14-14
No Snoop Processing.........................................................................................................14-14
Address Type Processing...................................................................................................14-15
NT Multicast................................................................................................................................14-15
Inter-Domain Communications...................................................................................................14-15
Doorbell Registers..............................................................................................................14-16
Message Registers.............................................................................................................14-17
Punch-Through Configuration Requests ....................................................................................14-18
Re-programming the Bus Number of the NT Function...............................................................14-19
Interrupts.....................................................................................................................................14-20
Virtual Channel Support..............................................................................................................14-21
Maximum Payload Size..............................................................................................................14-22
Power Management...................................................................................................................14-22
Bus Locking................................................................................................................................14-22
ECRC Support............................................................................................................................14-22
Access Control Services (ACS)..................................................................................................14-23
Error Detection and Handling by the NT Function......................................................................14-25
Physical Layer Errors .........................................................................................................14-26
Data Link Layer Errors........................................................................................................14-26
Transaction Layer Errors....................................................................................................14-26
NTB Inter-Partition Error Propagation ................................................................................14-31
Error Emulation Control in the NT Function........................................................................14-39
Non Transparent Operation Restrictions....................................................................................14-40
DMA Controller
Overview.......................................................................................................................................15-1
Base Address Registers...............................................................................................................15-1
DMA Channel Functional Description...........................................................................................15-1
Data Transfer and Addressing..............................................................................................15-2
DMA Descriptors ..................................................................................................................15-6
DMA Descriptor Processing ...............................................................................................15-15
TLP Attribute and Traffic Class Control..............................................................................15-20
Channel Interrupts..............................................................................................................15-21
DMA Outstanding Requests...............................................................................................15-21
Descriptor Prefetching........................................................................................................15-22
DMA Request Rate Control................................................................................................15-22

IDT Table of Contents
PES32NT24xG2 User Manual vi January 30, 2013
Notes
DMA Multicast ....................................................................................................................15-23
Interrupts.....................................................................................................................................15-24
Virtual Channel (VC) Support.....................................................................................................15-25
Access Control Services (ACS) Support ....................................................................................15-25
Power Management....................................................................................................................15-27
Bus Locking................................................................................................................................15-27
ECRC Support............................................................................................................................15-27
Error Handling.............................................................................................................................15-27
PCI Express Error Handling by the DMA Function.............................................................15-28
DMA Limitations and Usage Restrictions ...................................................................................15-36
Switch Events
Overview.......................................................................................................................................16-1
Switch Events...............................................................................................................................16-1
Link Up .................................................................................................................................16-2
Link Down.............................................................................................................................16-3
Fundamental Reset..............................................................................................................16-3
Hot Reset..............................................................................................................................16-3
Failover.................................................................................................................................16-3
Global Signals ......................................................................................................................16-4
Port AER Errors....................................................................................................................16-5
Multicast
Overview.......................................................................................................................................17-1
Transparent Multicast Operation ..................................................................................................17-1
Addressing and Routing.......................................................................................................17-1
Usage Restrictions ...............................................................................................................17-6
Non-Transparent Multicast Operation...........................................................................................17-6
NT Multicast Configuration...................................................................................................17-7
NT Multicast TLP Determination...........................................................................................17-8
NT Multicast TLP Routing.....................................................................................................17-8
NT Multicast Egress Processing...........................................................................................17-9
Usage Restrictions .............................................................................................................17-11
Temperature Sensor
Overview.......................................................................................................................................18-1
Register Organization
Overview.......................................................................................................................................19-1
Partial-Byte Access to Word and DWord Registers .............................................................19-3
Configuration Register Side-Effects .....................................................................................19-3
Address Maps...............................................................................................................................19-4
PCI-to-PCI Bridge Function Registers..................................................................................19-4
Proprietary Port-Specific Registers in the PCI-to-PCI Bridge Function..............................19-11
NT Function Registers........................................................................................................19-14
DMA Function Registers.....................................................................................................19-23
Switch Configuration and Status Registers........................................................................19-29

IDT Table of Contents
PES32NT24xG2 User Manual vii January 30, 2013
Notes
PCI-to-PCI Bridge Registers
Type 1 Configuration Header Registers .......................................................................................20-1
PCI Express Capability Structure ...............................................................................................20-13
PCI Power Management Capability Structure............................................................................20-36
Message Signaled Interrupt Capability Structure .......................................................................20-38
Subsystem ID and Subsystem Vendor ID ..................................................................................20-39
Extended Configuration Space Access Registers......................................................................20-40
Advanced Error Reporting (AER) Extended Capability...............................................................20-41
Device Serial Number Extended Capability................................................................................20-51
PCI Express Virtual Channel Capability .....................................................................................20-52
ACS Extended Capability ...........................................................................................................20-55
Multicast Extended Capability.....................................................................................................20-60
Proprietary Port Specific Registers
Port Control Register....................................................................................................................21-1
Upstream PCI-to-PCI Bridge Interrupt and Signaling...................................................................21-1
Port AER Mask Register...............................................................................................................21-3
Port Slot Control ...........................................................................................................................21-5
Internal Error Control and Status Registers..................................................................................21-7
Physical Layer Control and Status Registers .............................................................................21-28
Request Metering .......................................................................................................................21-32
WRR Port Arbitration Counts......................................................................................................21-33
Non-Transparent Multicast Overlay............................................................................................21-38
AER Error Emulation ..................................................................................................................21-40
Global Address Space Access Registers...................................................................................21-43
NT Endpoint Registers
Type 0 Configuration Header Registers .......................................................................................22-1
PCI Express Capability Structure ...............................................................................................22-13
PCI Power Management Capability Structure............................................................................22-28
Message Signaled Interrupt Capability Structure .......................................................................22-29
Subsystem ID and Subsystem Vendor ID ..................................................................................22-31
Extended Configuration Space Access Registers......................................................................22-31
Advanced Error Reporting (AER) Extended Capability...............................................................22-33
Device Serial Number Extended Capability................................................................................22-43
PCI Express Virtual Channel Capability .....................................................................................22-44
ACS Extended Capability ...........................................................................................................22-48
Multicast Extended Capability.....................................................................................................22-50
NT Registers...............................................................................................................................22-53
NT Control & Status............................................................................................................22-53
NT Interrupt and Signaling..................................................................................................22-54
Internal Error Reporting Masks...........................................................................................22-56
Doorbell Registers..............................................................................................................22-64
Message Registers.............................................................................................................22-65
BAR Configuration..............................................................................................................22-67
Mapping Table....................................................................................................................22-85
Lookup Table..............................................................................................................................22-88
AER Error Emulation ..................................................................................................................22-89
Punch-Through Configuration Registers ....................................................................................22-92
NT Multicast................................................................................................................................22-94
Global Address Space Access Registers...................................................................................22-95

IDT Table of Contents
PES32NT24xG2 User Manual viii January 30, 2013
Notes
DMA Function Registers
Type 0 Configuration Header Registers .......................................................................................23-1
PCI Express Capability Structure .................................................................................................23-9
PCI Power Management Capability Structure............................................................................23-22
Message Signaled Interrupt Capability Structure .......................................................................23-24
Extended Configuration Space Access Registers......................................................................23-25
Advanced Error Reporting (AER) Extended Capability...............................................................23-26
ACS Extended Capability ...........................................................................................................23-37
DMA Registers............................................................................................................................23-39
BAR Configuration..............................................................................................................23-39
DMA AER Error Emulation.................................................................................................23-39
Internal Error Reporting Masks...........................................................................................23-42
DMA Multicast Control........................................................................................................23-49
DMA Channel Registers.....................................................................................................23-50
Global Address Space Access Registers...................................................................................23-59
Switch Configuration and Status Registers
Switch Control and Status Registers............................................................................................24-1
Internal Switch Timers..................................................................................................................24-5
Switch Partition and Port Registers..............................................................................................24-7
Failover Capability Registers......................................................................................................24-13
Protection....................................................................................................................................24-15
Switch Event Registers...............................................................................................................24-16
Global Doorbells and Message Registers ..................................................................................24-25
SerDes Control and Status Registers.........................................................................................24-26
General Purpose I/O Registers...................................................................................................24-33
Hot-Plug and SMBus Interface Registers...................................................................................24-35
Temperature Sensor Registers...................................................................................................24-45
JTAG Boundary Scan
Introduction...................................................................................................................................25-1
Test Access Point.........................................................................................................................25-1
Signal Definitions..........................................................................................................................25-1
Boundary Scan Chain...................................................................................................................25-3
Test Data Register (DR)...............................................................................................................25-6
Boundary Scan Registers.....................................................................................................25-7
Instruction Register (IR)................................................................................................................25-8
EXTEST................................................................................................................................25-9
SAMPLE/PRELOAD.............................................................................................................25-9
BYPASS...............................................................................................................................25-9
CLAMP...............................................................................................................................25-10
IDCODE..............................................................................................................................25-10
VALIDATE..........................................................................................................................25-10
EXTEST_TRAIN.................................................................................................................25-10
EXTEST_PULSE................................................................................................................25-11
RESERVED........................................................................................................................25-11
Usage Considerations........................................................................................................25-11
Usage Models
Introduction...................................................................................................................................26-1
Boot-time Stack Reconfiguration..................................................................................................26-1
Port Clocking Configuration..........................................................................................................26-2
Boot-time Switch Partitioning........................................................................................................26-3

IDT Table of Contents
PES32NT24xG2 User Manual ix January 30, 2013
Notes
Switch Partitioning via serial EEPROM................................................................................26-4
Switch Partitioning via PCI Express Configuration Requests...............................................26-5
Dynamic Port and Partition Reconfiguration.................................................................................26-8
I/O Load Balancing: Downstream Port Migration .................................................................26-8
Non-Transparent Bridge (NTB) Usage Models...........................................................................26-11
PES32NT24xG2 as a Multiprocessor System Interconnect...............................................26-11
NT Crosslink & NT Punch-Through....................................................................................26-15
DMA Usage Models....................................................................................................................26-17
High-Performance Multiprocessor System.........................................................................26-17
Immediate Descriptor Usage..............................................................................................26-20
Failover.......................................................................................................................................26-20
Active / Passive Failover Configuration..............................................................................26-20
Active / Active Failover Configuration.................................................................................26-23
Failover with Two Crosslinked PES32NT24xG2 Switches.................................................26-27
NT Multicasting...........................................................................................................................26-30

IDT Table of Contents
PES32NT24xG2 User Manual x January 30, 2013
Notes