Sun Microsystems SPARC T5120 User manual

SUN SPARC
®
ENTERPRISE
T5120 AND T5220
SERVER ARCHITECTURE
Unleashing the UltraSPARC
®
T2 Processor with
CoolThreads
™ Technology
White Paper
October 2007

Sun Microsystems, Inc.
Table of Contents
Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
The Evolution of Chip Multithreading (CMT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Business Challenges for Web 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Rule-Changing Chip Multithreading (CMT) Technology . . . . . . . . . . . . . . . . . . . . . . . 3
Sun SPARC
®
Enterprise T5120 and T5220 Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Space, Watts, and Performance: Introducing the SWaP Metric . . . . . . . . . . . . . . . . 10
The UltraSPARC
®
T2 Processor with CoolThreads
™
Technology . . . . . . . . . . . . . . 11
UltraSPARC T2: The World's First Massively Threaded System on a Chip (SoC) . . . . 11
Taking Chip Multithreaded Design to the Next Level . . . . . . . . . . . . . . . . . . . . . . . . 12
UltraSPARC T2 Processor Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Sun SPARC Enterprise T5120 and T5220 Server Architecture . . . . . . . . . . . . . . . . 19
System-Level Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Sun SPARC Enterprise T5120 Server Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Sun SPARC Enterprise T5220 Server Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
System Management Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Enterprise-Class Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Scalability and Support for CoolThreads Technology . . . . . . . . . . . . . . . . . . . . . . . . 28
Solaris
™
CoolTools for SPARC: Performance and Rapid Time-to-Market . . . . . . . . . . 33
Sun Java
™
Enterprise System (Java ES) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
For More Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Executive Summary Sun Microsystems, Inc.
Executive Summary
Use of the Web is changing in fundamental ways, driven by Web 2.0 applications and
the thousands of people who join the global Internet every day through a proliferation
of new interactive devices. The character of applications and services is changing too.
Increasingly, user's don't need to install anything, upgrade anything, license anything,
subscribe to anything, or even buy anything in order to participate and transact. Web
users can even interact directly with content, changing it and improving it. Intellectual
property is shared, rather than locked away, and the most popular services are available
free of charge. Even very small transactions are now encouraged, becoming large in
aggregate. Social networking and other collaborative sites let like-minded people from
around the world share information on enormous range of topics and issues. Business
transactions too are now predominantly Web based.
Serving this dynamic and growing space is becoming very challenging for datacenter
operations. Services need to be able to start small and scale very rapidly, often doubling
capacity every three months even as they remain highly available. Infrastructure must
keep up with these enormous scalability demands, without generating additional
administrative burden. Unfortunately, most datacenters are already severely
constrained by both real estate and power — and energy costs are rising. There is also
a new appreciation for the role that the datacenter plays in reducing energy
consumption and pollution. Virtualization has emerged as an extremely important tool
as organizations seek to consolidate redundant infrastructure, simplify administration,
and leverage under-utilized systems. Security too has never been more important, with
increasing price of data loss and corruption. In addressing these challenges,
organizations can ill afford proprietary infrastructure that imposes arbitrary limitations.
Employing the UltraSPARC
®
T2 processor — the industry’s first massively threaded
system on a Chip (SoC) — Sun SPARC
®
Enterprise T5120 and T5220 servers offer
breakthrough performance and energy efficiency to drive Web 2.0 infrastructure and
address other demanding datacenter challenges. Next-generation CoolThreads
™
chip
multithreading (CMT) technology supports up to 64 threads in as little as one rack unit
(RU) — providing increased computational density while staying within variously
constrained envelopes of power and cooling. Very high levels of integration help reduce
latency, lower costs, and improve security and reliability. Balanced system design
provides support for a wide range of application types — from Web services to high
performance computing (HPC). Uniformity of management interfaces and adoption of
standards helps reduce administrative costs. With both the processor and the Solaris
™
Operating System (Solaris OS) available under open source licensing, organizations are
free to innovate and join with a world-wide technical community.

2
Th
e
E
vo
l
u
ti
on o
f
Chi
p
M
u
ltith
rea
di
ng
(CMT)
Sun Microsystems, Inc.
Chapter 1
The Evolution of Chip Multithreading (CMT)
By any measure, Sun’s first-generation CMT processors were an unprecedented success.
Sun Fire
™
/ Sun SPARC Enterprise T1000 and T2000 servers based on the UltraSPARC T1
processor with CoolThreads technology won enthusiastic praise, and generated the
fastest product ramp in Sun’s history. Delivering up to five times the throughput in a
quarter of the space and power, these systems even garnered the first ever rebate from
a power utility
1
— a trend that is being repeated across the world. Now CMT
technology is evolving rapidly to meet the constantly changing demands of a wide
range of Web and other applications.
Business Challenges for Web 2.0
Marked by the prevalence of Web services and service-oriented architecture (SOA), the
emerging Participation Age promises the ability to deliver rich new content and high-
bandwidth services to larger numbers of users than ever before. Through this
transition, organizations across many industries hope to address larger markets, reduce
costs, and gain better insights into their customers. At the same time, an increasingly
broad array of wired and wireless client devices are bringing network computing into
the everyday lives of millions of people. These trends are redefining datacenter
scalability and capacity requirements, even as they collide with fundamental real
estate, power, and cooling constraints.
• Building out for Web Scale
Web scale applications engender a new pace and urgency to infrastructure
deployment. Organizations must accelerate time to market and time to service,
while delivering scalable high-quality and high-performance applications and
services. Many need to be able to start small with the ability to scale very quickly,
with new customers and innovative new Web services often implying a doubling of
capacity in months rather than years.
At the same time, organizations must reduce their environmental impact by
working within the power, cooling, and space available in their current
datacenters. Operational costs too are receiving new scrutiny, along with system
administrative costs that can account for up to 40 percent of an IT budget.
Simplicity and speed are paramount, giving organizations the ability to respond
quickly to dynamic business conditions. Organizations are also striving to
eliminate vendor lock-in as they look to preserve previous, current, and future
investments. Open platforms built around open standards help provide maximum
flexibility while reducing costs of both entry and exit.
1.In August of 2006, Pacific Gas and Electric (PG&E) began offering a substantial energy rebate for
purchasing and deploying Sun Fire / Sun SPARC Enterprise T1000 and T2000 servers

3
Th
e
E
vo
l
u
ti
on o
f
Chi
p
M
u
ltith
rea
di
ng
(CMT)
Sun Microsystems, Inc.
• Driving Datacenter Virtualization and Eco-Efficiency
Coincident with the need to scale services, many datacenters are recognizing the
advantages of deploying fewer standard platforms to run a mixture of commercial
and technical workloads. This process involves consolidating under-utilized and
often sprawling server infrastructures with effective virtualization solutions that
serve to enhance business agility, improve disaster recovery, and reduce operating
costs. This focus can help reduce energy costs and break through datacenter
capacity constraints by improving the amount of realized performance for each
watt of power the datacenter consumes.
Eco-efficiency provides tangible benefits, improving ecology by reducing carbon
footprint to meet legislative and corporate social responsibility goals, even as it
improves the
economy of the organization paying the electric bill. As systems are
consolidated onto more dense and capable computing infrastructure, demand for
datacenter real estate is also reduced. With careful planning, this approach can
also improve service uptime and reliability by reducing hardware failures resulting
from excess heat load. Servers with high levels of standard reliability, availability,
and serviceability (RAS) are now considered a requirement.
• Securing the Enterprise at Speed:
Organizations are increasingly interested in securing all communications with
their customers and partners. Given the risks, end-to-end encryption is essential in
order to inspire confidence in security and confidentiality. Encryption is also
increasingly important for storage, helping to secure stored and archived data
even as it provides a mechanism to detect tampering and data corruption.
Unfortunately, the computational costs of increased encryption can increase the
burden on already over-taxed computational resources. Security also needs to take
place at line speed, without introducing bottlenecks that can impact the customer
experience or slow transactions. Solutions must help to ensure security and
privacy for clients and bring business compliance for the organization, all without
impacting performance or increasing costs.
Rule-Changing Chip Multithreading (CMT) Technology
Addressing these challenges has outstripped the capabilities of traditional processors
and systems, and required a fundamentally new approach.
Moore’s Law, and the Diminishing Returns of Traditional Processor
Design
The oft-quoted tenant of Moore’s Law states that the number of transistors that will fit
in a square inch of integrated circuitry will approximately double every two years. For
over three decades the pace of Moore’s law has held, driving processor performance to
new heights. Processor manufacturers have long exploited these gains in chip real

4
Th
e
E
vo
l
u
ti
on o
f
Chi
p
M
u
ltith
rea
di
ng
(CMT)
Sun Microsystems, Inc.
estate to build increasingly complex processors, with instruction-level parallelism (ILP)
as a goal. Today these traditional processors employ very high frequencies along with a
variety of sophisticated tactics to accelerate a single instruction pipeline, including:
• Large caches
• Superscalar designs
• Out-of-order execution
• Very high clock rates
• Deep pipelines
• Speculative pre-fetches
While these techniques have produced faster processors with impressive-sounding
multiple-gigahertz frequencies, they have largely resulted in complex, hot, and power-
hungry processors that are not well suited to the types of workloads often found in
modern datacenters. In fact, many datacenter workloads are simply unable to take
advantage of the hard-won ILP provided by these processors. Applications with high
shared memory and high simultaneous user or transaction counts are typically more
focused on processing a large number of simultaneous threads (thread-level
parallelism, TLP) rather than running a single thread as quickly as possible (ILP).
Making matters worse, the majority of ILP in existing applications has already been
extracted and further gains promise to be small. In addition, microprocessor frequency
scaling itself has leveled off because of microprocessor power issues. With higher clock
speeds, each successive processor generation has seemingly demanded more power
than the last, and microprocessor frequency scaling has leveled off in the 2-3 GHz range
as a result. Deploying pipelined Superscalar processors requires more power, limiting
this approach by the fundamental ability to cool the processors.
Chip Multiprocessing with Multicore Processors
To address these issues, many in the microprocessor industry have used the transistor
budget provided by Moore's Law to group two or even four conventional processor
cores on a single physical die — creating multicore processors (or chip multiprocessors,
CMP). The individual processor cores introduced by many CMP designs have no greater
performance than previous single-processor chips, and in fact, have been observed to
run single-threaded applications more slowly than single-core processor versions.
However, the aggregate chip performance increases since multiple programs (or
multiple threads) can be accommodated in parallel (thread level parallelism).
Unfortunately, most currently-available (or soon to be available) chip multiprocessors
simply replicate cores from existing (single-threaded) processor designs. This approach
typically yields only slight improvements in aggregate performance since it ignores key
performance issues such as memory speed and hardware thread context switching. As

5
Th
e
E
vo
l
u
ti
on o
f
Chi
p
M
u
ltith
rea
di
ng
(CMT)
Sun Microsystems, Inc.
a result, while these designs provide some additional throughput and scalability, they
can consume considerable power and generate significant heat — without a
commensurate increase in overall performance.
Chip Multithreading (CMT) with CoolThreads
™
Technology
Sun engineers were early to recognize the disparity between processor speeds and
memory access rates. While processor speeds continue to double every two years,
memory speeds have typically doubled only every six years. As a result, memory latency
now dominates much application performance, erasing even very impressive gains in
clock rates. This growing disconnect is the result of memory suppliers focusing on
density and cost as their design center, rather than speed.
Unfortunately, this relative gap between processor and memory speeds leaves ultra-fast
processors idle as much as 85 percent of the time, waiting for memory transactions to
complete. Ironically, as traditional processor execution pipelines get faster and more
complex, the effect of memory latency grows — fast, expensive processors spend more
cycles doing nothing. Worse still, idle processors continue to draw power and generate
heat. It is easy to see that frequency (gigahertz) is truly a misleading indicator of real
performance.
First introduced with the UltraSPARC T1 processor, chip multithreading takes advantage
of CMP advances, but adds a critical capability — the ability to scale with threads
rather than frequency . Unlike traditional single-threaded processors and even most
current multicore (CMP) processors, hardware multithreaded processor cores allow
rapid switching between active threads as other threads stall for memory. Figure 1
illustrates the difference between CMP, fine-grained hardware multithreading (FG-MT),
and chip multithreading. The key to this approach is that each core in a CMT processor
is designed to switch between multiple threads on each clock cycle. As a result, the
processor’s execution pipeline remains active doing real useful work, even as memory
operations for stalled threads continue in parallel.
Figure 1. Chip multithreading combines CMP and fine-grained hardware multithreading
Chip
Multiprocessing
(CMP)
Fine-Grained
Multithreading
(FG-MT)
Chip
Multithreading
(CMT)
(n cores
per processor)
(m strands
per core)
(n x m threads
per processor)
Memory Latency Compute

6
Th
e
E
vo
l
u
ti
on o
f
Chi
p
M
u
ltith
rea
di
ng
(CMT)
Sun Microsystems, Inc.
Chip multithreading provides real value since it increases the ability of the execution
pipeline to do actual work on any given clock cycle. Utilization of the processor pipeline
is greatly enhanced since a number of execution threads now share its resources. The
negative effects of memory latency are effectively masked, since the processor and
memory subsystems remain active in parallel to the processor execution pipeline.
Because these individual processor cores implement much simpler pipelines that focus
on scaling with threads rather than frequency (emphasizing TLP over ILP), they are also
substantially cooler and require significantly less electrical energy to operate. This
innovative approach results in CoolThreads processor technology — multiple physical
instruction execution pipelines (one for each core), with multiple active thread contexts
per core.
The UltraSPARC
®
T2 Processor with CoolThreads Technology
Unlike complex single-threaded processors, CMT processors utilize the available
transistor budget to implement multiple hardware multithreaded processor cores on a
chip die. The UltraSPARC T2 processor takes the CMT model to the next level, providing
up to eight cores with each core supporting up to eight threads via two independent
pipelines per core — effectively doubling the throughput of the UltraSPARC T1
processor. In addition, the UltraSPARC T2 processor uses this transistor budget to
implement the industry’s first massively threaded System on a Chip (SoC), with a single
processor die hosting:
• Up to 64 threads per processor (up to eight cores supporting eight threads each)
• On-chip Level-1 and Level-2 caches
• Per-core floating point capabilities
• Per-core cryptographic acceleration
• Two on-chip 10 Gb Ethernet interfaces
• On-chip PCI Express interface
Through this SoC design, the UltraSPARC T2 processor significantly enhances the
general purpose nature of the CPU by building in eight floating point units (1 per core).
Enhanced floating point capabilities open the UltraSPARC T2 to the world of compute-
intensive applications as well as the traditionally CMT friendly datacenter throughput
applications. No-cost security and cryptographic acceleration is provided by the on-chip,
per-core streaming accelerators. In addition, the ability to move data in and out of the
processor is significantly aided by an integrated PCI-Express interface and dual
10 Gigabit Ethernet interfaces.
Sun SPARC
®
Enterprise T5120 and T5220 Servers
Sun SPARC Enterprise T5120 and T5220 servers (Figure 2) are designed to leverage the
considerable resources of the UltraSPARC T2 processor in the form of cost-effective
general-purpose platforms. These systems deliver up to twice the throughput of their

7
Th
e
E
vo
l
u
ti
on o
f
Chi
p
M
u
ltith
rea
di
ng
(CMT)
Sun Microsystems, Inc.
predecessors, while leading competitors in terms of performance, performance per
watt, and SWaP performance (as evaluated by the Space, Watts, and Performance
metric detailed later in this section). These systems also extend the benefits of CMT
from multithreaded commercial workloads into technical workloads rich in floating
point operations.
With support for 64 threads, large memory, cryptographic acceleration, and integrated
on-chip 10 Gb Ethernet networking and I/O technology, these servers represent a
departure from traditional system design. The 1U Sun SPARC Enterprise T5120 is ideal
for providing high throughput within significant power, cooling, and space constraints,
delivering a 64-thread UltraSPARC T2 processor in a space- and power-efficient 1U
rackmount package. As a compute node with massively horizontally scaled
environments, the Sun SPARC Enterprise T5120 can help provide a substantial building
block for application tier, Web services, or even high performance computing (HPC)
infrastructure. Network infrastructure applications such as portal, directory, network
identity, file service, and backup are all a good fit for this server.
The Sun SPARC Enterprise T5220 server provides both throughput as well as
expandability, with extra I/O and internal disk options afforded by the 2U rackmount
form factor. With greater I/O and internal disk, typical workloads include demanding
mid-tier application server deployments or Web- and application-tier consolidation,
virtualization, and consolidation projects requiring maximum uptime with future
growth and integration into diverse environments. The Sun SPARC Enterprise T5220 is
also ideal for OLTP database deployments.
Figure 2. Sun SPARC Enterprise T5120 and T5220 servers

8
Th
e
E
vo
l
u
ti
on o
f
Chi
p
M
u
ltith
rea
di
ng
(CMT)
Sun Microsystems, Inc.
Designed to complement each other, as well as the rest of Sun’s server product line, the
Sun SPARC Enterprise T5120 and T5220 servers address the dynamic needs of the
modern datacenter.
• Efficient and Predictable Scalability
With support for 64 threads and large memories, Sun SPARC Enterprise T5120 and
T5220 servers are the first to utilize the 10 Gb Ethernet, I/O, and cryptographic
acceleration provided directly by the UltraSPARC T2 processor chip. This approach
provides leading levels of performance and scalability with extremely high levels
of power, heat, and space efficiency.
• Accelerated Time to Market
Sun SPARC Enterprise T5120 and T5220 servers running the Solaris OS provide full
binary compatibility with earlier UltraSPARC systems, preserving investments and
speeding time to market. Sun’s CoolTools for SPARC help accelerate application
selection, profiling, testing, tuning, debugging and deployment of key
applications on CMT systems.
• Simplified Management
Each Sun SPARC Enterprise T5120 and T5220 server provides an Integrated Lights
Out Management (ILOM) service processor, compatible with Sun’s x64 servers.
ILOM provides a command line interface (CLI), a Web-based graphical user
interface (GUI), and Intelligent Platform Management Interface (IPMI)
functionality to aid with out-of-band monitoring and administration. ILOM on
these systems also provides an Advanced Lights Out Management (ALOM)
backwards compatibility mode for administrators familiar with Sun Fire / Sun
SPARC Enterprise T1000 and T2000 servers.
• The Industry's Most Open Platform
Sun SPARC Enterprise servers are the industry’s most open platforms, providing
the only mainstream processor and hypervisor offered under the GNU General
Public License (GPL). These systems offer a choice of operating systems, including
the Solaris OS, Linux, and BSD variants. The Solaris OS is free and open, offering
full binary compatibility and enterprise-class features. Sun Java
™
Enterprise
System middleware is pre-loaded.
• Industry-Leading Tools for Virtualization and Consolidation
Sun’s chip multithreading (CMT) technology is ideal for consolidation, providing
low-level multithreading support for virtualization at every layer of the technology
stack. Sun Logical Domains exploit the UltraSPARC T2 processor’s 64 threads to
support multiple guest operating system instances, while Solaris Containers
provide virtualization within a single Solaris OS instance. The advanced ZFS file
system provides storage virtualization for storage and considerable scalability.

9
Th
e
E
vo
l
u
ti
on o
f
Chi
p
M
u
ltith
rea
di
ng
(CMT)
Sun Microsystems, Inc.
• A Tradition of Leading Eco Efficiency
Sun Fire / Sun SPARC Enterprise T1000 and T2000 servers were the industry's first
eco-responsible servers. Sun SPARC Enterprise T5120 and T5220 servers continue
this tradition by offering the best performance and performance-per-watt across a
wide range of commercial and technical workloads. In addition, the UltraSPARC T2
processor is the first and only processor to incorporate unique power
management features at both core and memory levels of the processor.
• System and Datacenter Reliability
Reliability is key to keeping applications available and costs down. With the
greater levels of integration provided by an SoC design, the Sun SPARC Enterprise
T5120 and T5220 servers offer greatly reduced part counts, and provide
commensurately higher levels of reliability, availability, and serviceability (RAS).
Lower power consumption and higher performance per watt greatly reduce
generated heat loads and the associated issues they cause. Technologies such as
Solaris Predictive Self Healing are integrated with the hardware, and help keep
systems available.
• Zero-Cost Security
Providing secure communications and data protection has never been more
important, with attempted electronic intrusion and theft at an all-time high. With
up to eight integrated cryptographic accelerators on each UltraSPARC T2
processor, there is simply no need to send plain text on the network or store plain
text in storage systems. Sun SPARC Enterprise T5120 and T5220 servers support
many more crypto operations per second than competitive processors and
dedicated crypto accelerator cards, with minimal impact to system overhead.
Table 1 compares the features of Sun SPARC Enterprise T5120 and T5220 servers.
Table 1. Sun SPARC Enterprise T5120 and T5220 server features
Feature Sun SPARC Enterprise T5120
Server
Sun SPARC Enterprise T5220
Server
CPUs Four-, six-, or eight-core 1.2 GHz
or 1.4 GHz UltraSPARC T2
processor
Four-, six-, or eight-core 1.2 GHz
or eight-core 1.4 GHz UltraSPARC
T2 processor
Threads Up to 64 Up to 64
Memory capacity Up to 64 GB
(1, 2, or 4 GB FBDIMM)
Up to 64 GB
(1, 2, of 4 GB FBDIMM)
Maximum internal
disk drives
Up to four SFF 2.5-inch SAS
73 or 146 GB disk drives,
RAID 0/1
Up to eight SFF SAS 2.5-inch SAS
73 or 146 GB disk drives,
RAID 0/1
Removable/pluggable
I/O
Slimline DVD-R
Four USB 2.0 ports
Slimline DVD-R
Four USB 2.0 ports
PCI One x8 PCI Express slot,
Two x4 PCI Express or XAUI
combo slots
a
Two x8 PCI Express
Two x4 PCI Express
Two x4 PCI Express or XAUI
combo slots
a

10
Th
e
E
vo
l
u
ti
on o
f
Chi
p
M
u
ltith
rea
di
ng
(CMT)
Sun Microsystems, Inc.
Space, Watts, and Performance: Introducing the SWaP Metric
Sun SPARC Enterprise T5120 and T5220 servers deliver leading performance across a
range of multithreaded workloads and benchmarks. However, with energy and real
estate costs and pressures, it is not enough to measure performance in isolation.
Delivering the required level of throughput in a fixed space and power envelope is
critical. Traditional system-to-system benchmarks are valuable as a way of comparing
one system to another, but are limited when it comes to understanding the power and
density attributes of the systems being compared. For this reason, Sun has developed
the SWaP metric, standing for Space, Watts, and Performance . Designed to provide a
simple and transparent measure of overall server efficiency, SWaP is calculated using
the following formula:
SWaP = Performance / (Space * Power Consumption) where,
• Performance is measured by industry-standard audited benchmarks such as those
sponsored by the Systems Performance Evaluation Corporation (SPEC)
• Space refers to the height of the server in rack units (RUs)
• Power is measured by watts used by the system, taken during actual benchmark runs
or from vendor’s site planning guides
For the latest SWaP results for a variety of benchmarks, please see sun.com/servers/
coolthreads/benchmarks .
Ethernet Four on-board Gigabit Ethernet
ports (10/100/1000)
Two 10 Gb Ethernet ports via
XAUI combo slots
Four on-board Gigabit Ethernet
ports (10/100/1000)
Two 10 Gb Ethernet ports via
XAUI combo slots
Power supplies Two hot-swappable AC 650W or
DC 660W power supply units
(N+1 redundancy)
Two hot-swappable AC 750W
power supply units
(N+1 redundancy)
Fans Four hot-swappable fan trays,
with 2 fans per tray,
N+1 redundancy
3 hot-swappable fan trays, with
2 fans per tray, N+1 redundancy
Form factor 1 rack unit (1U) 2 rack units (2U)
a.Optional XAUI adapter cards required for access to dual 10 Gb Ethernet ports on the
UltraSPARC T2 processor
Feature Sun SPARC Enterprise T5120
Server
Sun SPARC Enterprise T5220
Server

11
Th
e
Ult
ra
SPARC
T
2
P
rocessor w
ith
C
oo
lTh
rea
d
s
T
ec
h
no
l
ogy
Sun Microsystems, Inc.
Chapter 2
The UltraSPARC T2 Processor with CoolThreads
Technology
The UltraSPARC T2 processor is the industry’s first system on a chip (SoC), supplying the
most cores and threads of any general-purpose processor available, and integrating all
key system functions.
UltraSPARC T2: The World's First Massively Threaded System
on a Chip (SoC)
The UltraSPARC T2 processor eliminates the need for expensive custom hardware and
software development by integrating computing, networking, security, and I/O on to a
single chip. Binary compatible with earlier UltraSPARC processors, no other processor
delivers so much performance in so little space and with such small power
requirements — letting organizations rapidly scale the delivery of new network services
with maximum efficiency and predictability. The UltraSPARC T2 processor is shown in
Figure 3, to the left of the previous-generation UltraSPARC T1 processor. Even with twice
the computational throughput and significantly higher levels of integration, the
UltraSPARC T2 processor is physically smaller than the UltraSPARC T1 processor.
Figure 3. The UltraSPARC T2 (left) and UltraSPARC T1 Processors with CoolThreads Technology
Each UltraSPARC T2 processor supports:
• Up to eight cores @ 1.2 Ghz – 1.4 Ghz
• Eight threads per core for a total maximum of 64 threads per processor
• 4 MB L2 cache in eight banks (16-way set associative)
• Four on-chip memory controllers for support of up to 16 FBDIMMs
• Up to 64 GB of memory (4 GB FBDIMMs) with 60 GB/s memory bandwidth
• Eight fully pipelined floating point units (1 per core)
• Dual on-chip 10 Gb Ethernet interfaces
• Integral PCI-Express interface
• Eight stream processing units (cryptographic co-processors), one per core

12
Th
e
Ult
ra
SPARC
T
2
P
rocessor w
ith
C
oo
lTh
rea
d
s
T
ec
h
no
l
ogy
Sun Microsystems, Inc.
Table 2 provides a comparison between the UltraSPARC T2 and the UltraSPARC T1
processor.
Table 2. UltraSPARC T1 and T2 processor features
Taking Chip Multithreaded Design to the Next Level
When Sun’s in-house design team set out to design the next-generation of CMT
processors, they started with key goals in mind:
• Increasing computational capabilities to meet the growing demand from Web
applications by providing twice the throughput of the UltraSPARC T1 processor
• Supporting larger and more diverse workloads with greater floating point
performance
• Powering faster networking to serve new network-intensive content
• Providing end-to-end datacenter encryption
• Increasing service levels and reducing downtime
• Improving datacenter capacities while reducing costs
CMT architecture is ultimately very flexible, allowing different modular combinations of
processors, cores, and integrated components. The considerations listed above drove
an internal engineering effort that compared different approaches with regard to
making improvements on the successful UltraSPARC T1 architecture. For example,
Feature UltraSPARC T1 Processor UltraSPARC T2 processor
Cores per processor Up to 8 Up to 8
Threads per core 48
Threads per processor 32 64
Hypervisor Yes Yes
Memory 4 memory controllers,
4 DIMMs per controller
4 memory controllers,
up to 64 FBDIMMs
Caches 16 KB instruction cache,
8 KB data cache,
3 MB L2 cache (4 banks,
12-way associative)
16 KB instruction cache,
8 KB data cache,
4 MB L2 cache (8 banks,
16-way associative)
Technology 9-layer Cu metal, CMOS
process, 90 nm technology
65 nm technology
Floating point 1 FPU per chip 1 FPU per core,
8 FPUs per chip
Integer resources Single execution unit 2 integer execution units per
core
Cryptography Accelerated modular
arithmetic operations (RSA)
Stream processing unit per
core, support for the 10 most
popular ciphers
Additional on-chip
resources
— Dual 10 Gb Ethernet
interfaces,
PCI Express interface (x8)

13
Th
e
Ult
ra
SPARC
T
2
P
rocessor w
ith
C
oo
lTh
rea
d
s
T
ec
h
no
l
ogy
Sun Microsystems, Inc.
simply increasing the number of cores would have gained additional throughput, but
would have resulted in consuming extra die area, leaving no room for integrated
components such as floating point processors.
The final UltraSPARC T2 processor design recognizes that memory latency is truly the
bottleneck to improving performance. By increasing the number of threads supported
by each core, and by further increasing network bandwidth, the UltraSPARC T2 is able
provide approximately twice the throughput of the UltraSPARC T1 processor. Each
UltraSPARC T2 processor provides up to eight cores, with each core able to switch
between up to eight threads (64 threads per processor). In addition, each core provides
two integer execution units, so that a single UltraSPARC core is capable of executing
two threads at a time. Figure 4 provides a simplified high-level illustration of the thread
model supported by an eight-core UltraSPARC T2 processor.
Figure 4. A single eight-core UltraSPARC T2 processor supports up to 64 threads, with up to two threads
running in each core simultaneously
Core 1
Thread 1
Thread 8
.
.
.
Core 2
Core 3
Core 4
Core 5
Core 6
Core 7
Core 8
TimeMemory Latency Compute
Thread 1
Thread 8
.
.
.
Thread 1
Thread 8
.
.
.
Thread 1
Thread 8
.
.
.
Thread 1
Thread 8
.
.
.
Thread 1
Thread 8
.
.
.
Thread 1
Thread 8
.
.
.
Thread 1
Thread 8
.
.
.

14
Th
e
Ult
ra
SPARC
T
2
P
rocessor w
ith
C
oo
lTh
rea
d
s
T
ec
h
no
l
ogy
Sun Microsystems, Inc.
UltraSPARC T2 Processor Architecture
The UltraSPARC T2 processor extends Sun’s Throughput Computing initiative with an
elegant and robust architecture that delivers real performance to applications. A high-
level block diagram of the UltraSPARC T2 processor is shown in Figure 5.
Figure 5. The UltraSPARC T2 processor combines eight cores, memory management, cryptographic
support, 10 Gb Ethernet, and PCI Express on a single chip
The eight cores on the UltraSPARC T2 processor are interconnected with a full on-chip
non-blocking 8 x 9 crossbar switch. The crossbar connects each core to the eight banks
of L2 cache, and to the system interface unit for IO. The crossbar provides
approximately 300 GB/second of bandwidth and supports 8-byte writes from a core to a
bank and 16-byte reads from a bank to a core. The system interface unit connects
networking and I/O directly to memory through the individual cache banks. Using
FBDIMM memory supports dedicated northbound and southbound lanes to and from
the caches to accelerate performance and reduce latency. This approach provides
higher bandwidth than with DDR2 memory, with up to 42.4 GB/second of read
bandwidth and 21 GB/second of write bandwidth.
Each core provides its own fully-pipelined floating point and graphics unit (FGU), as well
as a stream processing unit (SPU). The FGUs greatly enhance floating point
performance over that of the UltraSPARC T1, while the SPUs provide wire-speed
cryptographic acceleration with over 10 popular ciphers supported, including DES,
Cross Bar
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$
C0 C1 C2 C3 C4 C5 C6 C7
FPU FPU FPU FPU FPU FPU FPU FPU
System Interface
Network
Interface Unit PCIe
10 Gigabit
Ethernet Ports (2)
x8 @ 2.0 GHz
FB DIMM FB DIMM FB DIMM FB DIMM
FB DIMM FB DIMM FB DIMM FB DIMM
SPU SPU SPU SPU SPU SPU SPU SPU
MCU MCU MCU MCU

15
Th
e
Ult
ra
SPARC
T
2
P
rocessor w
ith
C
oo
lTh
rea
d
s
T
ec
h
no
l
ogy
Sun Microsystems, Inc.
3DES, AES, RC4, SHA-1, SHA-256, MD5, RSA to 2048 key, ECC, and CRC32. Embedding
hardware cryptographic acceleration for these ciphers allows end-to-end encryption
with no penalty in either performance or cost.
UltraSPARC T2 Core Architecture and Pipelines
Figure 6 provides a block-level diagram representing a single UltraSPARC cores on the
UltraSPARC T2 processor (up to eight are supported per processor).
Figure 6. UltraSPARC T2 core block diagram
Components implemented in each core include:
• Trap Logic Unit (TLU) — The trap logic unit updates the machine state as well as
handling exceptions and interrupts.
• Instruction Fetch Unit (IFU) — The instruction fetch unit includes the 16KB
instruction cache (32-byte lines, 8-way set associative) and a 64-entry fully-associative
instruction translation lookup buffer (ITLB).
• Integer Execution Units (EXU) — Dual integer execution units are provided per core
with four threads sharing each unit. Eight register windows are provided per thread,
with 160 integer register file (IRF) entries per thread.
• Floating Point/Graphics Unit (FGU) — A floating point/graphics unit is provided
within each core and it is shared by all eight threads assigned to the core. 32 floating-
point register file entries are provided per thread.
• Stream Processing Unit (SPU) — Each core contains a stream processing unit that
provides cryptographic coprocessing.
•
Memory Management Unit (MMU) — The memory management unit provides a
hardware table walk (HWTW) and supports 8 KB, 64 KB, 4 MB, and 256 MB pages.
IFU
EXU0 EXU1
SPU FGU LSU
MMU/
HWTW
Gasket
TLU
Crossbar / L2 Cache
UltraSPARC T2 core

16
Th
e
Ult
ra
SPARC
T
2
P
rocessor w
ith
C
oo
lTh
rea
d
s
T
ec
h
no
l
ogy Sun Microsystems, Inc.
An eight-stage integer pipeline and a 12-stage floating-point pipeline are provided by
each UltraSPARC processor core (Figure 7). A new “pick” pipeline stage has been added
to choose two threads (out of the eight possible per core) to execute each cycle.
Figure 7. UltraSPARC T2 per-core integer and floating-point pipelines
To illustrate how the dual pipelines function, Figure 8 depicts the integer pipeline with
the load store unit (LSU). The instruction cache is shared by all eight threads within the
core. A least-recently-fetched algorithm is used to select the next thread to fetch. Each
thread is written into a thread-specific instruction buffer (IB) and each of the eight
threads is statically assigned to one of two thread groups within the core.
Figure 8. Threads are interleaved between pipeline stages with very few restrictions (integer pipeline
shown, letters depict pipeline stages, numbers depict different scheduled threads)
The “pick” stage chooses one thread each cycle within each thread group. Picking
within each thread group is independent of the other, and a least-recently-picked
algorithm is used to select the next thread to execute. The decode state resolves
resource conflicts that are not handled during the pick stage. As shown in the
illustration, threads are interleaved between pipeline stages with very few restrictions.
Any thread can be at the fetch or cache stage, before being split into either of the two
thread groups. Load/store and floating point units are shared between all eight
threads. Only one thread from either thread group can be scheduled on such a shared
unit.
Eight-Stage Integer Pipeline
Twelve-Stage Floating-Point Pipeline
Fetch Cache Pick Decode Execute Fx1 Fx2 Fx3 Fx4 Fx5 Fx6 FW
Fetch Cache Pick Decode Execute Mem Bypass W
LSU
Thread Group 0
IFU
IB0-3
IB0-3
IB0-3
IB0-3
P0
W2
B1
M3
E0
D2
IB0-3
IB0-3
IB0-3
IB4-7
P5
W6
B7
M4
E6
D7
Thread Group 1
W6
B1
M4
C6
F2

17
Th
e
Ult
ra
SPARC
T
2
P
rocessor w
ith
C
oo
lTh
rea
d
s
T
ec
h
no
l
ogy Sun Microsystems, Inc.
Integrated Networking
By providing integrated on-chip networking, the UltraSPARC T2 processor is able to
provide better networking performance. All network data is supplied directly from and
to main memory. Placing networking so close to memory reduces latency, provides
higher memory bandwidth, and eliminates inherent inefficiencies of I/O protocol
translation.
The UltraSPARC T2 processor provides two 10 Gb Ethernet ports with integrated serdes,
offering line-rate packet classification at up to 30 million packets/second (based on
layers 1-4 of the protocol stack). Multiple DMA engines (16 transmit and 16 receive DMA
channels) match DMAs to individual threads, providing binding flexibility between ports
and threads. Virtualization support includes provisions for eight partitions, and
interrupts may be bound to different hardware threads.
Stream Processing Unit
The stream processing unit on each UltraSPARC T2 core runs in parallel with the core at
the same frequency. Two independent sub-units are provided along with a DMA engine
that shares the core’s crossbar port:
• A Modular Arithmetic Unit (MAU) shares the FGU multiplier, providing RSA
encryption/decryption, binary and integer polynomial functions, as well as elliptic
curve cryptography (ECC)1
• The cipher/hash unit provides support for popular RC4, DES/3DES, AES-128/192/256,
MD5, SHA-1, and SHA-256 ciphers
The SPU is designed to achieve wire-speed encryption and decryption on both of the
processor’s 10 GB Ethernet ports.
Integral PCI Express Support
The UltraSPARC T2 processor provides an on-chip PCI Express interface that operates at
4 GB/second bidirectionally through a point-to-point dual-simplex chip interconnect. An
integral IOMMU supports I/O virtualization and process device isolation by using the
PCI Express BDF number. The total I/O bandwidth is 3-4 GB/second, with maximum
payload sizes of 128 to 512 bytes. An x8 serdes interface is provided for integration with
off-chip PCI Express switches.
1.Supported in a future Solaris OS release

18
Th
e
Ult
ra
SPARC
T
2
P
rocessor w
ith
C
oo
lTh
rea
d
s
T
ec
h
no
l
ogy Sun Microsystems, Inc.
Power Management
Beyond the inherent efficiencies of CMT design, the UltraSPARC T2 is the first processor
to incorporate unique power management features at both the core and memory levels
of the processor. These features include reduced instruction rates, parking of idle
threads and cores, and ability to turn off clocks in both cores and memory to reduce
power consumption. Substantial innovation is present in the areas of:
• Limiting speculation such as conditional branches not taken
• Extensive clock gating in the data path, control blocks, and arrays
• Power throttling that allows extra stall cycles to be injected into the decode stage