Nvidia TESLA K40 User manual

BD-06902-001_v05 | November 2013
Board Specification
TESLA K40 GPU ACCELERATOR

Tesla K40 GPU Accelerator BD-06902-001_v05 | ii
DOCUMENT CHANGE HISTORY
BD-06902-001_v05
Version Date Authors Description of Change
01 July 3, 2013 GG, SM Preliminary Information (Information contained
within this board specification is subject to
change)
02 August 1, 2013 GG, SM •
Updated product name
•Updated core clocks speeds
•Updated block diagram in Figure 1
03 September 19, 2013 GG, SM •Added new section: “NVIDIA GPU Boost on Tesla
K40”
•Updated “Power Specifications” section
•General updates throughout the board
specification
•Added board environmental conditions
04 September 25, 2013 GG, SM Updated boost clock
05 November 11, 2013 GG, SM, DV
•Updated MTBF numbers
•Added Figure 1: K40 Passive Board

Tesla K40 GPU Accelerator BD-06902-001_v05 | iii
TABLE OF CONTENTS
Overview .........................................................................................................1
Key Features .................................................................................................. 2
NVIDIA GPU Boost on Tesla K40............................................................................ 3
NVIDIA GPU Boost for HPC Workloads ................................................................. 3
API for NVIDIA GPU Boost on Tesla ........................................................................ 4
Tesla K40 Block Diagram .................................................................................... 6
Environmental Conditions................................................................................... 6
Configuration.................................................................................................. 7
Mechanical Specifications ....................................................................................8
PCI Express System........................................................................................... 8
Tesla K40 Bracket ............................................................................................ 8
Power Connectors ...........................................................................................10
Power Specifications......................................................................................... 13
Support Information ......................................................................................... 15
Certificates and Agencies..................................................................................15
Agencies ...................................................................................................15
Languages.....................................................................................................16

Tesla K40 GPU Accelerator BD-06902-001_v05 | iv
LIST OF FIGURES
Figure 1. K40 Passive Board ................................................................................ 1
Figure 2. Tesla K40 Block Diagram ........................................................................ 6
Figure 3. Tesla K40 GPU Accelerator ..................................................................... 8
Figure 4. Tesla K40 Bracket ................................................................................ 9
Figure 5. 6-Pin PCI Express Power Connector ..........................................................10
Figure 6. 8-Pin PCI Express Power Connector ..........................................................11
LIST OF TABLES
Table 1. nvidia-smi Commands............................................................................ 5
Table 2. Board Environmental Conditions............................................................... 6
Table 3. Board Configuration.............................................................................. 7
Table 4. 6-Pin PCI Express Power Connector Pinout .................................................12
Table 5. 8-Pin PCI Express Power Connector Pinout .................................................12
Table 6. Auxiliary Power Connectors ...................................................................13
Table 7. Power Requirements............................................................................14
Table 8. Languages Supported ...........................................................................16

Tesla K40 GPU Accelerator BD-06902-001_v05 | 1
OVERVIEW
The NVIDIA®Tesla®K40 graphics processing unit (GPU) is a PCI Express, dual-slot
computing module in the Tesla (267 mm length) form factor comprised of a single
GK110B GPU. The Tesla K40 is designed for servers and offers a total of 12 GB of
GDDR5 on-board memory and supports PCI Express Gen3. The Tesla K40 uses a passive
heat sink for cooling.
Tesla K40 boards ship with ECC enabled by default protecting the register files, cache
and DRAM. With ECC enabled, some of the memory is used for the ECC bits, so the
user available memory is reduced by ~6.25%. On the Tesla K40 the total available
memory with ECC turned on will be ~11.25 GB.
Figure 1. K40 Passive Board

Overview
Tesla K40 GPU Accelerator BD-06902-001_v05 | 2
KEY FEATURES
GPU
Number of processor cores: 2880
Core clocks
●Base clock: 745 MHz
●Boost clocks: 810 MHz and 875 MHz
Package size: 45 mm × 45 mm 2397-pin ball grid array (S-FCBGA)

Note: All boards ship with core clock set to the base clock value. Boost clocks can
be selected using NVML or NVSMI. Refer to the NVML/NVSMI documentation for
more details.
Board
PCI Express Gen3 ×16 system interface
Physical dimensions: 111.15 mm (height) × 267 mm (length), dual-slot
Thermal Solution
Passive heat sink
Display Connectors
None
Power Connectors
One 6-pin PCI Express power connector
One 8-pin PCI Express power connector
Memory
Memory clock: 3.0 GHz
Memory bandwidth 288 GB/sec
Interface: 384-bit
●Total board memory: 12 GB
●24 pieces of 256M × 16 GDDR5, SDRAM
BIOS
2Mbit serial ROM
BAR1 size: 16 GB

Overview
Tesla K40 GPU Accelerator BD-06902-001_v05 | 3
NVIDIA GPU BOOST ON TESLA K40
NVIDIA GPU Boost™is a feature available on Tesla K40. It makes use of any power
headroom to run the core clock to a higher frequency. Application workloads that have
power headroom can run at high GPU clocks to boost application performance.

Note: The memory clock remains constant at 3 GHz. It's likely that the effective
memory bandwidth utilization will change depending on the core clock frequency.
NVIDIA GPU Boost for HPC Workloads
NVIDIA GPU Boost for Tesla K40 is optimized to deliver a robust and deterministic
boost behavior for a wide range of HPC workloads.
Tesla K40 gives full control to end-users to select the core clock frequency that fits their
workload the best. The workload may have one or more of the following characteristics.
Problem set is spread across multiple GPUs and requires periodic synchronization.
Problem set spread across multiple GPUs and runs independent of each other.
Workload has “compute spikes.” For example, some portions of the workload are
extremely compute intensive pushing the power higher and some portions are
moderate.
Workload is compute intensive through-out without any spikes.
Workload requires fixed clocks and is sensitive to clocks fluctuating during the
execution.
Workload runs in a cluster where all GPUs need to start, finish, and run at the same
clocks.
Workload or end user requires predictable performance and repeatable results.
Datacenter is used to run different types of workload at different hours in a day to
better manage the power consumption.
Some boards in a cluster have access to better cooling than others.
By default the Tesla K40 ships with the core clock set to the base clock. HPC workloads
can have one or more characteristics as described. When selecting one of the supported
boost clocks a good strategy is to characterize the workload with the available boost
clocks. For example, DGEMM/Linpack are extremely demanding on power. Therefore,
the “base clock” may be the correct choice when running Linpack. Some workloads in
life sciences, manufacturing, CFD, CAD, etc., may have power headroom and can take
advantage of one of the boost clocks.

Overview
Tesla K40 GPU Accelerator BD-06902-001_v05 | 4
API FOR NVIDIA GPU BOOST ON TESLA
Tesla K40 gives full control to end-users to select the core clock frequency via NVML or
nvidia-smi. NVML is a C-based API for monitoring and managing the various states of
Tesla products. It provides a direct access to submit queries and commands via nvidia-
smi. NVML documentation is available at https://developer.nvidia.com/nvidia-
management-library-nvml

Overview
Tesla K40 GPU Accelerator BD-06902-001_v05 | 5
Table 1 gives a summary of the nvidia-smi commands for using NVIDIA GPU Boost
on Tesla.
Table 1. nvidia-smi Commands
Usage Command
View the clocks the Tesla board supports nvidia-smi –q –d SUPPORTED_CLOCKS
Set one of the supported clocks nvidia-smi -ac <MEM clock, Graphics clock>
Make the clock settings persistent across
driver unload nvidia-smi -pm 1
Make the clock settings revert to base clocks
after driver unloads (or turn off the
persistent mode)
nvidia-smi -pm 0
To view the clock in use, use the command nvidia-smi -q –d CLOCK
To reset clocks back to the base clock (as
specified in the board specification) nvidia-smi –rac
To allow “non-root” access to change
graphics clock nvidia-smi -acp 0
When using non-default applications clocks, driver persistence mode should be enabled.
Persistence mode ensures that the driver stays loaded even when no NVIDIA®CUDA®
or X applications are running on the GPU. This maintains current state, including
requested applications clocks. If persistence mode is not enabled, and no applications
are using the GPU, the driver will unload and any current user settings will revert back
to default for the next application. To enable persistence mode run 'sudo nvidia-smi -
pm 1'.
The driver will attempt to maintain requested applications clocks whenever a CUDA
context is running on the GPU. However, if no contexts are running the GPU will revert
back to idle clocks to save power and will stay there until the next context is created.
Thus, if the GPU is not busy, you may see idle current clocks even though requested
applications clocks are much higher.

Note: By default changing the application clocks requires root access. If the user
does not have root access, the user can request his or her cluster manager to allow
non-root control over application clocks. Once changed, this setting will persist for
the life of the driver before reverting back to root-only defaults. Persistence mode
should always be enabled whenever changing application clocks, or enabling non-
root permissions to do so.

Overview
Tesla K40 GPU Accelerator BD-06902-001_v05 | 6
TESLA K40 BLOCK DIAGRAM
Figure 2 is the block diagram for the Tesla K40 GPU dual-slot computing processor
module.
Figure 2. Tesla K40 Block Diagram
ENVIRONMENTAL CONDITIONS
Table 2 lists the environmental operating and storage conditions for the Tesla K40 board.
Table 2. Board Environmental Conditions
Specifications Conditions
Operating temperature 0 °C to 45 °C
Storage temperature -40 °C to 75 °C
Operating humidity 5% to 90% RH
Storage humidity 5% to 95% RH