IBM eserver i5 User manual

iSeries. mySeries.
iSeries. mySeries.
Common Europe Luxembourg
Introducing IBM eServer i5 & i5/OS
LPAR i5 “Up-to-Date” :
Logical Partitions on the IBM Power5
by Eddy PASTEGER
1
2

iSeries. mySeries.
iSeries. mySeries.
Logical Partitions on the IBM Power5
Introduction
3
iSeries. mySeries.
Introduction
Objectives
Understand the LPAR functionality as implemented on the IBM Power5 servers
Understand planning considerations and system requirements
Warning
This presentation is built on functionalities available on i5 models 5xx running i5/OS V5R3
4

iSeries. mySeries.
Logical Partitioning on the IBM Power5
Agenda
Chapter 1. Architecture
Chapter 2. Resources Management Concepts
Chapter 3. LPAR-Capable i5 Models
Chapter 4. Planning
Chapter 5. Hardware Management Console
Chapter 6. Virtual Partition Manager
5
iSeries. mySeries.
iSeries. mySeries.
Logical Partitions on the IBM Power5
Architecture
6

iSeries. mySeries.
Architecture
HMC
Service
Processor
Processor
Complex
Main
Storage
I/O
Subsystem
NVRAM
Permanent Temporary
The Service Processor
Standard built-in, independent component of every eServer i5 or p5
Own processor, memory and boot code
NVRAM stores the “firmware” and the LPAR configuration
7
iSeries. mySeries.
Architecture
The Service Processor (aka SP)
Once main power is connected ...
The SP activates its own “firmware” operating system
Control panel shows C1xx progress codes
Gathers vital product data (VPD) from the electronic components in the server
Runs diagnostic routines to verify its hardware and firmware
Control panel shows 01 and the green power LED is flashing
The server is in “SP Standby” mode
The server is ready to interface with HMC
HMC
Service
Processor
Processor
Complex
Main
Storage
I/O
Subsystem
NVRAM
Permanent Temporary
8

iSeries. mySeries.
Architecture
HMC
Service
Processor
Processor
Complex
Main
Storage
I/O
Subsystem
Power Hypervisor
P1 P2 P3 P4
The Service Processor (aka SP)
Once you power on the system ...
The SP turns on the system power
All I/O towers via SPCN
Control panel shows C7xx progress codes
Runs diagnostic routines to verify system processors and system main storage
Verifies that it can communicate with environmental sensors and monitors in the system
Control panel shows 01 and the green power LED is on
The server is in “Hypervisor Standby” mode
9
iSeries. mySeries.
Architecture
The Power Hypervisor
Previously part of the System Licensed Internal Code (SLIC)
Known as “Partition Licensed Internal Code (PLIC)”
PLIC is enhanced to be used with the eServer i5 hardware
Power Hypervisor is independent from any operating system
There is no longer the concept of a “Primary Partition”
There is no longer the need to IPL the whole system to make partitions changes
There is no longer a single point of failure with a “Primary Partition”
Also known as ...
Firmware
Server Firmware
Hypervisor
Power Hypervisor
pHyp
Underware
10

iSeries. mySeries.
Architecture
The Power Hypervisor Functions
Virtual processor support
Allocation of physical processor time to partition thru “virtual processors”
Virtual memory management
Allocation of physical memory to partition thru “virtual memory”
Virtual hardware support
Processors
OptiConnect
SCSI
Ethernet
Serial
Security an isolation between partition
Partition allowed to access only to their allocated resources
Enforced by the Power Hypervisor
Reinitializes processors, resets registers, flushes caches
Clears memory
Reset I/O devices
11
iSeries. mySeries.
Architecture
TIMI
HMC
Hardware
Power Hypervisor
P1 P2 P3 P4
SLIC
i5/OS
Starting an i5/OS partition
Hypervisor will operates a “virtual service processor” to load and start SLIC
SLIC is changed to interface with the new Power Hypervisor
TIMI and all the layers above are still in place
12

iSeries. mySeries.
Architecture
TIMI
HMC
Hardware
Power Hypervisor
P1 P2 P3 P4
SLIC
i5/OS
Open
Firmware
RTAS
Linux
Open
Firmware
RTAS
AIX
Starting a LINUX or AIX partition
Hypervisor will operates ...
Open Firmware contains boot-time drivers and the boot manager
RTAS is a service that passes platform-dependent system calls to the Hypervisor
Other is characteristic to the guest operating system
E.g. Linux has a “monolithic” kernel, while AIX has a “microkernel-hybrid” structure
13
iSeries. mySeries.
Architecture
HMC
Service
Processor
NVRAM
Permanent Temporary
Power Hypervisor
Service Partition
TIMI
SLIC
i5/OS
A B
Firmware
Firmware
MHxxxxx
Hypervisor Code Maintenance
14

iSeries. mySeries.
iSeries. mySeries.
Logical Partitions on the IBM Power5
Resources Management Concepts
15
iSeries. mySeries.
Resources Management Concepts
What is “logical partitioning” ?
The ability to make a single server run as if it were many independent systems
Each logical system is called a “Partition”
Each partition operates as an independent system
Each partition will receive a portion of the system resources :
Processor
Memory
I/O slots
Virtual I/O slots
16

iSeries. mySeries.
iSeries. mySeries.
Logical Partitions on the IBM Power5
Resources Management Concepts
Processor
17
iSeries. mySeries.
Resources Management Concepts
Processor Concepts
Dedicated vs. Shared Processors
Virtual Processors
Capped vs. Uncapped Processors
Considerations about LPAR and SMT
18

iSeries. mySeries.
Processor Concepts
Physical Processors
SharedDedicated
P1 P2 P3
Inactive
Processor Concepts
19
iSeries. mySeries.
Processor Concepts
Dedicated Processors
A “dedicated processor” refers to a whole processor that is dedicated to a single partition
One or more processors can be dedicated to a partition
The most simple way to distribute processors among logical partitions
Rules & requirements :
At least one dedicated processor
Granularity of movements : one processor
Processor moves are dynamic, without any IPL within the configured ranges
Unallocated processors are not used
20

iSeries. mySeries.
Processor Concepts
Shared Processors
A “shared processor” allows to assign partial processors to a partition
Physical processors lives in a “shared processor pool”
By default, any unassigned processor is a member of the shared processor pool
21
iSeries. mySeries.
Processor Concepts
20 milliseconds of
processor time
10 milliseconds
P1 P2 P3
Shared Processors
22

iSeries. mySeries.
Processor Concepts
20 milliseconds of
processor time
10 milliseconds
P1 P2 P3
Shared Processors
23
iSeries. mySeries.
Processor Concepts
Shared Processors
Rules & requirements :
A processing unit (PU) represents the cycles of a single processor
At least 0.10 PU (1 millisecond)
Granularity of movements : 0.01 PU
Processor moves are dynamic, without any IPL within the configured ranges
Unallocated PU are not used
24

iSeries. mySeries.
Processor Concepts
Potential Shared Processors Penalty
Increase the possibility that job’s won’t complete, and :
Have to be re-dispatched, and potentially have to reload cache
Increase the chance of a cache miss
Reduce the chance for processor/memory affinity
Increase the Power Hypervisor work :
Tracking each partition use of its allocated milliseconds
25
iSeries. mySeries.
Processor Concepts
Virtual Processors
The number of “virtual processors” represents :
The number of physical processors that the partition is able to use
The number of simultaneous jobs/threads execution
The number of processors that the operating system thinks he has to use
26

iSeries. mySeries.
Processor Concepts
20 milliseconds of
processor time
10 milliseconds
P1 P2 P3
Virtual Processors
27
iSeries. mySeries.
Processor Concepts
20 milliseconds of
processor time
10 milliseconds
P1 P2 P3
Virtual Processors
28

iSeries. mySeries.
Processor Concepts
Virtual Processors
Rules & requirements :
It is possible to allocate more virtual processors than there are in the shared processor pool
The actual number of processor in the shared processor pool is a “floating” number
A single physical processor can report in the system as a n-ways
You cannot allocate less than 0.10 PU per virtual processor
Example : requesting 0.20 PU to run on 4 VCPU means 0.20 / 4 = 0.05 < 0.10 !
Changes are dynamic, without any IPL within the configured ranges
29
iSeries. mySeries.
Processor Concepts
Uncapped Processors
Uncapped allows to potentially utilizes unused processor capacity in the system
Uncapped weight is used to determine how unused capacity is distributed :
P1 weight is 200
P2 weight is 100
P1 will receive 2/3 of the unused shared processor capacity
P2 will receive 1/3 of the unused shared processor capacity
If P1 does not need all 2/3 of capacity, then P2 could get more !
30

iSeries. mySeries.
Processor Concepts
Dedicated, Shared, Capped or Uncapped ?
How to decide on using :
Dedicated or shared processors ?
If using shared processors, it is to be capped or uncapped ?
And ... how many virtual processors do I allocate ?
The best performance may be achieved by using dedicated processors ...
However, dedicated processors cannot utilizes excess capacity !
Shared uncapped will allow use of excess capacity of the processor, but ...
Setting virtual processor number too low will limit uncapped usage
Setting virtual processor number too high will degrade performance
What does IBM recommends ?
It depends ...
What do I recommend ?
Keep things simple !
31
iSeries. mySeries.
Processor Concepts
Simple Recommendations
On small configurations, use shared uncapped processor :
Allocate desired processing units according needs
Keep a little portion of unused processing units
Configure uncapped weight according partition importance
Set virtual processor number to a “reasonable” value :
Set a value “close” to the number of allocated processing units
0.4 PU running on 1 VCPU
1.8 PU running on 2 VCPU
Take care of your workload pattern
1.0 PU running on 1 or 2 VCPU ?
Single job, single thread applications will not take benefits when running on multiple VCPU
Multiple job, multiple thread applications will run more efficiently when running on multiple VCPU
Take care of SMT capabilities !
Do not exceed the number of physical processors
32

iSeries. mySeries.
Processor Concepts
About Simultaneous Multi-Threading (SMT)
SMT is a technology that allows a single CPU to be seen logically as 2 logical CPU
Physical processor
Logical processors
SMT allows 2 threads to run simultaneously on the same processor
Using different execution units, or “pipes”
SMT optimizes the usage of the 7 execution units of a PowerPC processor
Hypervisor only deals with physical processors
Each operating system can use the physical processor as its needs
SMT can be used or not ... according partition’s operating system configuration
For i5/OS, refer to system value QPRCMLTTSK
33
iSeries. mySeries.
iSeries. mySeries.
Logical Partitions on the IBM Power5
Resources Management Concepts
Main Storage
34

iSeries. mySeries.
Main Storage Concepts
Memory Allocation Facts
Power4 Hypervisor allowed memory manipulation at the single megabyte (1MB) level
Memory in the Power5 machines cannot be manipulated that finely :
Memory must be assigned or moved in blocks
Theses blocks are called “Logical Memory Blocks (LMB)” or “Memory Regions”
About those “Memory Regions” :
LMB are sized in multiples of 16 MB
Future implementations will allow use of 16, 32, 64, 128 or 256 MB blocks
LMB size is a system-wide parameter
LMB is the smallest amount of memory that can be manipulated
35
iSeries. mySeries.
Main Storage Concepts
Memory Allocation Facts
Hypervisor uses memory to run the system !
Hypervisor memory used expands and contracts based on many variables :
Number of partitions
Amount of I/O attached to the system
Amount of virtual I/O slots used in each partition
HSL OptiConnect and Virtual OptiConnect links
Starting and/or stopping a partition affects hypervisor memory requirements
Hypervisor memory allocation made by 16 MB increments
Main Storage
P1 P2 P3
Hypervisor HPT1 HPT 2 HPT 3
36

iSeries. mySeries.
Main Storage Concepts
Memory Allocation Facts
A “Hardware Page Table (HPT)” is needed to manage memory
Power4 Hypervisor reserved HPT within partition memory
Power5 Hypervisor allocates HPT separately from partition memory
Size of the HPT depends of the maximum memory amount the partition can receive
Basic calculation : maximum memory divided by 64, rounded up to next power of 2
A partition can allocates a maximum of 12 GB (12288 MB)
HPT size will be 12288 / 64 = 192 ... next power of 2 is ... 256
HPT size will be 256 MB
Recommendation
Take care of HPT when allocating memory to partitions !
Plan to allocate 12288 - 256 = 12032 MB for the partition
Memory actually used will be 12288 MB
Main Storage size MUST be carefully planned !
37
iSeries. mySeries.
Main Storage Concepts
Memory Allocation Facts
Remember
A node is a packaged set of processors, caches (L1+L2+L3) and memory on a module
Remember about ... “home node”, “local node” and “remote node” ?
Remember about ... “local memory access” and “remote memory access”
Remember about ... “memory affinity” ?
!"#$
@ABC @ABD
@ABC @ABD @ABC @ABD
@ABC @ABD
!,64’>1$%,9"
+D
+E
+D
+E
+D
+E
+D
+E
!"#$
@ABC @ABD
@ABC @ABD @ABC @ABD
@ABC @ABD
!,64’>1$%,9"
+D
+E
+D
+E
+D
+E
+D
+E
38

iSeries. mySeries.
Main Storage Concepts
Memory Allocation Facts
Power4 Hypervisor allocated memory equally from all nodes
This leads to more “remote” memory access
Power5 hypervisor tries to do better ...
Processors are assigned on the fewest nodes possible
Configured memory will be assigned keeping as much memory as possible on the same node
The HPT for the partition will be assigned on the same node
How becomes memory affinity with dynamic processor/memory moves ?
The system will “do its best” ...
But there is no guarantee that the best configuration can be maintained
A full system restart (hypervisor) may be needed to return system in its optimal configuration !
39
iSeries. mySeries.
iSeries. mySeries.
Logical Partitions on the IBM Power5
Resources Management Concepts
I/O Subsystem
40