IBM Multiprise 3000 User manual

S/390
Multiprise 3000 Enterprise Server
R
IBM
Internal Disk Subsystem:
Reference Guide
SA22-1025-00

S/390IBM
Internal Disk Subsystem:
Reference Guide
SA22-1025-00

Note!
Before using this information and the product it supports, be sure to read the general information under Appendix D, “Notices”
on page D-1.
First Edition (November, 1999)
This edition, SA22-1025-00, applies to the IBM Internal Disk Subsystem.
Copyright International Business Machines Corporation 1999. All rights reserved.
Note to U.S. Government Users — Documentation related to restricted rights — Use, duplication or disclosure is subject to
restrictions set forth in GSA ADP Schedule Contract with IBM Corp.

Contents
Chapter 1. Overview of the S/390 Internal Disk Subsystem ......... 1-1
Host Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
RAID Disk Arrays .................................... 1-3
HDD Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
Internal Disk RAID Fast Write ............................ 1-4
Internal Disk Facilities ................................. 1-4
CKD Emulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
Control Unit and Device Emulation ........................ 1-5
LPAR Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
Enhanced HDD Reliability .............................. 1-5
Search Assist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-6
SSID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7
Long Busy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7
Internal Disk Components ............................... 1-7
Shared CPC resources ............................... 1-7
SSA Adapter Card ................................. 1-8
HDD Packaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-8
Internal Disk Logical Volume Configurations .................... 1-8
Logical Volumes Mapped onto RAID Arrays ................... 1-8
Logical Volume Address Resolution ........................ 1-9
Internal Disk Configuration Granularity ..................... 1-10
Commands and Facilities Not Supported ..................... 1-11
IOCP Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-11
RAS Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-12
Chapter 2. Processing Commands and Presenting Status .......... 2-1
Input/Output Channel Interface ............................ 2-1
Command Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Status Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Status Byte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Initial Status Byte ................................... 2-4
Status Pending Conditions ............................. 2-4
State-Change Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5
Effect of Subsystem and Device State on Status Presented to Command .. 2-5
Listing of Commands Appearing in Table 3 ................... 2-9
Contingent Allegiance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-9
Channel Command Retry ............................... 2-9
Channel Command Retry–Control Unit Initiated ................ 2-10
System and Selective Reset ............................. 2-10
Chapter 3. Elements of Data Access ....................... 3-1
Track Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Track Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Data Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Track Accessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
Track Orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
Record Accessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3
Data Transfer Commands ............................... 3-3
Formatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
File Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Copyright IBM Corp. 1999 iii

End-of-File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Chapter 4. Command Descriptions . . . . . . . . . . . . . . . . . . . . . . . . 4-1
Channel Command Word ............................... 4-1
Exception Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Command Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Multitrack Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Defective/Alternate Track Operations ......................... 4-3
Summary of Channel Commands ........................... 4-3
List of Channel Commands by Hex Code .................... 4-6
Addressing and Control Commands ......................... 4-7
Define Extent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8
Locate Record . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-13
Locate Record Extended ............................. 4-27
Prefix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-42
Seek Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-46
Seek . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-47
Seek Cylinder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-48
Seek Head . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-49
Recalibrate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-50
Set File Mask .................................... 4-51
Set Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-52
Read Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-53
Read Device Characteristics ........................... 4-54
Read Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-61
Read Home Address ............................... 4-62
Read Record Zero ................................. 4-63
Read Count, Key, and Data ........................... 4-64
Read Key and Data ................................ 4-66
Read Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-68
Read Count . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-70
Read Multiple Count, Key, and Data ...................... 4-71
Read Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-72
Read Track Data .................................. 4-74
Read IPL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-76
Search Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-77
Search Home Address Equal ........................... 4-78
Search ID Equal .................................. 4-79
Search ID High ................................... 4-80
Search ID Equal or High ............................. 4-80
Search Key Equal ................................. 4-81
Search Key High .................................. 4-82
Search Key Equal or High ............................ 4-82
Write Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-82
Write Home Address ................................ 4-84
Write Record Zero ................................. 4-85
Write Count, Key, and Data ............................ 4-86
Write Count, Key, and Data Next Track ..................... 4-88
Erase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-90
Write Update Key and Data ............................ 4-91
Write Key and Data ................................ 4-93
Write Update Data ................................. 4-95
Write Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-97
Write Full Track ................................... 4-99
iv Internal Disk Subsystem Reference Guide

Write Track Data ..................................4-102
Sense Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-104
Sense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-104
Sense ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-105
Miscellaneous Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-107
No-Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-107
Read and Reset Buffered Log ..........................4-108
Read Configuration Data .............................4-111
Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-117
Path Control Commands ...............................4-118
Device Reserve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-118
Device Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-120
Unconditional Reserve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-121
Reset Allegiance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-123
Set Path Group ID .................................4-125
Sense Path Group ID ...............................4-128
Suspend Multipath Reconnection ........................4-130
Subsystem Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-130
Perform Subsystem Function ...........................4-130
Sense Subsystem Status .............................4-136
Read Subsystem Data ...............................4-139
Chapter 5. 24-Byte Compatibility Sense Data .................. 5-1
24-Byte Sense Data Summary ............................ 5-2
Sense Data Formats ................................. 5-2
24-Byte Sense Data Description ........................... 5-4
Sense Bytes 0, 1, and 2 .............................. 5-4
Sense Byte 3–Count ................................. 5-6
Sense Byte 4–Logical Volume Address ...................... 5-6
Sense Byte 5–Low Cylinder Address ....................... 5-7
Sense Byte 6–High Cylinder Address and Track Address ........... 5-7
Sense Byte 7–Format/Message . . . . . . . . . . . . . . . . . . . . . . . . . . 5-7
Sense Bytes 8 through 23–Format-Dependent Information .......... 5-7
Sense Byte 25–Program Action Code ....................... 5-7
Sense Bytes 26 and 27–Configuration Data ................... 5-8
Sense Bytes 29 through 31–Cylinder and Head Address ............ 5-8
24-Byte Compatibility Sense Data Formats .................... 5-10
Format 0–Program or System Checks ..................... 5-10
Format 1–Device Equipment Checks ...................... 5-11
Sense Bytes 8 through 23 for Format 1 ..................... 5-12
Format 2–Internal Disk Control Unit Errors ................... 5-13
Format 3–Allegiance Terminated ......................... 5-13
Format 4–Unrecovered Data Checks ...................... 5-14
Format 5 to Format E –Not used ........................ 5-15
Format F–Subsystem Checks .......................... 5-15
Chapter 6. ECKD 32-Byte Sense Data ...................... 6-1
ECKD 32-Byte Sense Data Summary ........................ 6-2
ECKD 32-Byte Sense Data Description ....................... 6-4
Sense Byte 0 ..................................... 6-4
Sense Byte 1 ..................................... 6-4
Sense Byte 2–Control Unit Type .......................... 6-5
Sense Byte 3–Remaining Intent Count ...................... 6-5
Sense Byte 4–Logical Volume Address ...................... 6-5
Contents v

Sense Byte 5–Device Type Code ......................... 6-6
Sense Byte 6–Content and Format ........................ 6-6
Sense Bytes 7 through 19 ............................. 6-6
Sense Bytes 20 and 21–Subsystem Identifier .................. 6-6
Sense Bytes 22 and 23–Exception Code ..................... 6-6
Sense Byte 24–Logging and Message Control .................. 6-7
Sense Byte 25–Program Action Code ....................... 6-8
Sense Bytes 26 and 27–Configuration Data ................... 6-8
Sense Byte 28–Message Code .......................... 6-8
Sense Bytes 29 and 30–Cylinder Address .................... 6-9
Sense Byte 31–Head Address ........................... 6-9
Sense Data Dependent on Exception Class .................. 6-10
Chapter 7. Error Recovery Procedures ...................... 7-1
ERPs for 24-Byte Compatibility Sense ........................ 7-1
Error Recovery Actions ............................... 7-2
ERPs for ECKD 32-Byte Sense ............................ 7-5
ERP Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-6
ERP Message Requirements ............................ 7-9
Appendix A. Device Characteristics . . . . . . . . . . . . . . . . . . . . . . . A-1
Appendix B. Caching Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . B-1
Caching Algorithm Definitions ............................. B-1
Algorithms for Cache MISSES ............................. B-1
Algorithms for Cache HITS ...............................B-2
Algorithm Modifications due to Cache Hints ..................... B-2
Algorithm Special Cases ................................ B-3
Appendix C. Warranties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1
Statement of Limited Warranty ............................C-1
IBM Agreement for Licensed Internal Code .....................C-4
Appendix D. Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-1
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-2
Electronic Emission Notices ..............................D-2
List of Abbreviations ................................. X-1
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . X-3
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . X-7
vi Internal Disk Subsystem Reference Guide

Chapter 1. Overview of the S/390 Internal Disk Subsystem
This chapter gives an overview of the S/390 Internal Disk Subsystem, describes its
components, features, and facilities.
Internal Disk storage subsystem provides internal attachment of disk storage
integrated into the Central Processing Complex (CPC) of a S/390 CMOS processor.
Internal Disk consists of attachment hardware and microcode plus high capacity,
high performance Hard Disk Drives (HDDs) that are packaged under the same
CPC covers. The HDDs are organized into RAID arrays that insure data availability
while minimizing the cost of disk storage. The HDDs appear to the host programs
executing in a CPC as conventional Count, Key, Data (CKD) disks attached to a
CKD control unit. The programs use channel commands to manage the operations
of the disk subsystem.
The control unit and channel functions are executed within the CPC utilizing
processor and memory resources of the CPC:
A System Assist Processor (SAP) is used for the channel emulation and control
unit functions.
Disk cache space is allocated from S/390 main memory
The system Service Element (SE) is shared for configuration and service
functions.
Power, packaging, and cooling are all provided by the Multiprise3000 (host
processor).
From an implementation perspective these functions are an extension to the I/O
subsystem within the CPC.
Unique to the Internal Disk subsystem are Hard Disk Drives (HDDs) and logic cards
that carry adapter electronics. The logic cards are called SSA adapter cards. The
Hard Disk Drives (HDDs) used in Internal Disk are 3.5 inch, fixed block disk drives
with a data capacity of 18 GB. The SSA adapter card interfaces to the Multiprise
3000 via a PCI bus and provides an industry standard SSA (Serial Storage
Architecture) interface for connection of the HDDs.
The HDDs are mounted in the frame used for the packaging of the CPC. The
18GB UltraStar 18ZX drives are mounted in the HDD enclosure. The user may
choose to expand the storage beyond the capacity of the CPC frame by adding
expansion frames. The maximum configuration has one CPC frame and two
expansion frames.
The Multiprise 3000 has a minimum Internal Disk capacity of 72GBs and a
maximum capacity (inclusive of two expansion frames) of 792GBs.
The Internal Disk Subsystem offers the following advantages:
Minimization of software changes: If an operating system supports the device
being emulated, the Internal Disk subsystem can generally be installed without
any system software changes. Any combination of 3390-1, -2, -3, -9, and
3380-J, -E, -K logical volumes can be created on a RAID array. The total
number of logical volumes that can be created is only limited by the storage
capacity available on the array and the number of unit addresses that can be
assigned to an array. The latter one is 64 for an array with 5 HDDs (4+P), and
Copyright IBM Corp. 1999 1-1

96 for an array consisting of 7 HDDs (6+P). Also, logical volumes can be
added incrementally. This can even be done while I/O operations are ongoing
to other logical volumes which already exist on this array. In the same way,
logical volumes can also be selectively deleted again, freeing up space which
can then be used to create other logical volumes again.
Preloaded software is available
Internal Disk uses a RAID-5 organization for managing a collection of disks so
that high levels of data reliability and availability are maintained.
Usable disk storage capacity:
–72 to 216 GB in the CPC cage
–Up to 288 GB in each of the expansion cages
Enhanced reliability by Predictive Failure Analysis: A set of processes
continuously performed by the HDD is designed to predict disk drive failure
conditions before they occur.
End to end data protection
Low-cost disk subsystem: A high capacity Internal Disk subsystem with a very
low cost of ownership.
SSA Interface: The connection via the SSA adapter interface provides high
data transfer rates.
Concurrent Repair and Verify of HDDs.
Concurrent LIC patch for SAP microcode.
Control unit and data sharing among logical partitions.
Internal Disk RAID Fast Write (IDRFW) provides a fast write capability through
the SSA adapter card.
Host Processor
Internal Disk is packaged with the Multiprise 3000 which is a unique card on board
package that takes advantage of:
Integrated RAID Storage
Integrated Tape
Integrated Communications
Industry Standard PCI Adapters
S/390 ESCONChannels
S/390 Parallel Channels
The Multiprise 3000 presents a low cost server platform that maintains most of the
S/390 traditional attributes. It is a small, office environment server that provides
state of the art LAN/WAN connection capabilities at an attractive price. The base
package satisfies new customer/workloads that do not have the traditional heavy
I/O channel demands. The basic frame that houses the CPC also contains up to 24
ESCON channels. If more channels are needed, an expansion frame with 32
additional ESCON channels is available.
The Multiprise 3000 is the entry level processor for the S/390 product family
designed for installation in any standard office environment without special
provisions. The unit is powered from single phase 100 to 240 volts ac, at either 50
or 60 hertz. Dual line cords are provided, each connected to one of the two N+1
1-2 Internal Disk Subsystem Reference Guide

power supplies. Two cooling fans are also provided to assure that the power and
cooling system is fully redundant with concurrent repair. An Integrated Battery
Feature (IBF) is available.
Internal Disk in the Multiprise 3000 continues the evolution of disk capacity with the
introduction of RAID disk arrays and a new high performance 10,000 RPM Hard
Disk Drive, the 18GB UltraStar 18ZX.
RAID Disk Arrays
The Multiprise 3000 Internal Disk uses a RAID-5 organization for its disk arrays.
RAID is an acronym for Redundant Arrays of Independent Disks. The RAID
organization is a technique for managing a collection of disks so that high levels of
data reliability and availability are maintained while at the same time retaining
desirable cost and performance.
The original Internal Disk that was introduced on Multiprise systems implemented
mirroring to assure data reliability and availability. Mirroring is a recognized RAID
organization (RAID-1). However, RAID-1 requires a mirrored capacity that is equal
in size to the capacity of the primary storage.
The Multiprise 3000 Internal Disk supports RAID-5 array sizes of 4+P, and 6+P.
The 4+P array may be thought of as having 4 HDDs assigned to carry user data
and a P drive for Parity or redundancy. In reality, the redundancy is spread among
the 5 HDDs equally so there is no specific P drive. Distribution of the redundancy
among the drives avoids a performance bottleneck that would occur if one drive
were assigned all of the redundancy. The 4+P arrangement has 80% of its HDD
space available for user data. Similarly, the 6+P array may be thought of as having
6 HDDs for user data and a single Parity Drive. Almost 86% of the 6+P array can
contain user data. A RAID-1 organization uses 50% of the disk space for user data
and the other 50% for redundancy.
In the event of failure or loss of any HDD, all of its content can be reconstructed
from the content of the other HDDs in the array. Until the reconstruction is
complete, there is an exposure to loss of data since the redundancy is adequate for
only a single HDD loss. To assure that reconstruction begins without delay, spare
HDDs are included in the configuration. The spare drives are subjected to
periodical testing by the SSA adapter card to assure that they will be fully
operational if they are needed.
Another improvement in the Multiprise 3000 implementation is that management of
the RAID-5 arrays is relegated to the SSA adapter rather than being performed at
the SAP. This increases the maximum throughput for the Internal Disk subsystem.
HDD Characteristics
The high performance 18GB UltraStar 18ZX which has a rotational speed of 10,020
rpm is the HDD used on Internal Disk for the Multiprise 3000. Along with its high
rpm and lower latency, the 18GB UltraStar 18ZX has state of the art data rate and
seek times. Some general characteristics of the 18GB UltraStar 18ZX are:
Fixed block size of 524 bytes.
Chapter 1. Overview of the S/390 Internal Disk Subsystem 1-3

Drive is mounted on a carrier which facilitates installation into the HDD
enclosure and provides additional shock protection for the HDD. The carrier
includes 3 LEDs:
–Green LED indicating good power
–Another Green LED indicating activity
–Amber LED indicating a check
Other major disk drive parameters of interest can be seen in the table that follows.
Table 1-1. 18GB UltraStar 18ZX Parameters
Usable Capacity 18 GBytes
Rotational Speed 10,000 RPM
Average Read Seek 6.5 ms
Latency 2.99 ms
Media Transfer Rate 23.4 to 30.4 MB/sec
SSA Transfer Rate 40 MB/sec
HDD Buffer Capacity 4 MBytes
Internal Disk RAID Fast Write
To improve write performance, Internal Disk for Multiprise 3000 includes a 32MB
Non-Volatile RAM (NVRAM) memory. The NVRAM is incorporated on the SSA
adapter card. Some advantages of the NVRAM addition are:
The NVRAM is a centralized storage that dynamically services the most active
HDDs on the SSA adapter card.
The NVRAM operates without a dependency upon the Internal Battery Feature.
The NVRAM retains all of its data in the event of a system or room Emergency
Power Off (EPO).
The operating system should issue a Sense Subsystem Status command to
determine the RAID fast write status of the associated logical volume and issue a
message to the operator indicating that the state change occurred. The RAID fast
write status is conveyed in byte 26 of the returned data. See "Sense Subsystem
Status" on page 4-132.
The MVS and VSE operating systems allow the operator to issue a query at any
time to display the current RAID fast write status of any volume. Alternatively,
ICKDSF can be used to query the status of a RAID array that is undergoing rebuild.
This function will work for VM as well as MVS and VSE.
Internal Disk Facilities
The Internal Disk Subsystem provides the following facilities:
1-4 Internal Disk Subsystem Reference Guide

CKD Emulation
The Internal Disk Subsystem processes ECKD/CKD channel programs which
presuppose variable length Count, Key, and Data fields like those of a 3990
subsystem. Internal Disk constructs CKD track images in memory and executes the
ECKD and CKD channel programs, which operate on byte boundaries, against the
images. In contrast, the Internal Disk Subsystem HDDs are SSA devices, designed
for data transfer via fixed block data transfers rather than the variable length CKD
fields. To bridge between the fixed block disks and the variable length ECKD/CKD
architecture, the CKD track images in memory are mapped onto a series of fixed
blocks suitable for transfer to and from the disk subsystem.
Internal Disk provides the ECKD command support and the compatibility mode
CKD command support as specified in the ECKD Architecture document.
Control Unit and Device Emulation
To maintain compatibility with current software, Internal Disk emulates an existing
control unit and existing device types and models. The 3990-2 is the control unit
that is emulated. The 3990-2 appears to be attached to a dedicated ESCON
channel. The CKD emulation and the emulated ESCON attachment allow Internal
Disk to perform all channel data transfers as non-synchronous operations.
The logical volumes are emulated 3380 model J, E, or K volumes and
3390 model 1, 2, 3, or 9 volumes. The 3380 model J has the same track format
and number of cylinders as the 3380 Standard and the 3380 model D. By
emulating 3380 model J, E, or K volumes, all of the members of the 3380 family
are emulated.
The large capacity of the 18GB UltraStar 18ZX devices coupled with the RAID
organization allows multiple logical volumes to be mapped onto a physical RAID
array. The mappings of logical volumes that have been provided are detailed in the
"Internal Disk Logical Volume Configurations" section of this overview.
LPAR Sharing
LPAR sharing is the ability for individual logical volumes to be shared among logical
partitions. The control unit emulation function in the SAP provides the appropriate
serialization and ensures that required allegiances are maintained. For example, if
a logical volume is reserved to one logical partition, any I/O requests from other
logical partitions are held in abeyance until the reservation is removed. LPAR
sharing does not enable Internal Disk to be shared between LPARs on different
physical CPCs.
LPAR sharing is enabled by the customer in his definition of the I/O configuration
via HCD, VM Dynamic I/O Configuration, or IOCP.
Enhanced HDD Reliability
Chapter 1. Overview of the S/390 Internal Disk Subsystem 1-5

Background Scrubbing
Background scrubbing is provided by Internal Disk to further reduce the probability
of customer loss of data.
There is a small possibility that after a data failure occurs, the redundancy data is
also found to be unreadable, resulting in loss of data to the customer. Data
scrubbing is provided to help ensure that all the data in the disk subsystem is
periodically proven readable so that if a failure occurs, the redundancy exists to
fully restore the original data.
In the background, the SSA adapter card initiates reads of all the data on all the
HDDs in the subsystem. No data transfer to the system is involved. Problems with
the data are promptly corrected by use of the redundancy data within the RAID
array.
Predictive Failure Analysis
Internal Disk is notified of impending hardware failures by the disk drive, which is
continuously performing predictive failure analysis. This process predicts device
failure conditions before failure occurs by measuring various parameters of the
head and disk as well as device electronics. When one of the predictive functions
detects an anomaly, the disk drive notifies the attachment of the potential problem.
The attachment in turn notifies the CPC service element, which initiates a service
action request.
Media Maintenance
The 18GB UltraStar 18ZX HDDs used within Internal Disk incorporate their own
internal media maintenance capabilities. Internal Disk media maintenance strategy
is to capitalize on the HDD capabilities wherever possible. Media maintenance is
performed internal to the storage subsystem. Therefore, defective sectors are
reassigned internally and require no customer interaction.
Search Assist
The Internal Disk emulates a 3990-2 control unit. Many of the channel programs
that were written for the 3990 find a target record by repetitively executing a
Search, TIC (Transfer in Channel) loop at every record until either the search is
satisfied or an exception such as End of Cylinder or No Record Found occurs. An
example of such a channel program is the PDS Directory Search channel program
which has the following form:
Seek
Define Extent
Search ID Equal
TIC *-1 (To the previous command)
Read Count
Search Key Equal or High
TIC *-2 (To the previous Read Count command)
Read Data or other command
The above program has two loops in it. Internal Disk detects both loops. Each of
the CCWs is executed in sequence, however, unlike a channel attached outboard
subsystem, Internal Disk is able to detect the TIC command along with its
parameters. The address to which the TIC command is pointing reveals a
Search/TIC loop. The parameters from the Search commands are preserved within
the subsystem, so that a search algorithm to find the desired record can be
1-6 Internal Disk Subsystem Reference Guide

executed when a loop is detected. With only one transfer of command parameters
from the application program, either the desired record is found or an exception
condition occurs. A second transfer of parameters may then be needed to structure
the ending conditions for the loop, such as providing Read Count information that
points to the correct final record.
Since Internal Disk has access to the channel program and its parameters,
detection of the loops does not depend on mode settings or hints from supervisory
software.
SSID
Subsystem identifier (SSID) support is provided for the unique identification of
Internal Disk control units within a customer installation.Three fixed default values
of SSID are provided:
An SSID of x'FD00' is assigned to the logical subsystem that services the disks
within the CPC frame.
An SSID of x'FD01' is assigned to the logical subsystem that services the disks
within the first expansion frame.
An SSID of x'FD02' is assigned to the logical subsystem that services the disks
within the second expansion frame.
Long Busy
The 3990 mod 2 uses the long busy indication to indicate to the host that an
extended operation is being executed at the device. Internal Disk uses the long
busy indication in some rare instances of subsystem failure as a means for having
CCW chains redriven from the host system.
The 3990 Models 2 and 3 documentation used “State Change Pending”to describe
the condition that is called “Long Busy”in this document.
Internal Disk Components
The following section describes the basic components of the Internal Disk
Subsystem. The Internal Disk is ordered by capacity. Based on the capacity,
varying numbers of components are required.
An Internal Disk Subsystem consists of the attachment integrated in the CPC and
some number of HDDs configured into RAID arrays. The major components
comprising an Internal Disk Subsystem are the SSA adapter cards, 18GB UltraStar
18ZX disk drives, and packaging for the HDDs within the CPC frame or expansion
frames.
Shared CPC resources
The CPC provides physical mounting facilities as well as power, cooling and
service panel facilities for the Internal Disk Subsystem. Special Licensed Internal
Code (LIC) running on the System Assist Processor (SAP) performs channel,
control unit (3990-2), and device (3380-J/E/K or 3390-1/2/3/9) emulation functions
and manages a portion of the CPC memory as a subsystem read cache across all
Internal Disk Subsystem devices. The read cache capacity can range from 32MB to
2GB. The 2GB cache can only be selected on a Multiprise 3000 which has 4GB
Chapter 1. Overview of the S/390 Internal Disk Subsystem 1-7

total storage, not on 1GB and 2GB models. The size can be changed by the
operator in 32MB increments.
SSA Adapter Card
The SSA adapter card supplies an SSA interface that operates at up to 40MB/sec
for connection of the HDDs. Two levels of physical interface conversion occur
between the Multiprise 3000 main memory and the SSA HDDs. The host Multiprise
3000 provides a Self Timed Interface (STI). A special purpose electronics bridge
called the STI-PCI bridge converts the STI to the upper interface of the SSA
adapter card. The SSA adapter card accepts a 32 bit address and data PCI
interface as its system interface.
The functionality of the SSA adapter card is far more than simple interface
conversion.
The SSA adapter card handles management of the RAID-5 organization. In
addition, the card manages both a 64MB RAM, used as a disk cache, and 32MB of
NVRAM used as fast write cache. Having these microcode intensive functions
performed at the SSA adapter card frees the SAP to do more channel and control
unit emulation, thus increasing the transaction processing power of the Internal Disk
subsystem.
Other features of the SSA Adapter Card include:
Support of 2 SSA loops
Independent Packet Network (IPN) transactions
Hardware scatter/gather
Full duplex communication
HDD hot plugging
Error detection and correction
Data scrubbing
HDD Packaging
An HDD for use on Internal Disk is mounted on a carrier which facilitates
installation and provides additional shock protection. The HDD and its carrier are
then mounted into either the frame that contains the CEC, or one of the two
possible expansion frames.
Internal Disk Logical Volume Configurations
Logical Volumes Mapped onto RAID Arrays
A specific number of HDDs form a RAID array. The RAID array could be regarded
as a single very large capacity device. Operating Systems currently recognize
3380 and 3390 volumes, however, changes would be needed to directly recognize
this new, large capacity device. To make the RAID array usable with minimal
impact on existing S/390 software, Internal Disk divides the space into multiple
3380 and/or 3390 logical volumes and provides addressability to the logical
volumes.
The size of the logical volumes vary. (For example, the 3390-2 is twice as large as
a 3390-1). As building blocks for the logical volumes, each array is first divided into
a sequence of storage units. Storage units then become the grains from which
1-8 Internal Disk Subsystem Reference Guide

logical volumes are built. The number of storage units per array depends on the
size of the array. For the supported array sizes, these are the corresponding
number of storage units:
The following types of logical volumes can be mapped to an array, each requiring
the number of storage units shown:
There are no constraints on the types of logical volumes that can be mapped to an
array. However, the number of logical volumes are restricted by the storage units
available and the availability of logical volume addresses. For example, a 4+P array
has 73 possible storage units and 64 possible logical volume addresses, therefore:
24 logical volumes of 3390-3 can be accommodated using 72 of the 73 storage
units and 24 of the 64 logical volumes. 25 logical volumes of 3390-3 would
exceed the number of storage units available.
64 logical volumes of 3390-1 can be accommodated using 64 of the 73 storage
units and all 64 logical volumes. 65 logical volumes of 3390-1 would exceed
the number of logical volume addresses provided.
Table 1-2. Storage Units Provided by HDDs
Number of HDDs in the Array.*Number of Storage Units
4+P 73
6+P 109
Note: *Does not include spare HDDs
Table 1-3. Storage Units Required for Logical Volume Type
Type of Logical Volume Required Number of Storage Units
3390-1 1
3390-2 2
3390-3 3
3390-9 9
3380-J 1
3380-E 2
3380-K 3
Logical Volume Address Resolution
The following details the rules for number of logical volume addresses that are
supported.
A particular logical volume is reached via an addressing hierarchy:
CHPID Address
Only one CHPID address is used for all of the logical volumes on a Multiprise 3000.
The address is assigned as x'FD'.
Chapter 1. Overview of the S/390 Internal Disk Subsystem 1-9

Control Unit Logical Address
Each of the three enclosures of a fully populated Multiprise 3000 contains a disk
subsystem with a logical 3990 mod 2 control unit.
The logical control unit that controls the HDD enclosure in the CPC frame is
assigned control unit logical address x'0'.
The logical control unit that controls the HDD enclosure in the first expansion
frame is assigned control unit logical address x'1'.
The logical control unit that controls the HDD enclosure in the second
expansion frame is assigned control unit logical address x'2'.
Device Address (on the logical control unit)
Each array is assigned a fixed, contiguous range of device addresses. The number
of device addresses for a 4+P array is 64 and for a 6+P array it is 96. The three
possible arrays in a CPC frame are assigned the device addresses as follows:
First array, 4+P, addresses 0 to 63 (x'00' to x'3F).
Second array, 4+P, addresses 64 to 127 (x'40' to x'7F').
Third array, 4+P, addresses 128 to 191 (x'80' to x'BF').
The three possible arrays in an expansion frame are assigned the device
addresses as follows:
First array, 4+P, addresses 0 to 63 (x'00' to x'3F').
Second array, 6+P, addresses 64 to 159 (x'40' to x'9F').
Third array, 6+P, addresses 160 to 255 (x'A0' to x'FF').
Internal Disk Configuration Granularity
The Internal Disk subsystem may consist of as little as one CPC frame containing
one RAID array, and as much as the CPC frame plus two expansion frames with a
total of nine RAID arrays in the three frames.
Growth of a configuration within a frame is straight forward. The entry CEC frame
has one RAID array. Arrays may be added, one at a time, up to the maximum in
the frame of 3 RAID arrays. One SSA loop on a SSA adapter card manages all of
the HDDs within the CPC frame. The SSA adapter card is capable of supporting
two SSA loops, so one of the loops is unused in the single frame configuration.
Similarly, the expansion frames may contain 1, 2, or 3 RAID arrays. The supported
configurations in the CPC frame:
Table 1-4. Configurations in CPC Frame
Number Configuration Approximate Gross Disk Capacity
(GByte)
B1 4+P, Spare 4 * 18 = 72
B2 4+P, 4+P, Spare 8 * 18 = 144
B3 4+P, 4+P, 4+P, Spare 12 * 18 = 216
When the first expansion frame is added, another SSA adapter card is also added.
If a second expansion frame is added, the SSA adapter card in the CPC frame
services the HDD enclosure in that frame.
1-10 Internal Disk Subsystem Reference Guide

Table 1-5. Configurations in CPC Frame
Number Configuration Approximate Gross Disk Capacity
(GByte)
E1 4+P, Spare 4 * 18 = 72
E2 4+P, 6+P, Spare 10 * 18 = 180
E3 4+P, 6+P, 6+P, Spare 16 * 18 = 288
Commands and Facilities Not Supported
Internal Disk does not support the following commands and facilities of the 3990
Model 2 storage control unit:
3380 track compatibility mode
Alternate/defective track assignment
Device Support Track Facility
Diagnostic Track Facility
Diagnostic Control
Write Special Home Address
Read Special Home Address
Buffered log data
Multipath Reconnection
Write Inhibit
Fencing/Unfencing
Service Information Message (SIM)
Variable length Record Zero key field. Internal Disk supports only a zero length
for Record Zero key field.
Variable length Record Zero data field. Internal Disk supports only a fixed eight
byte length for Record Zero data field.
IOCP Definitions
The Input/Output Configuration Program (IOCP) utility is used to specify Internal
Disk configuration to the hardware. IOCP changes for Internal Disk include using
the DSD (Direct System Device) channel type and unique rules for control units and
devices defined on the new channel:
A channel path type of DSD (Direct System Device) is defined for Internal Disk
interfaces. The DSD CHPID can be dedicated or shared, or reconfigurable
(REC).
There is only one valid DSD path on the Multiprise 3000. It is x'FD'.
IOCP requires that the control unit type defined on a DSD CHPID begin with
"3990". Other characters (e.g., -2) may follow 3990, but the "UNIT=" keyword
value must always begin with 3990.
If multiple control units are defined on a CHPID, each must have a unique
Control Unit Logical Address specified. The "CUADD" keyword is used for this
and the value must be between 0-2. Each SSA loop is defined as a unique
CUADD value.
An Internal Disk control unit defined on a DSD CHPID must contain only one
path. The devices defined on a DSD control unit represent the logical volumes
Chapter 1. Overview of the S/390 Internal Disk Subsystem 1-11

defined for each RAID-5 array. Each logical volume may only be defined to one
control unit.
The Self Timed Interface (STI) connection between the processor and the SSA
adapter is specified as the single DSD CHPID. Control unit (CNTLUNIT) statements
are defined for each SSA loop in the storage facility. I/O device (IODEVICE)
statements are needed for each logical volume defined on each RAID-5 array on a
SSA loop.
For details about the IOCP specifications of Internal Disk, see Input/Output
Configuration Program User's Guide and ESCON Channel- to-Channel Reference
(GC38-0401).
RAS Characteristics
Internal Disk is a feature of the CPC platforms with which it is packaged. The RAS
characteristics of Internal Disk are intended to complement the RAS Characteristics
and Service Strategy of the host.
1-12 Internal Disk Subsystem Reference Guide