IBM 71413SU - System x3950 M2 Service manual

© Copyright IBM Corp. 2007, 2008. All rights reserved. ibm.com/redbooks 1
Redpaper
IBM System x3950 M2 and x3850 M2
Technical Introduction
Table of Contents
Summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Comparing the x3850 M2 with the x3850. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Comparing the x3950 M2 with the x3950. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
IBM ScaleXpander Option Kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Multinode configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Front and rear layouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Current models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
IBM XA-64e fourth-generation chipset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
XceL4v Dynamic Server Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
System memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
SAS disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
PCI subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Broadcom dual Gigabit Ethernet controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Light path diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Baseboard Management service processor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Remote Supervisor Adapter II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Trusted Platform Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Active Energy Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
The team that wrote this IBM Redpaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
David Watts
Robert Moon

2IBM System x3950 M2 and x3850 M2 Technical Introduction
Summary of changes
This edition of this paper reflects the changes that are summarized here. Revision bars
throughout the paper indicate where the changes have been made.
New information
򐂰The new IBM® System x3950 M2, page 3
򐂰Table comparing the x3950 M2 with the x3950, page 5
򐂰Support for 2-node configurations, with plans to support 3-node and 4-node complexes,
page 7
򐂰New ScaleXpander Option Kit to upgrade an x3850 M2 to an x3950 M2, page 6
򐂰Models of the x3950 M2, page 12
򐂰Models of the x3950 M2 supporting Windows® Server 2003, DataCenter Edition, page 12
򐂰Processor options for the x3950 M2 models, page 17
򐂰New 8 GB memory DIMMs to allow up to 256 GB of RAM per chassis, page 21
򐂰The stripe size of the LSI 1078 Integrated RAID controller is 64 KB and cannot be
changed, page 29
򐂰Added the FRU number for the battery backup unit for the ServeRAID™ MR10M, page 31
򐂰Added links to the ServeRAID MR10 documentation, page 31
Changed information
򐂰More information about the ServeRAID MR10k controller, page 29
򐂰Corrected the FRU for the replacement battery backup unit for the ServeRAID MR10k
controller, page 29
򐂰Arrays created by the LSI 1078 Integrated RAID controller can be migrated for use with
the ServeRAID MR10k, page 29

IBM System x3950 M2 and x3850 M2 Technical Introduction 3
Overview
This paper introduces the new IBM System x3950 M2 and the previously announced System
x3850 M2, and provides you with detailed technical information about their major
subsystems.
Delivering an industry-leading, 64-bit framework for high-performance scalable computing,
the IBM System x3850 M2 and x3950 M2 are built on the power of the latest eX4
Architecture, which is the fourth generation of the IBM Enterprise X-Architecture® technology.
eX4 Architecture drives the x3850 M2 and x3950 M2 to deliver the performance, availability,
expandability, and manageability required for the next generation of industry-standard
servers.
Figure 1 The IBM System x3850 M2 and IBM System x3950 M2
Both the x3850 M2 and x3950 M2 combine unmatched four-socket 64-bit performance, the
latest Intel® Xeon dual-core or quad-core processors, more flexible XpandOnDemand
modular scalability, and enhanced mission-critical availability to deliver an optimized
(enterprise or on-demand) solution for scale-up virtualization, database, mid-tier application
services or terminal services.
With its extensive chipset development experience, industry-leading performance, and
availability breakthroughs, IBM is uniquely positioned to propose a robust and powerful
server, offering innovation that delivers real business and IT results.
The x3850 M2 is a standalone server with four processor sockets up to 32 DIMM sockets.
The x3950 M2 also has four processor sockets and up to 32 DIMM sockets, but the x3950 M2
can also be connected to other x3950 M2 systems to form a single-image multinode complex.
With the addition of the ScaleXpander Option Kit, the x3850 M2 can be upgraded to the
x3950 M2 and can be connected to up to three other x3950 M2 to form a four-node complex
with 16 processor sockets. Installing the ScaleXpander Option Kit to an x3850 M2 makes it
effectively an x3950 M2.
The following configurations are possible:
򐂰One server: A single x3850 M2 or x3950 M2 server with two or four processors and up to
256 GB RAM installed
򐂰Two servers: A two-node complex comprised of two x3950 M2 servers, with four or eight
processors and up to 512 GB RAM installed
Note: In this paper, when we refer to an x3950 M2, we mean either an x3950 M2 or an
x3850 M2 that has the ScaleXpander Option Kit installed.

4IBM System x3950 M2 and x3850 M2 Technical Introduction
򐂰Three servers: A three-node complex comprised of three x3950 M2 servers, with six or 12
processors and up to 768 GB RAM installed
򐂰Four servers: A four-node complex comprised of four x3950 M2 servers, with eight or 16
processors and up to 1 TB RAM installed
The following are the key features of the x3850 M2 and x3950 M2:
򐂰Four-way1-capable server.
򐂰eX4 Architecture featuring the XA-64e fourth-generation chipset.
򐂰Two standard Intel Xeon® MP dual-core or quad-core processors, upgradable to
four-way. These processors support 64-bit addressing with the Intel 64 Technology
architecture.
򐂰Support for Intel Virtualization Technology (VT).
򐂰Support for an internal removable flash drive installed in a dedicated USB connector on
the system board.
򐂰4 GB or 8 GB memory standard expandable to 256 GB (using 8 GB DIMMs), using
high-performance PC2-5300 ECC DDR2 DIMMs.
򐂰Active Memory™ with Memory ProteXion, memory mirroring, memory hot-swap and
hot-add, and ChipKill.
򐂰Seven half-length 64-bit PCI Express x8 slots, two of which are hot-swap.
򐂰Integrated LSI 1078 serial-attached SCSI (SAS) controller.
򐂰Supports the RAID-0 and the RAID-1 standards. To enable additional RAID features and a
256 MB battery-backed cache, an optional ServeRAID-MR10k RAID controller is
available.
򐂰Four internal hot-swap drive bays for up to 584 GB of internal storage (using 146 GB
disks).
򐂰Integrated Dual-port Broadcom 5709C PCI Express Gigabit Ethernet controller.
򐂰Onboard Baseboard Management Controller and Remote Supervisor Adapter II adapter
are both standard.
򐂰Three-year warranty onsite, nine hours per day, five days per week, with a next business
day response.
The x3850 M2 and x3950 M2 are optimized for ERP, high-end database, and server
consolidation applications.
Comparing the x3850 M2 with the x3850
Table 1 on page 5 shows the major differences between the x3850 and the x3850 M2.
Note: Support for three-node and four-node configurations is planned for 2Q08.
1 4-way means 4 processor sockets. In this document, we use way to indicate a processor socket regardless of
whether it is a dual-core processor or a quad-core processor.

IBM System x3950 M2 and x3850 M2 Technical Introduction 5
Table 1 Major differences between x3850 and x3850 M2
Comparing the x3950 M2 with the x3950
Table 2 shows the major differences between the x3950 and the x3950 M2.
Table 2 Major differences between x3950 and x3950 M2
Feature System x3850 server System x3850 M2 server
Processors Dual-core Intel Xeon 7100 series Dual-core Intel Xeon E7210 and
quad-core Intel Xeon E7300 series
processors
Frontside bus Two 667 MHz (two processors on each
bus)
Four 1066 MHz (one processor on each
bus)
Memory controller Hurricane 3.0 Hurricane 4.0
Memory Maximum of 4 memory cards, each with
4 DDR2 DIMM slots running at 333 MHz
supporting a total of 16 DDR2 DIMMs
Maximum of 4 memory card, each with
8 DDR2 DIMM slots running at 533 MHz
supporting a total of 32 DDR2 DIMMs
Scalability Not supported Upgradeable to support multinode scaling
with the ScaleXpander Option Kit,
44E4249
Disk subsystem Adaptec AIC9410 SAS LSI 1078 SAS
External disk port No Yes (SAS x4)
RAID support Standard not supported only via optional
ServeRAID-8i
Standard RAID-0 and RAID-1; additional
RAID features via optional
ServeRAID-MR10k
PCI-X slots Two or six, depending on model None
PCI Express slots Some models have four PCI Express x8
full-length slots
Seven PCI Express x8 half-length slots
Active PCI slots (hot-swap) Six Two
Video controller ATI™ Radeon™ 7000M 16 MB onboard ATI ES1000™ 16MB memory on RSA II
USB ports Three (front: one, rear: two) Six (front: two, rear: three, internal: one)
Keyboard and mouse
connectors
PS/2 USB
Service processor RSA II SlimLine adapter
(optional on some models)
RSA II PCI-X adapter
Mechanical 3U height 4U height
Trusted Platform Module Not implemented Trusted Platform Module (TPM) with TCG
V1.2 compliance
Power supplies One or two 1300 W power supplies,
depending on model
Two 1440 W power supplies
Feature x3950 server x3950 M2 server
Enterprise X-Architecture Third-generation XA-64e chipset Fourth generation XA-64e chipset

6IBM System x3950 M2 and x3850 M2 Technical Introduction
IBM ScaleXpander Option Kit
Unlike the x3850, the x3850 M2 can be converted to an x3950 M2 through the use of the
ScaleXpander Option Kit, part number 44E4249. After this kit is installed the x3850 M2
functionally becomes an x3950 M2, and is therefore able to form part of a multi-way
configuration.
The IBM XpandOnDemand Scalability Kit contains the following items:
򐂰Scalability cable 3.08m
Processors Dual-core Intel Xeon 7100 series Dual-core Intel Xeon E7210 and
quad-core Intel Xeon E7300 series
processors
Frontside bus Two 667 MHz (two processors on each
bus)
Four 1066 MHz (one processor on each
bus)
Memory controller Hurricane 3.0 Hurricane 4.0
Maximum SMP 32 sockets using eight chassis.
With dual-core processors, maximum of
64 cores.
16 sockets using four chassis.
With quad-core processors, maximum of
64 cores.
Memory 16 DDR2 DIMM sockets per node.
Maximum of 4 memory cards, each with 4
DDR2 DIMM slots running at 333 MHz.
64 GB maximum per node.
512 GB maximum with 8 nodes.
32 DDR2 DIMM sockets per node.
Maximum of 4 memory card, each with 8
DDR2 DIMM slots running at 533 MHz.
256 GB maximum per node.
1 TB maximum with 4 nodes.
Internal disks Six hot-swap bays Four hot-swap bays
Disk subsystem Adaptec AIC9410, no external port LSI 1078, external SAS 4x port
RAID support Standard not supported only via optional
ServeRAID-8i
Standard RAID-0 and RAID-1, additional
RAID features via optional
ServeRAID-MR10k
PCI-X slots per node Two or six, depending on model None
PCI Express slots per node Some models have four PCI Express x8
full-length slots
Seven PCI Express x8 half-length slots
Active PCI slots Six Two
Ethernet controller Broadcom 5704 dual Gigabit Ethernet Broadcom 5709C dual Gigabit Ethernet
Video controller ATI Radeon 7000M 16 MB onboard ATI ES1000 16MB memory on RSA II
Keyboard and mouse
connectors
PS/2 USB
Service processor RSA II SlimLine standard RSA II standard
Trusted Platform Module Not implemented Trusted Platform Module (TPM) with TCG
V1.2 compliance.
Power supply Two 1300W supplies Two 1440W supplies
Mechanical 3U height 4U height
Feature x3950 server x3950 M2 server

IBM System x3950 M2 and x3850 M2 Technical Introduction 7
򐂰Larger cable management arm (replaces the existing arm to allow the easy installation of
the scalability cables)
򐂰ScaleXpander chip (transforms the x3850 M2 into an x3950 M2)
򐂰x3950 M2 bezel (replaces the existing bezel, and shows that the x3850 M2 has the kit
installed and is now functionally equal to an x3950 M2)
The ScaleXpander chip goes in a dedicated socket towards the front of the system board as
shown in Figure 2.
Figure 2 The ScaleXpander chip installed in the x3850 M2 enables the server to scale
Multinode configurations
The x3950 M2 is the base building block, or node, for a scalable system. At their most basic,
these nodes are comprised of four-way SMP-capable systems with processors, memory, and
I/O devices. The x3950 M2 is the building block that allows supported 8-way, 12-way and
16-way configurations by adding additional x3950 M2s as required.
Unlike with the System x3950 and xSeries® 460, there is no special modular expansion
enclosure for the x3950 M2. The multinode configuration is simply formed by using another
x3950 M2 or an x3850 M2 that has the ScaleXpander Option Kit installed as described in
“IBM ScaleXpander Option Kit” on page 6.
Important: The IBM ScaleXpander Option Kit is only required for an x3850 M2 that you
want to convert to an x3950 M2 and enable it to scale.
The x3950 M2 ships with the necessary components (including a scalability cable) to be
able to form an N-way configuration, and therefore does not require this option.
Note: When we refer to an x3950 M2, we mean either an x3950 M2 or an x3850 M2 that
has the ScaleXpander Option Kit installed.

8IBM System x3950 M2 and x3850 M2 Technical Introduction
The x3950 M2 can form a multinode configuration by adding one or more x3950 M2 servers.
The following configurations are possible, as shown in Figure 3:
򐂰A two-node complex comprised of two x3950 M2 servers, with four or eight processors
and up to 512 GB RAM installed
򐂰A three-node complex comprised of three x3950 M2 servers, with six or 12 processors
and up to 768 GB RAM installed
򐂰A four-node complex comprised of four x3950 M2 servers, with eight or 16 processors and
up to 1 TB RAM installed
Figure 3 Supported multinode configurations
The following configuration rules apply:
򐂰Combinations of x3950 M2s
It is only supported to have multi-node complexes that are made up of x3950 M2 servers
Other nodes like the older x460, MXE-460, x3950 (machine type 8872 and 8878) and
x3950 E (machine type 8874 and 8879) are not supported.
򐂰Processors
Each x3950 M2 server must have either two or four processors installed, and all
processors in the complex must be the same speed and cache size.
򐂰Memory
For performance reasons, you should have the same amount of memory in each node.
However, a minimum of 4 GB of RAM is required in each node because otherwise, the
node will not be able to join the multinode configuration.
򐂰Firmware
All system firmware, including the system BIOS, diagnostics, BMC firmware, and RSA II
firmware, must be at the same level across all systems.
Note: Support for three-node and four-node configurations is planned for 2Q08.
Four nodes*
8-way or 16-way
(Each node is
2-way or 4-way)
Up to 1 TB RAM
x3950 M2
x3950 M2
x3950 M2
x3950 M2
One node
2-way or 4-way
Up to 256 GB RAM
x3950 M2
Two nodes
4-way or 8-way
(Each node is
2-way or 4-way)
Up to 512 GB RAM
x3950 M2
x3950 M2
Three nodes*
6-way or 12-way
(Each node is
2-way or 4-way)
Up to 768 GB RAM
x3950 M2
x3950 M2
x3950 M2
* 3-node and 4-node configurations are
planned to be supported in 2Q08

IBM System x3950 M2 and x3850 M2 Technical Introduction 9
Updating the system BIOS in every node in a scalable system can be performed from the
primary node. The server diagnostics, as well as the RSA II and BMC firmware, must be
individually updated on each node, but this can be performed remotely, as described:
– The RSA II firmware can be updated using the RSA II Web interface or IBM Director.
– The server diagnostics and BMC firmware can be updated with an RSA II remote
console session using the remote diskette function.
򐂰Disk drives installed in any of the x3950 M2 are seen by the operating system as normal
disk drives.
򐂰You can only have a maximum of two optional ServeRAID-MR10k adapters in the
multinode configuration. The drives in the other nodes will need to remain connected
using the built-in SAS controller.
򐂰All PCI Express slots and onboard Gigabit Ethernet ports in the x3950 M2 are visible to
the operating system, as well.
A fully configured, four-node, scalable system with quad-core processors would have
64 cores, 1024 GB of memory (using 8 GB DIMMs), 28 PCI Express adapters, 2.3 TB of raw
disk space and eight Gigabit Ethernet connections.
Scalable systems setup
Certain tasks must be performed before a multinode configuration can be operated
successfully:
򐂰All system firmware, including the system BIOS, diagnostics, BMC firmware and RSA II
firmware, must be at the same level across all systems.
򐂰Memory settings for HPMA, HAM, FAMM in BIOS must be standardized across all
systems in the multinode configuration while the nodes are still logically separate (that is,
pre-merge). Refer to Table 11 on page 28 for details about this topic.
򐂰You will need separate KVM connections to each node. Alternatively, you can configure
them using an RSA II remote console session.
򐂰Post-merge settings that can be applied to the primary node are:
– Advanced Setup - PCI settings
– Passwords
– Boot order
Scalability ports and cables
Nodes in an x3950 M2 multi-node complex are connected together by a number of scalability
cables. These cables are attached to three SMP Expansion Ports (commonly referred to as
scalability ports) on the rear of each system, as shown in Figure 6 on page 11.
The scalability ports interface directly to the eX4 Architecture chipset and allow high speed
communication between processors located in different chassis. The ports act like hardware
extensions to the processor local buses. They direct read and write cycles to the appropriate
memory or I/O resources, as well as maintain cache coherency between the processors.
These scalability ports are connected together with scalability cables to enable configuration
of multinode scalable systems up to 16-way.
Note: The US law restricting the export of high capacity servers was relaxed in 2002.
Therefore, this restriction no longer affects the x3850 M2 and x3950 M2.

10 IBM System x3950 M2 and x3850 M2 Technical Introduction
Currently, only one-node and two-node configurations are supported, although plans are in
place to support three-node and four-node configurations in 2Q08.
The scalability cabling needed to form a two-node complex is shown in Figure 4.
Figure 4 Cabling for a two-node configuration
Two 3.08m cables are required, as shown in Figure 4. The cables are supplied as follows:
򐂰One cable is part of the ScaleXpander Option Kit, part number 44E4249, as described in
“IBM ScaleXpander Option Kit” on page 6, for use with x3850 M2 servers.
򐂰One cable is included with x3950 M2 servers (not with x3850 M2).
The following key points relate to scalability cabling:
򐂰The x3950 M2 comes with a new cable management arm that makes installation and
management of the cables easier. The ScaleXpander Option Kit for the x3850 M2 ships
with this new arm to replace its standard cable management arm.
򐂰The scalability ports cannot be used as high speed interconnects for clustering purposes.
Use the integrated Gigabit Ethernet controller instead.
򐂰The connections do offer redundancy. If a connection is lost (for example, if a cable is
unplugged), there will be an indication on the light path diagnostic panel showing a link
failure, and the server will disable the failing port and switch traffic to the remaining path.
There will be a slight performance degradation in this mode. After the cable is reattached,
the server will re-enable the path and clear the error from the light path diagnostic panel.
򐂰In multinode configurations, the inbound communication between each node now is
handled by the scalability cables, and not by the RSA II Ethernet ports (as it was with the
x3950).
Partitioning
Partitioning is the concept of logically splitting up a multinode complex into separate
systems. You can then install an operating system on a partition and have it run
independently from all other partitions.
The advantage of partitioning is that you can create and delete partitions without having to
recable the complex. The only requirement is that partitions be formed on node boundaries.
The interface where you set up and maintain partitions is an extension of the Remote
Supervisor Adapter II Web interface. It is used to create, delete, control, and view scalable
partitions.
Note: Connectivity for three-node and four-node configurations is not supported until
2Q08. When this support is announced, this document will be updated with cabling
information.
Two-node configuration
Port 3
x3950 M2
(primary) Port 3
x3950 M2 RSAII
RSAII
Ethernet
network
Port 1 Port 2
Port 1 Port 2 3.08m cable