Lenovo System x3850 X6 User manual

© Copyright Lenovo 2015. All rights reserved. ibm.com/redbooks 1
Lenovo System x3850 X6 and x3950 X6
Quick Start Guide
Abstract
The Lenovo® System x3850 X6 and x3950 X6 servers are new four-socket and eight-socket
servers that deliver fast application performance thanks to an innovative scalable design and
new storage technology that is designed to optimize overall solution performance. These
servers are based on the highly scalable Intel Xeon processor E7-4800 v2 and E7-8800 v2
product families and are the first servers designed and optimized for eXFlash™
memory-channel storage, the latest in ultra-low latency flash storage technology.
This quick start guide addresses the preferred practices for setting up the x3850 X6 and
x3950 X6 servers. It provides all of the information necessary to unpack a system and be up
and running in the minimal amount of time. The paper provides insightful information from
experts in the field and in development and covers the system architecture and major
subsystems such as memory, I/O, power and cooling. The paper also offers a pre-installation
checklist and tips on troubleshooting. This paper is intended for pre-sales and post-sales
technical support professionals.
Contents
Introducing the x3850 X5 and x3950 X6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
I/O subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Power subsystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Cooling subsystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Pre-installation checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Troubleshooting tips. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Related publications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
John R. Encizo
Timothy M. Wiwel
Loc X. Nguyen

2Lenovo System x3850 X6 and x3950 X6 Quick Start Guide
Introducing the x3850 X5 and x3950 X6
The Lenovo X6 product portfolio represents the sixth generation of servers that are built upon
Enterprise X-Architecture. Enterprise X-Architecture is the culmination of generations of
Lenovo technology and innovation that is derived from the experience in high-end enterprise
servers. Now, with the X6 servers, scalable systems can be expanded on demand and
configured by using a building block approach that optimizes system design for your workload
requirements. These servers scale to more processor cores, memory, and I/O than previous
systems, enabling them to handle greater workloads than the systems that they supersede.
Power efficiency and server density are optimized, making them affordable to own and
operate.
The Lenovo System x3850 X6 and x3950 X6 servers deliver fast application performance
thanks to an innovative scalable design and new storage technology that is designed to
optimize overall solution performance. The X6 servers are the first servers designed and
optimized for eXFlash memory-channel storage. With eXFlash memory-channel storage, they
can deliver up to 12.8 TB of ultra-low latency flash storage. With the new Intel Xeon processor
E7-4800 v2 and E7-8800 v2 product families, the x3850 X6 and x3950 X6 servers can deliver
up to 6.0 TB or 12 TB of memory and 60 or 120 cores of processing power, respectively.
Figure 1 shows the x3850 X6 server.
Figure 1 x3850 X6 server
The x3850 X6 server has the following key characteristics:
򐂰Up to four Intel Xeon processor E7-4800 v2 or E7-8800 v2 product family processors
򐂰Up to 96 DIMM slots (24 DIMM slots per processor) for up to 6 TB of memory (using
64 GB DIMMs)
򐂰Up to 1600 MHz DDR3 memory speeds and up to 2667 MHz SMI2 link speeds
򐂰Up to 12.8 TB of eXFlash memory-channel storage
򐂰Up to eight 2.5-inch hot-swap drives or up to 16 1.8-inch hot-swap solid-state drives
(SSDs)
򐂰Support for 12 Gbps serial-attached SCSI (SAS) connectivity for the internal storage

Lenovo System x3850 X6 and x3950 X6 Quick Start Guide 3
򐂰Mezzanine LOM (ML) slot for the integrated network interface controller (NIC) functionality
(choice of dual-port 10 GbE or quad-port 1 GbE adapters)
򐂰Up to 11 PCIe 3.0 I/O slots
򐂰Internal USB port for the embedded hypervisor
򐂰Three USB ports on the front of the server, 2x USB 3.0 and 1x USB 2.0
Figure 2 shows the x3950 X6 server.
Figure 2 x3950 X6 server
The x3950 X6 server has the following key characteristics:
򐂰Up to eight Intel Xeon processor E7-8800 v2 product family processors
򐂰Up to 192 DIMM slots (24 DIMM slots per processor) for up to 12 TB of memory (using
64 GB DIMMs)
򐂰Up to 1600 MHz DDR3 memory speeds and up to 2667 MHz SMI2 link speeds
򐂰Up to 12.8 TB of eXFlash memory-channel storage
򐂰Up to 16 2.5-inch hot-swap drives or up to 30 two 1.8-inch hot-swap SSDs
򐂰Support for 12 Gbps SAS connectivity for the internal storage
򐂰Two ML slots for the integrated NIC functionality (choice of dual-port 10 GbE or quad-port
1 GbE adapters)
򐂰Up to 22 PCIe 3.0 I/O slots
򐂰Two internal USB ports for the embedded hypervisors
򐂰Six USB ports on the front of the server, 4x USB 3.0 and 2x USB 2.0

4Lenovo System x3850 X6 and x3950 X6 Quick Start Guide
The supported operating systems for both x3850 X6 server and x3950 X6 server include:
򐂰Microsoft Windows Server 2008 R2
򐂰Microsoft Windows Server 2012
򐂰Microsoft Windows Server 2012 R2
򐂰Red Hat Enterprise Linux 6.5+
򐂰SUSE Linux Enterprise Server 11 SP1+
򐂰VMware vSphere (ESXi) 5.1 U1+
򐂰VMware vSphere (ESXi) 5.5
Architecture
To ensure proper function of the X6 server, the system should be configured correctly at the
time of purchase. The block diagram of the system in Figure 3 shows that proper memory and
I/O placement is critical for optimal performance. With the move to the Intel Ivy Bridge
processors in the X6 platform, the I/O is now directly attached to the processors. Additionally,
memory population rules have become more complex because each channel now has three
DIMMs and different memory modes.
There are a few general preferred practices in regards to configuring the system. Memory
should be installed in quantities of eight DIMMs per CPU for optimal performance and to
avoid any potential issues with memory installation order interfering with the memory mode
chosen. For best I/O performance, install the correct amount of adapters so that as many
processors as possible have a direct connection to I/O.
Figure 3 x3850 X6 system architecture
SMI
links
MB 2
MB 1
X6 DDR3
Compute Book
QPI links
Intel
Xeon
CPU 1
MB 2
MB 1
MB 2
MB 1
MB 2
MB 1
MB 2
MB 1
DMI
links
Slot 7: PCIe 3.0 x16 (x16)
Slot 9: PCIe 3.0 x16 (x16)
MB 2
MB 1
MB 2
MB 1
MB 2
MB 1
8x USB
Serial
Management
Video
IMM2
PCIe x1
Slot 10: Mezz LOM (x8)
Slot 12: PCIe 3.0 x16 (x8)
Intel
Xeon
CPU 2
Slot 8: PCIe 3.0 x16 (x8)
Slot 11: PCIe 3.0 x16 (x8)
Intel
Xeon
CPU 4
Intel
Xeon
CPU 3
Slot 1: PCIe 3.0 x16 (x16)
Slot 2: PCIe 3.0 x8 (x8)
Slot 3: PCIe 3.0 x8 (x8)
Slot 4: PCIe 3.0 x16 (x16)
Slot 5: PCIe 2.0 x8 (x8)
Slot 6: PCIe 3.0 x16 (x16)
Storage Book
Primary I/O Book
PCIe 3.0 lanes
Full Length I/O Book
Half Length I/O Book
PCIe 3.0 lanes
Intel
I/O Hub
PCIe
switch

Lenovo System x3850 X6 and x3950 X6 Quick Start Guide 5
Memory subsystem
The memory subsystem can operate in one of the following modes:
򐂰Independent (Performance) mode
򐂰Lockstep (RAS) mode
In Performance mode, each channel on an SMI is addressed independently, essentially
providing the ability to run two transactions at once. In Lockstep mode, a pair of DIMMs
spread across both channels on an SMI is addressed simultaneously. Lockstep mode
provides higher memory frequency and memory reliability (Double Device Data Correction
+1 bit) but at the expense of higher memory bandwidth. Conversely, Independent mode
provides higher memory bandwidth at the cost of advanced memory RAS features and
memory frequency (Single Device Data Correction only).
Figure 4 shows the two memory modes.
Figure 4 Memory modes: Performance mode (left) and RAS mode (right)
Intel Xeon processor
DIMM
Memory
controller
DIMM
DIMM
Memory
controller
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
Memory
buffer
Memory
buffer
Memory
buffer
Memory
buffer
Data 1Data 0
Intel Xeon processor
DIMM
Memory
controller
DIMM
DIMM
Memory
controller
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
DIMM
Memory
buffer
Memory
buffer
Memory
buffer
Memory
buffer
Data
Lockstep
channel
SMI2
links
DDR3
links
Memory performance mode Memory RAS mode

6Lenovo System x3850 X6 and x3950 X6 Quick Start Guide
Table 1 shows the characteristics of the supported DIMMs and the memory speeds.
Table 1 Characteristics of the supported DIMMs and the memory speeds
DIMM
Specification
RDIMM LR-DIMM
Ranks Single-rank DIMM Dual-rank DIMM 4R LR-DIMM 8R LR-DIMM
Part number 00D5024 (4 GB)
00D5036 (8 GB)
46W0672 (16 GB) 46W0676
(32 GB)
46W0741
(64 GB)
Rated speed 1600 MHz 1600 MHz 1600 MHz 1333 MHz
Rated voltage 1.35 V 1.35 V 1.35 V 1.35 V
Operating voltage 1.35 V 1.5 V 1.35 V 1.5 V 1.35 V 1.5 V 1.35 V 1.5 V
Maximum quantity
supported
96 96 96 96
Maximum DIMM
capacity
8 GB 16 GB 32 GB 64 GB
Maximum memory
capacity
768 GB 1.5 TB 3 TB 6 TB
Maximum operating speed: Independent (Performance) mode
1 DIMM/channel 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz
2 DIMMs/channel 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz
3 DIMMs/channel 1066 MHz 1333 MHz 1066 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz
Maximum operating speed: Lockstep (RAS) mode
1 DIMM/channel 1333 MHz 1600 MHz 1333 MHz 1600 MHz 1333 MHz 1600 MHz 1333 MHz 1333 MHz
2 DIMMs/channel 1333 MHz 1600 MHz 1333 MHz 1600 MHz 1333 MHz 1600 MHz 1333 MHz 1333 MHz
3 DIMMs/channel 1066 MHz 1333 MHz 1066 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz

Lenovo System x3850 X6 and x3950 X6 Quick Start Guide 7
Figure 5 shows the Intel Xeon E7 memory controller architecture. Note that memory is
populated from the furthest DIMM (Slot 0) to the closest DIMM slot (Slot 2).
Figure 5 Xeon E7 memory controller architecture
Intel
Xeon
processor
Memory
controller 1
Memory
buffer 1
DIMM 19 DIMM 21 DIMM 20
Slot2Slot1Slot0
Slot2Slot1Slot0
DIMM 1 DIMM 3 DIMM 2
Memory
controller 2
Memory
buffer 2
DIMM 6 DIMM 4 DIMM 5
Slot2Slot1Slot0
Slot2Slot1Slot0
DIMM 24 DIMM 22 DIMM 23
Memory
buffer 3
DIMM 16 DIMM 18 DIMM 17
Slot2Slot1Slot0
Slot2Slot1Slot0
DIMM 10 DIMM 12 DIMM 11
Memory
channel 0
Memory
channel 1
Memory
channel 2
Memory
channel 3
Memory
buffer 0
DIMM 9 DIMM 7 DIMM 8
Front DIMMs
Slot2Slot1Slot0
Slot2Slot1Slot0
DIMM 15 DIMM 13 DIMM 14
DDR3 ch 0
DDR3 ch 1
DDR3 ch 0
DDR3 ch 1
DDR3 ch 0
DDR3 ch 1
DDR3 ch 0
DDR3 ch 1
Back DIMMs

8Lenovo System x3850 X6 and x3950 X6 Quick Start Guide
Figure 6 and Figure 7 show the central processing unit (CPU) board layout.
Figure 6 CPU layout (left side of board)
Figure 7 CPU layout (right side of board)
DIMM 1 DIMM 2 DIMM 3
DIMM 4 DIMM 5 DIMM 6 DIMM 10 DIMM 11 DIMM 12
DIMM 7 DIMM 8 DIMM 9Microprocessor
(Left side of board)
DIMM 24
DIMM 23
DIMM 22
DIMM 19DIMM 20
DIMM 21
DIMM 15
DIMM 14
DIMM 13
DIMM 18
DIMM 17
DIMM 16
(Right side of board)

Lenovo System x3850 X6 and x3950 X6 Quick Start Guide 9
Table 2 shows the memory layout.
Table 2 Memory layout
Table 3 shows the Independent mode population sequence.
Table 3 Independent mode population sequence
Table 4 shows the Lockstep (RAS) mode population sequence.
Table 4 Lockstep (RAS) mode population sequence
CPU memory channel DDR3 channel DIMM number DDR3 slot number
Channel 0 DDR3 Channel 0 9, 8, 7 0, 1, 2
DDR3 Channel 1 15, 14, 13 0, 1, 2
Channel 1 DDR3 Channel 0 19, 20, 21 0, 1, 2
DDR3 Channel 1 1, 2, 3 0, 1, 2
Channel 2 DDR3 Channel 0 6, 5, 4 0, 1, 2
DDR3 Channel 1 24, 23, 22 0, 1, 2
Channel 3 DDR3 Channel 0 16, 17, 18 0, 1, 2
DDR3 Channel 1 10, 11, 12 0, 1, 2
Installation Population sequence DIMM number DDR3 slot number
1st 1 SMI0, Channel0, Slot0 Front
2nd 2 SMI2, Channel0, Slot0 Front
3rd 3 SMI1, Channel1, Slot0 Front
4th 4 SMI3, Channel1, Slot0 Front
5th 5 SMI0, Channel1, Slot0 Back
6th 6 SMI2, Channel1, Slot0 Back
7th 7 SMI1, Channel0, Slot0 Back
8th 8 SMI3, Channel0, Slot0 Back
... 9 - 16 Repeat order, move to Slot1 Four front, four back
... 17 - 24 Repeat order, move to Slot 2 Four front, four back
Installation Population sequence Controller location Physical location
1st 1 & 2 SMI0, Channel0, Slot0 & SMI0, Channel1, Slot0 Front & back
2nd 3 & 4 SMI2, Channel0, Slot0 & SMI2, Channel1, Slot0 Front & back
3rd 5 & 6 SMI1, Channel1, Slot0 & SMI1, Channel0, Slot0 Front & back
4th 7 & 8 SMI3, Channel1, Slot0 & SMI3, Channel0, Slot0 Front & back
5th 9 & 10 Repeat order, move to Slot1 Front & back
6th 11 & 12 Repeat order, move to Slot1 Front & back
7th 13 & 14 Repeat order, move to Slot1 Front & back

10 Lenovo System x3850 X6 and x3950 X6 Quick Start Guide
Table 5 shows the memory population order for both Independent (Performance) mode and
Lockstep (RAS) mode.
Table 5 Memory population order for both Independent (Performance) mode and Lockstep (RAS)
8th 15 & 16 Repeat order, move to Slot1 Front & back
... 18 - 24 Repeat order, move to Slot2 Front & back
Installation Population sequence Controller location Physical location
DIMM installation order DIMM slots
Performance mode Performance + mirroring RAS mode RAS + mirroring
1 DIMM 9 DIMM 9, 19 DIMM 9, 15 DIMM 1, 9, 15, 19
2 DIMM 6 DIMM 6, 16 DIMM 6, 24 DIMM 6, 10, 16, 24
3 DIMM 1 DIMM 1, 15 DIMM 1, 19 DIMM 2, 8, 14, 20
4 DIMM 10 DIMM 10, 24 DIMM 10, 16 DIMM 5, 11, 17, 23
5 DIMM 15 DIMM 8, 20 DIMM 8, 14 DIMM 3, 7, 13, 21
6 DIMM 24 DIMM 5, 17 DIMM 5, 23 DIMM 4, 12, 18, 22
7 DIMM 19 DIMM 2, 14 DIMM 2, 20 Not applicable
8 DIMM 16 DIMM 11, 23 DIMM 11, 17 ....
9 DIMM 8 DIMM 7, 21 DIMM 7, 13
10 DIMM 5 DIMM 4, 18 DIMM 4, 22
11 DIMM 2 DIMM 3, 13 DIMM 3, 21
12 DIMM 11 DIMM 12, 22 DIMM 12, 18
13 DIMM 14 Not applicable Not applicable
14 DIMM 23 .... ....
15 DIMM 20
16 DIMM 17
17 DIMM 7
18 DIMM 4
19 DIMM 3
20 DIMM 12
21 DIMM 13
22 DIMM 22
23 DIMM 21
24 DIMM 18