Open QCT F06S User manual

Quanta Confidential
QCT F06S/ F06T 19” 2U 4Node/2Node
System “Carmel”
Revision 0.1
2016/1/28
Contributed by: Alan Chang, Quanta Computer Inc.

F06S/F06T 2U System Contribution
1
Revision History:
Revision
Date
Description
0.1
2016-1-28
Release for Open Compute Project

F06S/F06T 2U System Contribution
2
Table of Contents
1 INTRODUCTION........................................................................................................................................4
2 PRODUCT ARCHITECTURE OVERVIEW .....................................................................................................7
2.1 F06S 2U4N System Produt Features................................................................................................................7
2.1.1 Product Features ......................................................................................................................................7
2.1.2 System Block Diagram ..............................................................................................................................8
2.2 F06T 2U2N System Produt Features................................................................................................................9
2.2.1 Product Features ......................................................................................................................................9
2.2.2 System Block Diagram ............................................................................................................................10
2.3 Related Board Placement..............................................................................................................................11
2.3.1 Motherboard Placement ........................................................................................................................11
2.3.2 Interposer Board Placement ..................................................................................................................12
2.4 Related Board Dimension..............................................................................................................................13
2.4.1 Sled Dimension.......................................................................................................................................13
2.4.2 Motherboard Dimension........................................................................................................................14
2.4.3 Interposer Board Dimension ..................................................................................................................14
3 PRODUCT FEATURES ..............................................................................................................................15
3.1 Processor .......................................................................................................................................................15
3.2 Memory .........................................................................................................................................................15
3.2.1 DIMM Nomenclature..............................................................................................................................16
3.3 PCH ................................................................................................................................................................16
3.4 BMC ...............................................................................................................................................................17
3.5 Clocks.............................................................................................................................................................17
3.6 SATA...............................................................................................................................................................17
3.6.1 SATA PORT CONNECTIVITY.....................................................................................................................17
3.7 USB ................................................................................................................................................................18
3.8 PCIe BUS ........................................................................................................................................................19
3.8.1 PCIe PORT CONNECTIVITY ......................................................................................................................19
3.9 PCIe Interface ................................................................................................................................................20
3.9.1 PCIe Connector PIN Definition................................................................................................................21
3.10 LAN on Motherboard (LOM) .......................................................................................................................26
3.11 Share-NIC (BMC management port) ...........................................................................................................26
3.12 LPC BUS........................................................................................................................................................27

F06S/F06T 2U System Contribution
3
3.13 TPM .............................................................................................................................................................27
3.14 Serial port ....................................................................................................................................................27
3.15 FANs.............................................................................................................................................................27
3.16 Jumper Definition........................................................................................................................................27
3.17 Debug header Information..........................................................................................................................29
3.17.1 XDP Support ..........................................................................................................................................29
3.17.2 SMB Debug Header (JP7).......................................................................................................................29
3.17.3 BMC Debug Header (J1 and J10) ...........................................................................................................29
3.18 Product Sensors...........................................................................................................................................30
3.19 SMBUS.........................................................................................................................................................31
3.20 Power Consumption....................................................................................................................................32
3.20.1 Power Supply PINOUT ...........................................................................................................................32
3.21 LED Definition..............................................................................................................................................33
3.21.1 HDD Active LED......................................................................................................................................33
3.21.2 Power+ID LED ........................................................................................................................................33
3.21.3 Beep LED................................................................................................................................................33
3.21.4 Debug Port Switch and LED...................................................................................................................34
3.21.5 BMC Heart beat LED..............................................................................................................................34
4 PRODUCT SYSTEM REQUIREMENTS.......................................................................................................35
4.1 F06S 2U4N System View................................................................................................................................35
4.2 F06T 2U2N System View................................................................................................................................36
4.3 System BIOS and BMC Firmware...................................................................................................................37
4.4 Environmental Requirements........................................................................................................................37
4.5 Power Supply Unit .........................................................................................................................................37
4.6 FAN ................................................................................................................................................................38
4.7 Front Panel LED Function and Behavior ........................................................................................................38
4.8 Regulatory Compliance Specifications ..........................................................................................................39

F06S/F06T 2U System Contribution
4
1INTRODUCTION
Both F06S and F06T (family code name Carmel) are 2RU height systems with OCP compliant 2P server board for
standard 19” EIA rack. F06S is a 2U4N System with four hot-pluggable motherboard Sleds while F06T is a 2U2N
System with two hot-pluggable motherboard Sleds. This specification is to define the interfaces, connection
topology of various buses and control signals between different modules in the system. It includes the pin
definition and basic electrical parameters.
1.1 F06S 2U4N System
F06S 2U4N System is a 2RU 4-Node (2U4N) system for standard 19” EIA rack which comprises of four F06S Sleds,
two power suppler units, four dual-rotor system fans, and twenty-four front loading SFF HDD/SSD trays. F06S
2U4N System consists of four high performance 2P servers featuring Intel Grantley platform. Each F06S Sled
contains an OCP compliant 2P server board (Facebook Server Intel motherboard v3.1,
http://www.opencompute.org/wiki/Motherboard/SpecsAndDesigns ), OCP mezzanine v1 card, Interposer board
and Riser board. It connects with SFP+ connectors of OCP mezzanine v1 card or RJ45 connector of Intel I210 LOM.
The BMC (Baseboard Management Controller) is used to manage the sled itself which supports both in-band
management and out-of-band management through the sideband control of Ethernet LAN.
Figure 1-1 F06S 2U4N System Overview
1.2 F06T 2U2N System
F06T 2U2N System is a 2RU 2-Node (2U2N) system for standard 19” EIA rack which comprises of two F06T sleds,
two power supply units, four system fans and twenty-four front loading SFF HDD/SSD trays. F06T 2U2N System
consists of two high performance 2P servers featuring Intel Grantley platform. Each F06T Sled contains an OCP
compliant 2P server board (Facebook Server Intel motherboard v3.1,

F06S/F06T 2U System Contribution
5
http://www.opencompute.org/wiki/Motherboard/SpecsAndDesigns ), and it connects with SFP+ connectors of
OCP mezzanine v1 card or RJ45 connector of Intel I210 LOM. The BMC (Baseboard Management Controller) is
used to manage the sled itself which supports both in-band management and out-of-band management through
the sideband control of Ethernet LAN.
Figure 1-2 F06T 2U2N System Overview
1.3 F06 OCP motherboard
F06 OCP motherboard is the one used in F06S / F06T Sleds and it is fully compatible with OCP 2P server board
(Facebook Server Intel motherboard v3.1) powered by Intel E5-2600 v3 / v4 CPU architecture.
The Facebook Server Intel motherboard's form factor is 6.5 x 20 inches.
Figure 1-3: Facebook Server Intel motherboard Placement
(quoted from Facebook Server Intel Next Generation Xeon motherboard v3.1,
http://www.opencompute.org/wiki/Motherboard/SpecsAndDesigns )

F06S/F06T 2U System Contribution
6
The figure below illustrates the functional block diagram of Facebook Server Intel motherboard
Figure 1-4 Facebook Server Intel motherboard functional Block Diagram
(quoted from Facebook Server Intel Next Generation Xeon motherboard v3.1,
http://www.opencompute.org/wiki/Motherboard/SpecsAndDesigns )
The major differences between F06 OCP motherboard and Facebook Server Intel motherboard is as table 1-1
shown.
F06 OCP
motherboard
for F06S 2U4N
F06 OCP
motherboard
for F06T 2U2N
Facebook Server Intel
motherboard
Mini-SAS connectors (J9/J11)
Stuff
Stuff
Empty
AirMax 3x8 connector (J39)
Stuff
Stuff
Empty
Guide Pin (JP17)
Stuff
Stuff
Empty
Power connector (J38)
Stuff
Stuff
Empty , Wiring a Power cable
mSATA connector (J7)
Empty
Empty
Stuff
M.2 connector (J8)
Stuff
Stuff
Empty
Fan connectors(J35,J36)
Empty
Empty
Stuff
Table 1-1 BOM Difference between F06 OCP motherboard and Facebook Server Intel motherboard

F06S/F06T 2U System Contribution
7
2PRODUCT ARCHITECTURE OVERVIEW
2.1 F06S 2U4N System Produt Features
2.1.1 Product Features
F06S 2U4N System’s ingredients and features are shown as follows.
Item
Features
Form Factor
2U Chassis
31.1"x17.48"x3.44"(789.94 x 444 x 87.5mm) [LxW xH]
Baseboard size / Quantity
20”x 6.5”(508 x 165mm), 12 layer, 2.16mm / up to 4 pcs per system
CPU (per node)
(2) Intel® Xeon® processor E5-2600 V3 and V4 family (up to 135W)
Max Processor Wattage
135W, VRD 12.5
QPI Speed
9.6 GT/s, 8.0GT/s, 6.4GT/s
Chipset
Intel® C610 series
Memory
(per node)
(16) RDIMM slots
Up to 256GB (16GB x16) for DDR4 RDIMM
PCIe Expansion Slot
(per node)
(1) PCIe x16 Gen3 riser slot for Full-height, Half-length card per node
(1) PCIe x8 Gen3 OCP mezzanine v1 card slot
Rear IO
(per node)
(1) USB 3.0 port
(1) GbE or 10G BASE-T RJ45 port
Front IO
(per node)
(1) Power Button
(1) USB 2.0 port
Network
(per node)
Intel I210 Single GbE (1 MACs and 1 PHY Integrated)
(Optional) Intel x520 10G / Mellanox CX4-Lx 25G mezzanine card
(Optional) QCT 82599ES 10G/ QCT CX3pro 10G mezzanine card
Storage
(per node)
(6) 2.5" SATA hot-plug drives
(1) M.2 (support 2242 and 2260)
USB
(per node)
(1) USB3.0 port on Rear panel
(1) USB2.0 port on Front panel (Optional)
Series Port (per node)
(1) Serial port on debug card (Optional)
FAN
2U 4 dual-rotor FAN connectors on FAN board
ACPI
ACPI compliance, S0, S5 support. (* S1 and S3 are not supported.)
TPM (per node)
(1) TPM header, support TPM 1.2

F06S/F06T 2U System Contribution
8
Power-Supply
(2) 1600W high efficiency PSU, 220VAC 50/60Hz
Chassis
2U chassis in 19” EIA rack
Table 2-1 F06S 2U4N System Feature List
2.1.2 System Block Diagram
Figure 2-1 F06S 2U4N System Block diagram

F06S/F06T 2U System Contribution
9
2.2 F06T 2U2N System Produt Features
2.2.1 Product Features
F06T 2U2N System’s ingredients and features are shown as follows.
Board Name
F06T 2U2N System
Form Factor
2U Chassis (Carmel Chassis)
31.1"x17.48"x3.44"(789.94 x 444 x 87.5mm) [LxW xH]
Baseboard size / Quantity
20” x 6.5” (508 x 165mm), 12 layer, 2.16mm / 2 pcs per system
CPU (per node)
(2) Intel® Xeon® processor E5-2600 V3 and V4 family (up to 135W)
Max Processor Wattage
135W, VRD 12.5
QPI Speed
9.6 GT/s, 8.0GT/s, 6.4GT/s
Chipset
Intel® C610 series
Memory
(per node)
(16) DDR4 RDIMM slots
Up to 256GB (16GB x16) of memory for RDIMM
PCIe Expansion Slot
(per node)
(3) PCIe x16 Gen3 riser slot for Full-height, Half-length card
(1) PCIe x8 Gen3 OCP mezzanine v1 card slot
Rear IO
(per node)
(1) USB 3.0 port
(1) GbE or 10G BASE-T RJ45 port
Front IO
(per node)
(1) Power Button
(1) USB 2.0 port
Network
(per node)
I210 Single GbE (1 MACs and 1 PHY Integrated)
(Optional) Intel x520 10G / Mellanox CX4-Lx 25G mezzanine card
(Optional) QCT 82599ES 10G / QCT CX3pro 10G mezzanine card
Storage
(per node)
(12) 2.5" SATA hot-plug drives (2U Chassis)
(1) M.2 (support 2242 and 2260)
USB
(per node)
(1) USB3.0 port on Rear
(1) USB2.0 port on Front panel (Optional)
Series Port (per node)
(1) Serial port on debug bard (Optional)
Fan
2U 4 dual-rotor fan connectors on FAN board
ACPI
ACPI compliance, S0, S5 support. (* S1 and S3 are note supported.)
TPM (per node)
(1) TPM header, support TPM specification 1.2
Power-Supply
(2) 1600W high efficiency PSU, 220VAC 50/60Hz
Chassis
2U chassis in 19” EIA rack
Table 2-2 F06T 2U2N System Feature List

F06S/F06T 2U System Contribution
10
2.2.2 System Block Diagram
Figure 2-2 F06T 2U2N System Block diagram

F06S/F06T 2U System Contribution
11
2.3 Related Board Placement
2.3.1 Motherboard Placement
Figure 2-3 F06 OCP motherboard key part placement.
Item
Ref Designator / Silkscreen @PCB
Description
1
J1 / null
Debug port
2
U4 / USB
USB 3.0 connector
3
J6 / MEZZ CONN
OCP Mezzanine v1 connector
4
J12 / TPM
TPM connector
5
U12 / null
AST1250
6
J3 / ETH0
RJ45 for LOM
7
J7 / MSATA CONN
mSATA connector
8
J4B1 / PCI-EX24SLOT
PCIe x24 connector
9
J21,J22,J23,J24 / A7,A6.A5,A4
DIMM slot for CPU0
10
U48 / CPU0
CPU0
11
U64 / CPU1
CPU1
12
J30,J31,J32,J33 / B3,B2,B1,B0
DIMM slot for CPU1
13
J36 / Fan SYS1
FAN connector
14
J39 / B2B CONN
AirMax 3x8 connector
15
J38 / PWR CONN
AirMax Power connector
16
J36 / Fan SYS0
FAN connector
17
J26,J27,J28,J29 / B4,B5,B6,B7
DIMM slot for CPU1
18
J25 / CPU XDP
XDP connector
19
J17,J18,J19,J20 / A0,A1,A2,A3
DIMM slot for CPU0
21
JP9 / HDD_PWR
SATA HDD PWR connector

F06S/F06T 2U System Contribution
12
22
J11 / MINISAS2
Mini-SAS connector
23
J9 / MINISAS1
Mini-SAS connector
24
J5 / SATA
SATA connector
Table 2-2 F06 OCP motherboard key part location list
2.3.2 Interposer Board Placement
Figure 2-4 Interposer board key part placement
Item
Ref Designator / Silkscreen @PCB
Description
1
JP2 / CONN GUIDE
Guide pin for Middle-plane
2
J2 / B2B CONN
B2B connector for Middle-plane
3
J1 / MINISAS HD
Mini-SAS HD connector for motherboard
4
J5 / USB
USB connector for motherboard
5
JP1 / null
Debug I2C header for AST1250 of motherboard
6
U9 / null
CPLD
7
J4 / null
CPLD programming header
8
J6 / null
CPLD reset header
Table 2-3 Interposer board key part location list

F06S/F06T 2U System Contribution
13
2.4 Related Board Dimension
2.4.1 Sled Dimension
F06S Sled related dimension is 537.86mm (L, 21.17 “) x 173.75mm (W, 6.84”) x 40.5mm (H, 1.59”).
Figure 2-5 F06S Sled dimension
F06T Sled related dimension is 537.86mm (L, 21.17 “) x 173.75mm (W, 6.84”) x 81.5mm (H, 3.2”).
Figure 2-6 F06T Sled dimension

F06S/F06T 2U System Contribution
14
2.4.2 Motherboard Dimension
The Motherboard related dimension is 508mm (L, 20 “) x 165mm (W, 6.5”).
Figure 2-7 F06 OCP motherboard dimension
2.4.3 Interposer Board Dimension
The Interposer board related dimension is 162mm (L, 6.38 “) x 60mm (W, 2.37”)
Figure 2-8 Interposer board dimension

F06S/F06T 2U System Contribution
15
3PRODUCT FEATURES
3.1 Processor
The processor of Grantley platform is Xeon E5-2600 v3 and v4 (socket LGA2011 R3), the processor has internal
voltage regulator (IVR). The key features are as shown below:
-Up to 18 cores (E5-2600 v3), 22 cores (E5-2600 v4)
-Up to 145W TDP
-Up to 10 x 4 PCIe Gen3
-Support 4 channels DDR4 RDIMM/LRDIMM (total 16 DIMMs)
-Single Processor mode is supported.
3.2 Memory
F06 OCP motherboard supports total 16 DDR4 DIMMs. Following is memory support tables for E5-2600 v3 and v4
processors.

F06S/F06T 2U System Contribution
16
3.2.1 DIMM Nomenclature
DIMMs are organized into physical slots on DDR4 memory channels that belong to processor sockets. The memory
channels from Socket 0 (CPU-0) are identified as Channel A0~A7. The memory channels from Socket 1 (CPU-1) are
identified as Channel B0~B7.
The DIMM identifiers printed on the motherboard provide information about the channel, and therefore the
processor, to which they belong. For example, DIMM_A0 is the first slot on Channel A of processor 0. DIMM_B0 is
the first DIMM socket on Channel B of processor 1.
Figure 3-1 F06 2U4N motherboard DIMM nomenclature
3.3 PCH
The PCH of Grantley platform is C610. The key features are as shown below:
-Integrated system clocks w/ support for ext. clock buffers
-10 SATA ports capable of 6Gb/s
-Up to 6 ports of USB3; 8 ports of USB2
-Up to 8 x1 PCIe Gen2

F06S/F06T 2U System Contribution
17
3.4 BMC
The Board Management Controller of F06 OCP motherboard is adopting ASPEED AST1250 that is a highly
integrated single-chip solution, integrating several devices typically found on servers.
3.5 Clocks
The Grantley platform has three different clock architectures, external clock architecture (exCLK), integrated
system clock (isCLK) architecture and hybrid architecture. The C610 (PCH) is capable of providing both exCLK and
isCLK. F06 OCP motherboard uses isCLK mode. The external clock generator will not be necessary.
3.6 SATA
The C610 (PCH) supports total 10 SATA-III 6Gbs ports. The PCH contains three SATA controller modes, while IDE,
ACHI and RAID mode.
The project supports 10 SATAIII ports, while one SATA port connects to 7-pin connector, eight SATA ports connect
to two internal mini-SAS connectors for system HDD Back-Plane connection and one SATA port transfer to mSATA
connection.
3.6.1 SATA PORT CONNECTIVITY
Figure 3-2 F06 OCP motherboard SATA connectivity

F06S/F06T 2U System Contribution
18
3.7 USB
The C610 (PCH) supports total 14 USB ports, 6 USB 2.0 ports and 8 USB 3.0 ports. The USB port distribution is as
follows:
-ASPEED BMC AST1250 occupies 2 USB 2.0 ports (one 1.1 and one 2.0-this one is reserved for AST2400,
AST1250 won’t use USB2.0 channel)
-one Rear USB3.0 port is necessary for this project
-one Front-panel USB2.0 port is optional for 2U chassis
The USB ports on the products are not required to be powered from STBY.
Figure 3-3 F06 OCP motherboard USB connectivity

F06S/F06T 2U System Contribution
19
3.8 PCIe BUS
PCI Express* Gen1, Gen2 and Gen 3 are dual-simplex point-to point serial differential low-voltage interconnects.
The signaling bit rate is 2.5 Gbit/s one direction per lane for Gen1 (8b/10b encoding), 5.0 Gb/s one direction per
lane for Gen2 (8b/10b encoding) and 8.0 Gb/s one direction per lane for Gen3 (128b/130b encoding). Each port
consists of a transmitter and receiver pair. A link between the ports of two devices is a collection of lanes (x1, x2,
x4, x8, x16).
3.8.1 PCIe PORT CONNECTIVITY
The following diagram lists the usage of the Intel Xeon E5 v3 and C610 (PCH) PCIe bus segments in F06 2U4N
project.
Figure 3-4 F06 OCP motherboard PCIe connectivity