Western Digital Ultrastar Serv60+8 User manual

User Guide
Ultrastar®Serv60+8
Regulatory Model: H4060-S
December 2018
Rev. 1.3
1ET1110
Long Live Data ™| www.hgst.com

Contents
Revision History..............................................................................................7
Copyright........................................................................................................8
Chapter 1 Overview....................................................................................9
1.1 Ultrastar®Serv60+8 Description........................................................9
1.2 System Architecture Overview.........................................................10
1.3 System Level Block Diagram...........................................................11
1.4 Ultrastar Serv60+8 Specification Summary.....................................12
1.5 Ultrastar Serv60+8 Layout...............................................................15
1.6 List of Customer Replaceable Units (CRUs)....................................17
1.7 Supported Operating Systems.........................................................19
1.8 LEDs................................................................................................20
1.8.1 Front and Rear IO LEDs...............................................................20
1.8.2 IOM LEDs.....................................................................................21
1.8.3 PSU LED......................................................................................22
1.8.4 Drive Assembly LED....................................................................23
1.9 List of Compatible Drives.................................................................24
Chapter 2 System Management Overview.............................................34
2.1 Storage Subsystem Firmware Features Overview...........................34
2.2 OOBM Management Overview........................................................35
2.3 SCSI Enclosure Services Page 02...................................................35
Chapter 3 Component Overviews...........................................................36
3.1 Chassis Description.........................................................................36
3.1.1 Chassis Specifications.................................................................36
3.1.2 Chassis Layout.............................................................................37
3.2 IOM Description...............................................................................38
ii
Contents
User Guide

3.2.1 IOM Specifications.......................................................................38
3.2.2 IOM Layout...................................................................................39
3.2.3 IOM Blank Description.................................................................39
3.3 PSU Description...............................................................................40
3.3.1 PSU Specifications.......................................................................40
3.3.2 PSU Layout..................................................................................41
3.4 System Fan Description...................................................................42
3.5 Rails Description..............................................................................42
3.5.1 Rails Specifications......................................................................43
3.5.2 Rails Layout..................................................................................43
3.5.3 Rear Cover Alignment Bracket Description..................................44
3.6 CMA Description..............................................................................44
3.6.1 CMA Specifications......................................................................45
3.6.2 CMA Layout..................................................................................45
3.7 Drive Assembly Description.............................................................46
3.7.1 Drive Assembly Specifications.....................................................46
3.7.2 Drive Assembly Layout.................................................................47
3.7.3 2.5" Drive Carrier Description.......................................................48
3.7.3.1 Operating the 2.5" Drive Carrier...............................................48
3.7.4 Drive Blank Description................................................................50
3.8 System SSD Description..................................................................52
3.8.1 System SSD Specifications..........................................................52
Chapter 4 Part Replacement...................................................................53
4.1 Part Replacement Service Window.................................................53
4.2 IOM Replacement............................................................................53
4.3 PSU Replacement............................................................................57
4.4 System Fan Replacement................................................................61
4.5 Drive Assembly Replacement..........................................................64
4.6 System SSD Replacement...............................................................69
4.7 DIMM Replacement.........................................................................72
iii
Contents
User Guide

4.8 CPU and Heat Sink Module Replacement.......................................99
4.9 Heat Sink Replacement.................................................................128
4.10 M.2 Replacement.........................................................................161
4.11 Add-in Card Replacement............................................................178
4.12 CMA Replacement.......................................................................196
4.13 Rails Replacement.......................................................................201
4.14 Chassis Replacement..................................................................225
Chapter 5 System Management............................................................277
5.1 Storage Subsystem Firmware Upgrade.........................................277
5.1.1 Downloading Firmware from the Support Portal........................277
5.1.2 Linux Upgrade Preparation........................................................279
5.1.3 Linux Upgrade to New Firmware................................................279
5.1.4 Non-Automatic Firmware Activation in Linux..............................281
5.1.5 Windows Upgrade Preparation...................................................282
5.1.6 Windows Upgrade to New Firmware..........................................283
5.1.7 Non-Automatic Firmware Activation in Windows........................284
5.1.8 Upgrading Firmware with OOBM...............................................285
5.2 BIOS Firmware Update..................................................................286
5.2.1 Updating BIOS Firmware using EFI Boot Mode.........................287
5.2.2 Updating BIOS Firmware using the Legacy Boot Mode.............289
5.3 HGST BMC Web GUI....................................................................291
5.3.1 BMC Web GUI Overview............................................................291
5.3.1.1 Dashboard..............................................................................292
5.3.2 User Management......................................................................293
5.3.2.1 User Privileges.......................................................................294
5.3.2.2 Changing the Default Administrator Credentials....................296
5.3.2.3 Logging into the BMC Web GUI.............................................303
5.3.2.4 Creating a New User..............................................................304
5.3.2.5 Modifying a User....................................................................307
5.3.2.6 Deleting a User.......................................................................310
5.3.2.7 Ending a User Session...........................................................312
iv
Contents
User Guide

5.3.3 Security Features.......................................................................313
5.3.3.1 Enabling Security Settings.....................................................314
5.3.4 Network Settings........................................................................317
5.3.4.1 Configuring the IPv4 Network Settings...................................317
5.3.4.2 Configuring the IPv6 Network Settings...................................320
5.3.4.3 Configuring VLAN Settings.....................................................323
5.3.5 BMC Web GUI Firmware............................................................324
5.3.5.1 Updating BMC Web GUI Firmware........................................324
5.3.5.2 Updating the Server BMC Firmware Using Redfish OOBM....329
5.3.6 Event Logs.................................................................................331
5.3.6.1 Viewing Event Logs................................................................332
5.4 Partially Populated Enclosures.......................................................335
5.4.1 Partial Population Configurations...............................................335
5.4.2 Installing Drives..........................................................................338
Chapter 6 Safety.....................................................................................342
6.1 Electrostatic Discharge..................................................................342
6.2 Optimizing Location........................................................................342
6.3 Power Connections........................................................................342
6.4 Power Cords...................................................................................343
6.5 Rackmountable Systems...............................................................343
6.6 Safety and Service.........................................................................343
6.7 Safety Warnings and Cautions.......................................................344
Chapter 7 Disclaimers............................................................................345
7.1 Restricted Access Location............................................................345
7.2 Safety Compliance.........................................................................345
7.3 Electromagnetic Compatibility (EMC) Class A Compliance ..........345
7.4 Country Certifications....................................................................346
Chapter 8 Regulatory Statements.........................................................347
v
Contents
User Guide

8.1 Europe (CE Declaration of Conformity)..........................................347
8.2 FCC Class A Notice.......................................................................347
8.3 ICES-003 Class A Notice—Avis NMB-003, Classe A....................347
8.4 Japanese Compliance Statement, Class A ITE.............................347
8.5 Taiwan Warning Label Statement, Class A ITE.............................348
Chapter 9 Points of Contact..................................................................349
Appendix A Site and System Operation Requirements............................350
A.1 Power Requirements.....................................................................350
A.2 Supported Operating Systems......................................................350
A.3 List of Supported Server Components..........................................351
A.4 Ultrastar Serv60+8 Rack Requirements........................................352
A.4.1 Compatible Hardware Configuration..........................................354
A.5 ESD...............................................................................................355
A.6 Enclosure Cooling..........................................................................355
A.7 Host Connectivity...........................................................................356
vi
Contents
User Guide

Revision History
CommentRevisionDate
Initial ReleaseRevision 1.0June 2018
Added the following sections:Revision 1.1July 2018
•DIMM Replacement on page 72
•CPU and Heat Sink Module Replacement on page 99
•Heat Sink Replacement on page 128
•M.2 Replacement on page 161
•Add-in Card Replacement on page 178
Updated the following sections:Revision 1.2August 2018
•List of Customer Replaceable Units (CRUs) on page 17
•CPU and Heat Sink Module Replacement on page 99
•Heat Sink Replacement on page 128
•Storage Subsystem Firmware Upgrade on page 277
Updated the following sections:Revision 1.3December2018
•List of Customer Replaceable Units (CRUs) on page 17
•List of Compatible Drives on page 24
•LEDs on page 20
Added the following sections:
•BIOS Firmware Update on page 286
•Updating the Server BMC Firmware Using Redfish
OOBM on page 329
7
Revision History
User Guide

Copyright
The following paragraph does not apply to the United Kingdom or any country where such provisions
are inconsistent with local law: HGST a Western Digital company PROVIDES THIS PUBLICATION "AS
IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR
PURPOSE. Some states do not allow disclaimer or express or implied warranties in certain transactions,
therefore, this statement may not apply to you.
This publication could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. HGST may
make improvements or changes in any products or programs described in this publication at any time.
It is possible that this publication may contain reference to, or information about, HGST products (machines
andprograms),programming,orservicesthat are notannounced in yourcountry. Such referencesorinformation
must not be construed to mean that HGST intends to announce such HGST products, programming, or
services in your country.
Technical information about this product is available by contacting your local HGST representative or on the
Internet at: support.hgst.com
HGST may have patents or pending patent applications covering subject matter in this document. The furnishing
of this document does not give you any license to these patents.
©2017-2018 Western Digital Corporation or its affiliates.
Western Digital
5601 Great Oaks Parkway
San Jose, CA 95119
Long Live Data™is a trademark of HGST, Inc. and its affiliates in the United States and/or other countries.
HGST trademarks are authorized for use in countries and jurisdictions in which HGST has the right to use,
market and advertise the brands.
Other product names are trademarks or registered trademarks of their respective owners.
One MB is equal to one million bytes, one GB is equal to one billion bytes, one TB equals 1,000GB (one trillion
bytes) and one PB equals 1,000TB when referring to storage capacity. Usable capacity will vary from the raw
capacity due to object storage methodologies and other factors.
References in this publication to HGST products, programs or services do not imply that HGST intends to
make these available in all countries in which HGST operates.
Product information is provided for information purposes only and does not constitute a warranty.
Information is true as of the date of publication and is subject to change. Actual results may vary. This publication
is for general guidance only. Photographs may show design models.
8
Copyright
User Guide

Chapter
1
1 Overview
This section provides a high level overview of the features of the Ultrastar Serv60+8.
1.1 Ultrastar®Serv60+8 Description
Figure 1: Ultrastar Serv60+8
The Ultrastar®Serv60+8 is a 4U form factor, high density, rack-mounted storage enclosure that is capable
of hosting up to 60 HDD SAS or SATA drives. The maximum data storage capacity of the Ultrastar Serv60+8
is720TB using 12TB HGST Ultrastar®He12 HDDs plus an additional 61.44 TB using HGST Ultrastar®SS300
SSDs in the system SSD slots. (For a full list of compatible drives and total storage capacities, see the List
of Compatible Drives on page 24.) The Ultrastar Serv60+8 also integrates an Intel Skylake based server
front end. The server is built around an Intel Sawtooth Pass S2600ST motherboard and supports 2 LGA3647-0
(Socket P) processor sockets CPUs, 16 (8 per CPU) slots of DDR4 ECC DIMM memory, and leverages an
AST2500 BMC for out-of-band management of the server subsystem.The enclosure runs on an input voltage
of 200 - 240 VAC and consumes ~1800W of power under typical conditions. It requires a maximum of ~2000W
at full load.
Note: Max and Typical Power Consumption values represent the output power to the system. Input
power will vary depending on the PSU efficiency and load sharing between PSUs
The system contains three externally facing Half-Height, Half-Length (HHHL) PCIe x16 slots, and 3 externally
facing HHHL PCIe x8 slots.
It is designed to fit within a 4U rack space and requires 1200 mm (47.24in.) of usable rack space, frame to
frame. A fully loaded system will add 95.25 kg / 210 lbs. of static load when fully loaded with drives.
• 4U Storage Server
• Supports up to 60 HDD Drives
• Can support 3.5” drives and 2.5” SSD drives (2.5" requires an adapter) in the 60 available drive bays.
Supports an additional 8 2.5" drives in the system SSD slots located in the center channel
• Up to 12W per drive slot for the 60 HDD data storage drives (Cannot exceed 85A on the 5V rail) and 25W
per slot for the 8 system SSD slots
• House and control six (6) N+1 redundant 80mm fans
• Powered by two (2) redundant 2000W PSUs
• Toolless replacement of all Customer Replaceable Units (CRUs)
• Fits within a standard EIA-310 rack including all necessary cable management
• Supports Highline Input Power
9
1 Overview
User Guide 1.1 Ultrastar®Serv60+8 Description

• Dual Skylake LGA3647 Socket Processors
• 16 DDR4 ECC DIMM slots
• ASPEED AST2500 BMC
• Lewisburg PCH C624 Chipset
1.2 System Architecture Overview
Storage Subsystem Overview
The Ultrastar Serv60+8 IOM uses a cascaded expander design to allow for connection to all 60 HDD drives.
The minimum configuration comes with two of the six PCIe slots pre-populated with a combination of internal
SAS HBAs and RAID adapters that act as an interconnect between the server subsystem and the storage
subsystem by connecting to the system SSDs and the IOM. The exact configuration of the PCIe slots may
vary depending the specific version of the Ultrastar Serv60+8. The eight system SSDs are connected via one
of these SAS HBAs.
The storage subsystem out-of-band management microprocessor provides an Ethernet connection using a
Redfish/RESTful API to access the various enclosure services. All the SES enclosure information can be
obtainedthrough the out-of-bandmanagement port.Major use casesforthisfeature includeobtaining storage
subsystemhealth information, locating enclosurecomponents using theIDENT LEDs,and updating expander
and storage OOBM firmware.
The system FPGAs control and report the states of the four storage subsystem fans, drive LEDs, and T10
drive power disable signals on the 60 HDD data storage drives.
Note: To use T10 power disable, the drives installed must also support this
feature.
The I2C architecture is designed to support only one single master on any given bus. The primary expander
will be the master on each I2C bus. There are eight I2C buses used in the Ultrastar Serv60+8 enclosure. The
number of devices on each bus are balanced to allow communication to peripheral devices and not overload
any one bus. The devices connected on the I2C buses include the enclosure VPDs, temp sensors, baseboard
FPGAs, and SAS connectors among others.
Four of the six total fans are controlled by the storage subsystem. Status and control for these four fans is
handled by the in-band SES management interface called the SEP and the Redfish OOBM.
Server Subsystem Overview
The server subsystem is situated in an individual compartment at the rear of the unit. It is comprised of a dual
socket Intel Sawtooth Pass S2600ST that supports LGA3647 socket Intel Xeon processors. The Ultrastar
Serv60+8 has 16 DDR4 ECC DIMM slots for memory that support a clock speed of 2666 MHz, and it leverages
the Intel C624 chipset. The motherboard also hosts an ASPEED AST2500 BMC for out-of-band management
of the server components. The server motherboard has a total of six PCIe slots that support PCIe 3.0, two
M.2 ports that can host compatible SSDs for the system OS.
10
1 Overview
User Guide 1.2 System Architecture Overview

1.3 System Level Block Diagram
The following image shows the system block diagram for the Ultrastar Serv60+8. The block diagram is split
into two, one for the storage subsystem and one for the server subsystem.
Figure 2: System Block Diagram for the Storage Subsystem
Figure 3: System Block Diagram for the Server Subsystem
11
1 Overview
User Guide 1.3 System Level Block Diagram

1.4 Ultrastar Serv60+8 Specification Summary
Table 1: Environmental Specification Summary
OperationalNon-OperationalSpecication
5°C to 35°C-40°C to 70°CTemperature
20°C per hour maximum30°C per hour maximumTemperature Gradient
1°C per 300m above 900m1°C per 300m above 3000mTemperature De-rating
8-90% Non-Condensing8-90% Non-CondensingRelative Humidity
30% per hour maximum30% per hour maximumRelative Humidity Gradient
12
1 Overview
User Guide 1.4 Ultrastar Serv60+8 Specification Summary

OperationalNon-OperationalSpecication
-300m to 3048m / -984 ft. to 10,000 ft.-300m to 12,000m / -984 ft. to 39,370 ftAltitude
Table 2: Electrical Specifications
ValueSpecication
~2000WMax Power Consumption
~1800WTypical Power Consumption
Note: Max and Typical Power Consumption
values represent the output power to the
system. Input power will vary depending on
thePSU efficiency and loadsharing between
PSUs
200 - 240 VACInput Voltage
C14PSU Connector Type
80 PLUS PlatinumPSU Efciency
AC line inrush current shall not exceed 35A peak
measured per IPC-9592B(L) Appendix C-4.1.
Inrush Current Maximum (per PSU)
15ATypical Inrush Current (per PSU)
Caution: The Ultrastar Serv60+8 can only be plugged into highline. If the unit is plugged into lowline,
the PSU will report a "Critical" state when status pages are queried using SES. In this case, the enclosure
will power up, but the drives will not.
Table 3: Mechanical Specifications
OperationalNon-OperationalSpecication
5G, 0 - peak, 11ms half sine; 3
positive and 3 negative pulses in each
10G, 0 - peak,11ms half sine; 3
positive and 3 negative pulses in each
axis shock
Shock
axis- minimum 6 seconds between
shocks to allow for write/read
recovery
0.10G,0 - peak swept sine; 5 -500Hz;
1 complete sweep @ 1/2octave per
minute
0.75G, 0 - peak swept sine; 5 -500Hz;
1 complete sweep @ 1/2 octave per
minute
Vibration
95.25 kg / 210 lbs.Weight
W: 447mm x L: 1197 mm x H: 175mm / W: 17.6in. x L: 47.13in.. x H:
6.89in.
Dimensions with CMA
1026 mm / 40.39in.Length without CMA
13
1 Overview
User Guide 1.4 Ultrastar Serv60+8 Specification Summary

OperationalNon-OperationalSpecication
450mm with (17.72in.) with 465mm (18.31in.) ± 1.5mm nominal hole
spacing. See EIA-310 Rack Standard
Required Rack Width
1200 mm (47.24in.) of usable rack space, frame to frameRequired Rack Depth
4URack Units (U)
32 in. - 36 in.Vertical Rack Rail Spacing
Table 4: Performance Specifications
ValueSpecication
Intel Sawtooth Pass S2600STMother Board
Intel SkylakeProcessor Type
LGA3647-0 (Socket P) processor socketsSocket
2Number of Processors
Lewisburg PCH C624 ChipsetChipset
DDR4 ECC DIMMMemory Type
16 (8 per CPU)Number of Memory Slots
IntegratedGraphics
AST2500Sever BMC Chip
60 HDD / 8 SSDNumber of Drive Slots
12Gbps SAS / 6Gbps SATAData Transfer Rates
720TB using 12TB HGST Ultrastar®He12 HDDs / 61.44
TB using HGST Ultrastar®SS300 SSDs
Max Raw Data Storage Capacity
14
1 Overview
User Guide 1.4 Ultrastar Serv60+8 Specification Summary

1.5 Ultrastar Serv60+8 Layout
Figure 4: Front and Rear Product Layout
Table 5: Front and Rear Component Identification
ComponentNumber
Enclosure Handles1
CMA2
PSUs3
Rear IO Ports4
Chassis Cover5
Rails6
15
1 Overview
User Guide 1.5 Ultrastar Serv60+8 Layout

The following is an image of the layout of the major system components inside the Ultrastar Serv60+8.
Figure 5: Component Layout
Figure 6: Rear IO Ports
16
1 Overview
User Guide 1.5 Ultrastar Serv60+8 Layout

1.6 List of Customer Replaceable Units (CRUs)
The following table lists the replaceable components and their part numbers.
Table 6: List of Replaceable Components
Part NumberComponent
1EX1801Ultrastar Serv60+8 Chassis with Single IOM and PSUs
1EX1828IOM
1EX0431IOM Blank
1EX1171PSU 2000W
1EX1173System Fan
1EX0435Rails
1EX1174CMA
1EX1800Bracket PCIe Cache Vault Mount
1EX1464Heat Sink
1EX1563Intel®Xeon®Silver 4110 CPU w/ Heat Sink
1EX1322Intel®Xeon®Silver 4114 CPU w/ Heat Sink
1EX1323Intel®Xeon®Gold 5118 CPU w/ Heat Sink
1EX1564Intel®Xeon®Gold 5120 CPU w/ Heat Sink
1EX1565Intel®Xeon®Gold 6132 CPU w/ Heat Sink
1EX1566Intel®Xeon®Gold 6138 CPU w/ Heat Sink
1EX1324Intel®Xeon®Gold 6140 CPU w/ Heat Sink
1EX1567Intel®Xeon®Gold 6150 CPU w/ Heat Sink
1EX1568Intel®Xeon®Gold 6152 CPU w/ Heat Sink
1EX1775Intel®Xeon®Platinum 8160 CPU w/ Heat Sink
1EX1325Samsung®64GB LRDIMM (DDR4 4R 2666MHz)
1EX1328Samsung®16GB RDIMM (DDR4 1R 2666MHz)
1EX1142Samsung®32GB RDIMM (DDR4 2R 2666MHz)
1EX1329Samsung®64GB RDIMM (DDR4 4R 2666MHz)
1EX1330Samsung®128GB RDIMM (DDR4 8R 2666MHz)
1EX1332Micron®16GB RDIMM (DDR4 2R 2666MHz)
1EX1335Micron®32GB RDIMM (DDR4 2R 2666MHz)
1EX1337Micron®64GB LRDIMM (DDR4 4R 2666MHz)
1EX1769Intel®I350-T2V2 NIC
17
1 Overview
User Guide 1.6 List of Customer Replaceable Units (CRUs)

Part NumberComponent
1EX1770Intel®I350-T4V2 NIC
1EX1267Intel®X520-DA2 NIC
1EX1772Intel®X520-SR2 NIC
1EX1270Intel®X540-T2 NIC
1EX1268Intel®X710-DA2 NIC
1EX1771Intel®X710-DA4 NIC
1EX1774QLogic QLE2692 NIC
1EX1773QLogic QLE2694 NIC
1EX1161Chelsio T6225-CR NIC
1EX1271Mellanox®ConnectX-4 Lx (MCX4121A-ACAT)
1EX1261Broadcom®9300-8e
1EX1168Broadcom®9300-8i
1EX1355Broadcom®9300-16i
1EX1259Broadcom®9305-16e
1EX1169Broadcom®9305-16i
1EX1263Broadcom®MegaRAID 9361-8i RAID card without Cache
1EX1262Broadcom®MegaRAID 9361-16i RAID card without Cache
1EX1264Broadcom®MegaRAID 9361-24i RAID card without Cache
1EX1470Broadcom®cache protection module kit for 9361-8i
1EX1467Broadcom®cache protection module kit for 9361-16i and 9361-24i
1EX04383.5 in. Drive Carrier, Qty=1
1EX14732.5 in. Drive Carrier for center channel, Qty=1
1EX04293.5 in. Drive Blank, Qty=1
1EX14592.5 in. Drive Blank for center channel, Qty=1
1EX1529Power Cable for PDU, C13-C14, 18AWG, 2m, Qty=1
1EX1531HD Mini-SAS to HD Mini-SAS, 2m, Qty=2
1EX1533HD Mini-SAS to HD Mini-SAS, 3m, Qty=2
18
1 Overview
User Guide 1.6 List of Customer Replaceable Units (CRUs)

1.7 Supported Operating Systems
Table 7: Compatible Operating Systems
Operating System
Microsoft®Windows Server 2012 R2 x64
Microsoft®Windows Server 2016 R1 x64
RedHat®Red Hat Enterprise Linux (x86_64) 7.4
RedHat®Red Hat Enterprise Linux (x86_64) 7.3
RedHat®Red Hat Enterprise Linux (x86_64) 7.2
Ubuntu®Ubuntu Server 16.04 Kernel 4.4
19
1 Overview
User Guide 1.7 Supported Operating Systems

1.8 LEDs
1.8.1 Front and Rear IO LEDs
The Ultrastar Serv60+8 has three LEDs on the front and three on the rear that mirror each other and provide
general status. This allows the status of the enclosure to be visible from either side of the rack.
Figure 7: Front LEDs
12 3
Figure 8: Rear IO LEDs
Table 8: Front and Rear LED Identification
BehaviorColorLED NameNumber
Blink @ 1 Hz – Blinks only when Identication has been
activated.
BlueIdentication1
Blink @ 1 Hz - Enclosure has a fault
Off - Enclosure is functioning properly
AmberFault2
Solid - Powered OnGreenPower3
20
1 Overview
User Guide 1.8 LEDs