Western Digital Ultrastar Serv60+8 User manual

User Guide
Ultrastar®Serv60+8
Regulatory Model: H4060-S
December 2018
Rev. 1.3
1ET1110
Long Live Data ™| www.hgst.com

Contents
Revision History..............................................................................................7
Copyright........................................................................................................8
Chapter 1 Overview....................................................................................9
1.1 Ultrastar®Serv60+8 Description........................................................9
1.2 System Architecture Overview.........................................................10
1.3 System Level Block Diagram...........................................................11
1.4 Ultrastar Serv60+8 Specification Summary.....................................12
1.5 Ultrastar Serv60+8 Layout...............................................................15
1.6 List of Customer Replaceable Units (CRUs)....................................17
1.7 Supported Operating Systems.........................................................19
1.8 LEDs................................................................................................20
1.8.1 Front and Rear IO LEDs...............................................................20
1.8.2 IOM LEDs.....................................................................................21
1.8.3 PSU LED......................................................................................22
1.8.4 Drive Assembly LED....................................................................23
1.9 List of Compatible Drives.................................................................24
Chapter 2 System Management Overview.............................................34
2.1 Storage Subsystem Firmware Features Overview...........................34
2.2 OOBM Management Overview........................................................35
2.3 SCSI Enclosure Services Page 02...................................................35
Chapter 3 Component Overviews...........................................................36
3.1 Chassis Description.........................................................................36
3.1.1 Chassis Specifications.................................................................36
3.1.2 Chassis Layout.............................................................................37
3.2 IOM Description...............................................................................38
ii
Contents
User Guide

3.2.1 IOM Specifications.......................................................................38
3.2.2 IOM Layout...................................................................................39
3.2.3 IOM Blank Description.................................................................39
3.3 PSU Description...............................................................................40
3.3.1 PSU Specifications.......................................................................40
3.3.2 PSU Layout..................................................................................41
3.4 System Fan Description...................................................................42
3.5 Rails Description..............................................................................42
3.5.1 Rails Specifications......................................................................43
3.5.2 Rails Layout..................................................................................43
3.5.3 Rear Cover Alignment Bracket Description..................................44
3.6 CMA Description..............................................................................44
3.6.1 CMA Specifications......................................................................45
3.6.2 CMA Layout..................................................................................45
3.7 Drive Assembly Description.............................................................46
3.7.1 Drive Assembly Specifications.....................................................46
3.7.2 Drive Assembly Layout.................................................................47
3.7.3 2.5" Drive Carrier Description.......................................................48
3.7.3.1 Operating the 2.5" Drive Carrier...............................................48
3.7.4 Drive Blank Description................................................................50
3.8 System SSD Description..................................................................52
3.8.1 System SSD Specifications..........................................................52
Chapter 4 Part Replacement...................................................................53
4.1 Part Replacement Service Window.................................................53
4.2 IOM Replacement............................................................................53
4.3 PSU Replacement............................................................................57
4.4 System Fan Replacement................................................................61
4.5 Drive Assembly Replacement..........................................................64
4.6 System SSD Replacement...............................................................69
4.7 DIMM Replacement.........................................................................72
iii
Contents
User Guide

4.8 CPU and Heat Sink Module Replacement.......................................99
4.9 Heat Sink Replacement.................................................................128
4.10 M.2 Replacement.........................................................................161
4.11 Add-in Card Replacement............................................................178
4.12 CMA Replacement.......................................................................196
4.13 Rails Replacement.......................................................................201
4.14 Chassis Replacement..................................................................225
Chapter 5 System Management............................................................277
5.1 Storage Subsystem Firmware Upgrade.........................................277
5.1.1 Downloading Firmware from the Support Portal........................277
5.1.2 Linux Upgrade Preparation........................................................279
5.1.3 Linux Upgrade to New Firmware................................................279
5.1.4 Non-Automatic Firmware Activation in Linux..............................281
5.1.5 Windows Upgrade Preparation...................................................282
5.1.6 Windows Upgrade to New Firmware..........................................283
5.1.7 Non-Automatic Firmware Activation in Windows........................284
5.1.8 Upgrading Firmware with OOBM...............................................285
5.2 BIOS Firmware Update..................................................................286
5.2.1 Updating BIOS Firmware using EFI Boot Mode.........................287
5.2.2 Updating BIOS Firmware using the Legacy Boot Mode.............289
5.3 HGST BMC Web GUI....................................................................291
5.3.1 BMC Web GUI Overview............................................................291
5.3.1.1 Dashboard..............................................................................292
5.3.2 User Management......................................................................293
5.3.2.1 User Privileges.......................................................................294
5.3.2.2 Changing the Default Administrator Credentials....................296
5.3.2.3 Logging into the BMC Web GUI.............................................303
5.3.2.4 Creating a New User..............................................................304
5.3.2.5 Modifying a User....................................................................307
5.3.2.6 Deleting a User.......................................................................310
5.3.2.7 Ending a User Session...........................................................312
iv
Contents
User Guide

5.3.3 Security Features.......................................................................313
5.3.3.1 Enabling Security Settings.....................................................314
5.3.4 Network Settings........................................................................317
5.3.4.1 Configuring the IPv4 Network Settings...................................317
5.3.4.2 Configuring the IPv6 Network Settings...................................320
5.3.4.3 Configuring VLAN Settings.....................................................323
5.3.5 BMC Web GUI Firmware............................................................324
5.3.5.1 Updating BMC Web GUI Firmware........................................324
5.3.5.2 Updating the Server BMC Firmware Using Redfish OOBM....329
5.3.6 Event Logs.................................................................................331
5.3.6.1 Viewing Event Logs................................................................332
5.4 Partially Populated Enclosures.......................................................335
5.4.1 Partial Population Configurations...............................................335
5.4.2 Installing Drives..........................................................................338
Chapter 6 Safety.....................................................................................342
6.1 Electrostatic Discharge..................................................................342
6.2 Optimizing Location........................................................................342
6.3 Power Connections........................................................................342
6.4 Power Cords...................................................................................343
6.5 Rackmountable Systems...............................................................343
6.6 Safety and Service.........................................................................343
6.7 Safety Warnings and Cautions.......................................................344
Chapter 7 Disclaimers............................................................................345
7.1 Restricted Access Location............................................................345
7.2 Safety Compliance.........................................................................345
7.3 Electromagnetic Compatibility (EMC) Class A Compliance ..........345
7.4 Country Certifications....................................................................346
Chapter 8 Regulatory Statements.........................................................347
v
Contents
User Guide

8.1 Europe (CE Declaration of Conformity)..........................................347
8.2 FCC Class A Notice.......................................................................347
8.3 ICES-003 Class A Notice—Avis NMB-003, Classe A....................347
8.4 Japanese Compliance Statement, Class A ITE.............................347
8.5 Taiwan Warning Label Statement, Class A ITE.............................348
Chapter 9 Points of Contact..................................................................349
Appendix A Site and System Operation Requirements............................350
A.1 Power Requirements.....................................................................350
A.2 Supported Operating Systems......................................................350
A.3 List of Supported Server Components..........................................351
A.4 Ultrastar Serv60+8 Rack Requirements........................................352
A.4.1 Compatible Hardware Configuration..........................................354
A.5 ESD...............................................................................................355
A.6 Enclosure Cooling..........................................................................355
A.7 Host Connectivity...........................................................................356
vi
Contents
User Guide

Revision History
CommentRevisionDate
Initial ReleaseRevision 1.0June 2018
Added the following sections:Revision 1.1July 2018
•DIMM Replacement on page 72
•CPU and Heat Sink Module Replacement on page 99
•Heat Sink Replacement on page 128
•M.2 Replacement on page 161
•Add-in Card Replacement on page 178
Updated the following sections:Revision 1.2August 2018
•List of Customer Replaceable Units (CRUs) on page 17
•CPU and Heat Sink Module Replacement on page 99
•Heat Sink Replacement on page 128
•Storage Subsystem Firmware Upgrade on page 277
Updated the following sections:Revision 1.3December2018
•List of Customer Replaceable Units (CRUs) on page 17
•List of Compatible Drives on page 24
•LEDs on page 20
Added the following sections:
•BIOS Firmware Update on page 286
•Updating the Server BMC Firmware Using Redfish
OOBM on page 329
7
Revision History
User Guide

Copyright
The following paragraph does not apply to the United Kingdom or any country where such provisions
are inconsistent with local law: HGST a Western Digital company PROVIDES THIS PUBLICATION "AS
IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR
PURPOSE. Some states do not allow disclaimer or express or implied warranties in certain transactions,
therefore, this statement may not apply to you.
This publication could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. HGST may
make improvements or changes in any products or programs described in this publication at any time.
It is possible that this publication may contain reference to, or information about, HGST products (machines
andprograms),programming,orservicesthat are notannounced in yourcountry. Such referencesorinformation
must not be construed to mean that HGST intends to announce such HGST products, programming, or
services in your country.
Technical information about this product is available by contacting your local HGST representative or on the
Internet at: support.hgst.com
HGST may have patents or pending patent applications covering subject matter in this document. The furnishing
of this document does not give you any license to these patents.
©2017-2018 Western Digital Corporation or its affiliates.
Western Digital
5601 Great Oaks Parkway
San Jose, CA 95119
Long Live Data™is a trademark of HGST, Inc. and its affiliates in the United States and/or other countries.
HGST trademarks are authorized for use in countries and jurisdictions in which HGST has the right to use,
market and advertise the brands.
Other product names are trademarks or registered trademarks of their respective owners.
One MB is equal to one million bytes, one GB is equal to one billion bytes, one TB equals 1,000GB (one trillion
bytes) and one PB equals 1,000TB when referring to storage capacity. Usable capacity will vary from the raw
capacity due to object storage methodologies and other factors.
References in this publication to HGST products, programs or services do not imply that HGST intends to
make these available in all countries in which HGST operates.
Product information is provided for information purposes only and does not constitute a warranty.
Information is true as of the date of publication and is subject to change. Actual results may vary. This publication
is for general guidance only. Photographs may show design models.
8
Copyright
User Guide

Chapter
1
1 Overview
This section provides a high level overview of the features of the Ultrastar Serv60+8.
1.1 Ultrastar®Serv60+8 Description
Figure 1: Ultrastar Serv60+8
The Ultrastar®Serv60+8 is a 4U form factor, high density, rack-mounted storage enclosure that is capable
of hosting up to 60 HDD SAS or SATA drives. The maximum data storage capacity of the Ultrastar Serv60+8
is720TB using 12TB HGST Ultrastar®He12 HDDs plus an additional 61.44 TB using HGST Ultrastar®SS300
SSDs in the system SSD slots. (For a full list of compatible drives and total storage capacities, see the List
of Compatible Drives on page 24.) The Ultrastar Serv60+8 also integrates an Intel Skylake based server
front end. The server is built around an Intel Sawtooth Pass S2600ST motherboard and supports 2 LGA3647-0
(Socket P) processor sockets CPUs, 16 (8 per CPU) slots of DDR4 ECC DIMM memory, and leverages an
AST2500 BMC for out-of-band management of the server subsystem.The enclosure runs on an input voltage
of 200 - 240 VAC and consumes ~1800W of power under typical conditions. It requires a maximum of ~2000W
at full load.
Note: Max and Typical Power Consumption values represent the output power to the system. Input
power will vary depending on the PSU efficiency and load sharing between PSUs
The system contains three externally facing Half-Height, Half-Length (HHHL) PCIe x16 slots, and 3 externally
facing HHHL PCIe x8 slots.
It is designed to fit within a 4U rack space and requires 1200 mm (47.24in.) of usable rack space, frame to
frame. A fully loaded system will add 95.25 kg / 210 lbs. of static load when fully loaded with drives.
• 4U Storage Server
• Supports up to 60 HDD Drives
• Can support 3.5” drives and 2.5” SSD drives (2.5" requires an adapter) in the 60 available drive bays.
Supports an additional 8 2.5" drives in the system SSD slots located in the center channel
• Up to 12W per drive slot for the 60 HDD data storage drives (Cannot exceed 85A on the 5V rail) and 25W
per slot for the 8 system SSD slots
• House and control six (6) N+1 redundant 80mm fans
• Powered by two (2) redundant 2000W PSUs
• Toolless replacement of all Customer Replaceable Units (CRUs)
• Fits within a standard EIA-310 rack including all necessary cable management
• Supports Highline Input Power
9
1 Overview
User Guide 1.1 Ultrastar®Serv60+8 Description

• Dual Skylake LGA3647 Socket Processors
• 16 DDR4 ECC DIMM slots
• ASPEED AST2500 BMC
• Lewisburg PCH C624 Chipset
1.2 System Architecture Overview
Storage Subsystem Overview
The Ultrastar Serv60+8 IOM uses a cascaded expander design to allow for connection to all 60 HDD drives.
The minimum configuration comes with two of the six PCIe slots pre-populated with a combination of internal
SAS HBAs and RAID adapters that act as an interconnect between the server subsystem and the storage
subsystem by connecting to the system SSDs and the IOM. The exact configuration of the PCIe slots may
vary depending the specific version of the Ultrastar Serv60+8. The eight system SSDs are connected via one
of these SAS HBAs.
The storage subsystem out-of-band management microprocessor provides an Ethernet connection using a
Redfish/RESTful API to access the various enclosure services. All the SES enclosure information can be
obtainedthrough the out-of-bandmanagement port.Major use casesforthisfeature includeobtaining storage
subsystemhealth information, locating enclosurecomponents using theIDENT LEDs,and updating expander
and storage OOBM firmware.
The system FPGAs control and report the states of the four storage subsystem fans, drive LEDs, and T10
drive power disable signals on the 60 HDD data storage drives.
Note: To use T10 power disable, the drives installed must also support this
feature.
The I2C architecture is designed to support only one single master on any given bus. The primary expander
will be the master on each I2C bus. There are eight I2C buses used in the Ultrastar Serv60+8 enclosure. The
number of devices on each bus are balanced to allow communication to peripheral devices and not overload
any one bus. The devices connected on the I2C buses include the enclosure VPDs, temp sensors, baseboard
FPGAs, and SAS connectors among others.
Four of the six total fans are controlled by the storage subsystem. Status and control for these four fans is
handled by the in-band SES management interface called the SEP and the Redfish OOBM.
Server Subsystem Overview
The server subsystem is situated in an individual compartment at the rear of the unit. It is comprised of a dual
socket Intel Sawtooth Pass S2600ST that supports LGA3647 socket Intel Xeon processors. The Ultrastar
Serv60+8 has 16 DDR4 ECC DIMM slots for memory that support a clock speed of 2666 MHz, and it leverages
the Intel C624 chipset. The motherboard also hosts an ASPEED AST2500 BMC for out-of-band management
of the server components. The server motherboard has a total of six PCIe slots that support PCIe 3.0, two
M.2 ports that can host compatible SSDs for the system OS.
10
1 Overview
User Guide 1.2 System Architecture Overview