Western Digital Ultrastar DC ME200 User manual

Ultrastar®DC ME200 Memory Extension Drive
PRODUCT MANUAL

- 2 -
Revision History
Revision Date Description Reference
1.0 January 10, 2019 Initial release
2.0 March 22, 2019 Added best practices for benchmarking and
enhanced troubleshooting.
3.0 May 07, 2019 Update to links within Sections 2.3, 2.4, and 8.1.

- 3 -
Western Digital Technologies, Inc. or its affiliates' (collectively “Western Digital”) general policy does
not recommend the use of its products in life support applications where in a failure or malfunction of
the product may directly threaten life or injury. Without limitation to the foregoing, Western Digital
shall not be liable for any loss, injury, or damage caused by use in such conditions.
This document is for information use only and is subject to change without prior notice. Western
Digital assumes no responsibility for any errors that may appear in this document, nor for incidental or
consequential damages resulting from the furnishing, performance or use of this material.
Absent a written agreement signed by Western Digital or its authorized representative to the
contrary, Western Digital explicitly disclaims any express and implied warranties and indemnities of
any kind that may, or could, be associated with this document and related material, and any user of
this document or related material agrees to such disclaimer as a precondition to receipt and usage
hereof.
Each user of this document or any product referred to herein expressly waives all guaranties and
warranties of any kind associated with this document any related materials or such product, whether
expressed or implied, including without limitation, any implied warranty of merchantability or fitness
for a particular purpose or non-infringement. Each user of this document or any product referred to
herein also expressly agrees Western Digital shall not be liable for any incidental, punitive, indirect,
special, or consequential damages, including without limitation physical injury or death, property
damage, lost data, loss of profits or costs of procurement of substitute goods, technology, or
services, arising out of or related to this document, any related materials or any product referred to
herein, regardless of whether such damages are based on tort, warranty, contract, or any other legal
theory, even if advised of the possibility of such damages.
This document and its contents, including diagrams, schematics, methodology, work product, and
intellectual property rights described in, associated with, or implied by this document, are the sole
and exclusive property of Western Digital. No intellectual property license, express or implied, is
granted by Western Digital associated with the document recipient's receipt, access and/or use of
this document or the products referred to herein; Western Digital retains all rights hereto.
Western Digital, the Western Digital logo, and Ultrastar are registered trademarks or trademarks of
Western Digital Corporation or its affiliates in the US and/or other countries. Apache, Apache Ignite,
and Apache Spark are either registered trademarks or trademarks of the Apache Software Foundation
in the United States and/or other countries. Docker is a trademark or registered trademark of Docker,
Inc. in the United States and/or other countries. Intel and Xeon are trademarks of Intel Corporation or
its subsidiaries in the U.S. and/or other countries. Linux® is the registered trademark of Linus Torvalds
in the U.S. and other countries. Oracle and MySQL are registered trademarks of Oracle and/or its
affiliates. Red Hat and Red Hat Enterprise Linux are registered trademarks of Red Hat, Inc. in the U.S.
and other countries. Redis is a trademark of Redis Labs Ltd. Any rights therein are reserved to Redis
Labs Ltd. Any use by Western Digital is for referential purposes only and does not indicate any
sponsorship, endorsement or affiliation between Redis and Western Digital. SAP and HANA are the
trademark(s) or registered trademark(s) of SAP SE in Germany and in several other countries. ScaleMP
and the ScaleMP logo are trademarks of ScaleMP, Inc. All other marks are the property of their
respective owners. Product specifications subject to change without notice. Pictures shown may
vary from actual products. Not all products are available in all regions of the world.
© 2019 Western Digital Corporation or its affiliates. All rights reserved.

- 4 -
Table of Contents
TABLE OF CONTENTS
1. INTRODUCTION ................................................................................................................. 7
2. INSTALLING ULTRASTAR MEMORY DRIVE............................................................................. 8
2.1 Operating System Prerequisites .................................................................................................... 8
2.2 Preloaded Software ........................................................................................................................ 8
2.3 Installing Ultrastar Memory Tools .................................................................................................. 9
2.4 Update Ultrastar Memory Software..............................................................................................10
2.5 BIOS Configuration..........................................................................................................................11
2.5.1 Configure Boot Device............................................................................................................11
2.5.2 Configuring BIOS to Support Intel® Virtualization Technology......................................... 12
3. ULTRASTAR MEMORY DRIVE BOOT, SETTINGS AND DIAGNOSTICS MENU..............................14
3.1 Boot Process ................................................................................................................................... 14
3.2 Guest OS ......................................................................................................................................... 16
3.3 System Settings and Diagnostics .................................................................................................18
4. HARDWARE CONFIGURATION ........................................................................................... 27
4.1 Recommended Configuration.......................................................................................................27
4.2 DRAM to Ultrastar Memory Drive Ratio ........................................................................................27
5. WORKLOADS PERFORMANCE & OPTIMIZATIONS ................................................................30
5.1 Target Workloads for Ultrastar Memory ..................................................................................... 30
5.1.1 Workloads That Do Not Benefit From Ultrastar Memory Drive........................................... 31
5.2 Optimized Workload Settings....................................................................................................... 31
5.2.1 Operating System................................................................................................................... 31
5.2.2 Memory Settings and Memory Allocators........................................................................... 32
5.2.3 Application Settings.............................................................................................................. 32
6. BENCHMARKING METHODOLOGY AND PERFORMANCE COLLECTION .................................... 33
6.1 Benchmarking Methodology .........................................................................................................33
6.2 Expected Results and Demonstrated Value ............................................................................... 34
6.3 Performance Data Collection ....................................................................................................... 34
6.4 Out-of-the-Box Large Memory Benchmark ................................................................................. 36
7. TROUBLESHOOTING ......................................................................................................... 37
7.1 Ultrastar Memory Drive Fails to Start............................................................................................37
7.1.1 Common Installation Error Codes..........................................................................................37
7.2 Collecting Information for Escalating to Support ...................................................................... 38
8. RELOAD THE ULTRASTAR MEMORY DRIVE SOFTWARE........................................................ 39
8.1 Reload to Ultrastar Memory Device............................................................................................. 39
8.2 Installing Ultrastar Memory Drive Software to a Different Bootable Media.............................. 41
9. SPECIFICATIONS .............................................................................................................. 43

- 5 -
List of Figures
LIST OF FIGURES
Figure 1-1 Block Diagram of the Ultrastar Memory Drive.............................................................................7
Figure 2-1 Preloaded Software ..................................................................................................................... 8
Figure 2-2 Installing Ultrastar Memory Tools ............................................................................................. 9
Figure 2-3 Change Boot Order / Priorities..................................................................................................11
Figure 2-4 Intel Virtualization Technology Configuration ......................................................................... 12
Figure 2-5 Intel Virtualization Technology for Direct I/O Configuration................................................. 13
Figure 3-1 First Screen ................................................................................................................................. 14
Figure 3-2 Ultrastar Memory Drive Boot.................................................................................................... 14
Figure 3-3 Configuration Display................................................................................................................ 15
Figure 3-4 Guest OS Loads ......................................................................................................................... 15
Figure 3-5 Performance Data Collection .................................................................................................... 16
Figure 3-6 Software Installer........................................................................................................................ 16
Figure 3-7 Software Installer ........................................................................................................................ 17
Figure 3-8 Systems Settings and Diagnostics (1) ...................................................................................... 18
Figure 3-9 Systems Settings and Diagnostics (2) ...................................................................................... 19
Figure 3-10 Systems Settings and Diagnostics (3).................................................................................... 20
Figure 3-11 Unify Remote Memory Domains ............................................................................................... 21
Figure 3-12 Expand Local Memory Domains............................................................................................... 21
Figure 3-13 Hardware Tab ........................................................................................................................... 22
Figure 3-14 Boot Tab.................................................................................................................................... 23
Figure 3-15 Diagnostics Tab........................................................................................................................ 24
Figure 3-16 Review Diagnostic Test ........................................................................................................... 25
Figure 3-17 Diagnostics Results .................................................................................................................. 25
Figure 3-18 TFTP Option.............................................................................................................................. 26
Figure 5-1 Performance Data Collection .................................................................................................... 32
Figure 5-2 Performance Data Collection ................................................................................................... 32
Figure 6-1 Figure 1 Type .............................................................................................................................. 34
Figure 6-2 Performance Data Collection ................................................................................................... 35
Figure 6-3 Performance Data Collection ................................................................................................... 35
Figure 6-4 Figure 1 Type.............................................................................................................................. 36
Figure 8-1 Software Installer ...................................................................................................................... 40
Figure 8-2 Software Installer ...................................................................................................................... 42

- 6 -
List of Tables
LIST OF TABLES
Table 4-1 Performance-optimized Software-defined Memory (SDM) Capacity ..................................... 28
Table 4-2 Maximum Software-defined Memory (SDM) capacity for Ultrastar Memory ......................... 29
Table 5-1 Workloads and Applications ...................................................................................................... 30
Table 7-1 Common Installation Error Codes...............................................................................................37
Table 7-2 Workloads and Applications...................................................................................................... 38
Table 9-1 Specifications............................................................................................................................... 43

Introduction
- 7 -
1.0 INTRODUCTION
The Ultrastar®DC ME200 Memory Extension Drive can be used to immediately scale existing
system memory, promote server consolidation, and reduce the complexity of splitting large
multi-TB data sets across multiple servers. The Ultrastar memory drive drives applications
with large amounts of system memory at a fraction of the cost of DRAM by combining
Ultrastar NVMe™ SSDs with Western Digital memory extension technology. Advanced
software algorithms work to maintain DRAM-like performance across a variety of applications,
especially targeting highly parallel workloads with high numbers of transactions. Once
installed, the solution is transparent, requiring no changes to the existing OS and applications.
This document provides details for users and administrators on the design, installation,
features, and specifications of Ultrastar memory drives.
As shown in Figure 1-1, the Ultrastar memory drive executes directly on the hardware, below
the operating system, and creates a hybrid system memory from DRAM and NVM. This hybrid
system memory operates transparently as volatile system memory, as if it were all DRAM.
Figure 1-1. Block Diagram of the Ultrastar Memory Drive

Installing Ultrastar Memory Drive
- 8 -
2.0 INSTALLING ULTRASTAR MEMORY DRIVE
2.1 Operating System Prerequisites
Linux®x86 64 bit, kernel versions 2.6.32 or higher.
See Release Notes for supported Linux distributions.
Ultrastar memory drives also supports Open Source hypervisors such as KVM, as
shipped with the major Linux distributions.
2.2 Preloaded Software
The Ultrastar memory drive is a bootable device, shipped with preloaded software to
create software-defined memory. Booting a system directly from the Ultrastar
memory drive requires the system to support a UEFI boot.
To setup the system from a different local bootable flash media, or if the system does
not support UEFI boot, refer to Chapter 8 - Reload the Ultrastar memory Software.
If the preloaded software on the Ultrastar memory drive is accidentally deleted or
corrupted, refer to Chapter 8 - Reload the Ultrastar Memory Software.
Figure 2-1. Preloaded Software
Installing the Ultrastar memory drive is as simple as physically connecting the
Ultrastar memory device to the right slot within the system (refer to the system’s
manual for information on connecting NVMe devices to it).
For optimal performance, it is highly recommended to install at least one
Ultrastar memory drive per processor.
Configure the system’s BIOS to boot from one of the Ultrastar memory
devices installed. Once setup is complete and changes, reboot the system.
NOTE: It is recommended to install a supported Linux OS; if Linux OS is pre-
installed, continue with the boot process. Once prompted, select the device onto
which the OS is installed as the boot device.
While it is recommended to have Linux OS pre-installed before installation, users
may opt to install Linux OS before or after installing the Ultrastar memory drive.

Installing Ultrastar Memory Drive
- 9 -
2.3 Installing Ultrastar Memory Tools
If installing a system for purposes other than production use, such as software
development, performance tuning, or quality assurance, it is recommended for users
to also install the Ultrastar memory tools as it can provide valuable system
information.
The software installer can be obtained from the download section of the product
support page:
https://www.westerndigital.com/support/wdc/data-center-drives/ssd/ultrastar-dc-me200
Figure 2-2. Installing Ultrastar Memory Tools
NOTE: The Ultrastar memory tools are installed by default within /usr/local/
{bin,etc}.
#./wmem_installer-8.6.2535.77.sh in -t
Preparing installer...
Please take a moment to read the below
================================================================================
Welcome to WDC Ultrastar DC ME200 memory version 8.6.2535.77 !
--------------------------------------------------------------------------------
* By installing and/or using this software you acknowledge that you have read and
agreed to the agreement published at
https://www.get-wmen.com/EULA
* Parts if this program, as provided in binary form, include open source code under
one or more open source licenses. For further details and notices please see
https://www.scalemp.com/opensource.
================================================================================
Type “accept” to accept the terms and conditions above: accept
Terms and conditions accepted.
Installing WDC Ultrastar DC ME200 memory tools...
DC Ultrastar DC ME200 memory tools requires SUID permission to run as
non-root user.
Allow WDC Ultrastar DC ME200 memory tools to be run by all users (recommended)?
[N/Y] Y
WDC Ultrastar DC ME200 memory tools installation is complete.

Installing Ultrastar Memory Drive
- 10 -
2.4 Update Ultrastar Memory Software
A new software version of the Ultrastar memory drive is released periodically, which
can include bug fixes, new features, and performance enhancements.
The latest software revision is available at:
https://www.westerndigital.com/support/wdc/data-center-drives/ssd/ultrastar-dc-me200
To update to the latest software version, proceed with the following:
1. Before starting the update, verify the Ultrastar memory drive is operating and
supporting the installed OS.
2. Log into the system as root.
3. Download the latest software version to a directory on the target Linux
server.
4. Make the installer file executable:
5. Launch the installer with the update flag, and follow on-screen instructions.
6. Press the U key to start the update process.
7. Reboot the system.
The new version of the Ultrastar memory software will load and made available upon
reboot of the system.
See Section 2.3 - Installing Ultrastar Memory Tools, for more information.
# chmod +x wmem_installer-x.x.x.x.sh
# ./wmem_installer-x.x.x.x.sh up
Please take a moment to read the following:
================================================================================
...
Writing part 1 out of 7: 100%
Writing part 2 out of 7: 100%
Writing part 3 out of 7: 100%
Writing part 4 out of 7: 100%
Writing part 5 out of 7: 100%
Writing part 6 out of 7: 100%
Writing part 7 out of 7: 100%
WDC Ultrastar DC ME200 memory installation is complete.
Installing WDC Ultrastar DC ME200 memory tools...
WDC Ultrastar DC ME200 memory tools requires SUID permission to run as non-root
user.
Allow Ultrastar DC ME200 memory tools to be run by all users (recommended)
[N/Y]? Y

Installing Ultrastar Memory Drive
- 11 -
2.5 BIOS Configuration
The BIOS menus of the servers onto which they are installed may differ from the ones
found below. Please verify the configuration of boot options in your systems’ BIOS
manual.
2.5.1 Configure Boot Device
1. Boot the machine and access the BIOS Setup Utility during system load.
2. Navigate via the Menu to (Advanced) Boot Options:
If the system is capable of booting directly from the NVMe (using UEFI), set
the Boot mode to UEFI.(See section 8.2 - Installing Ultrastar Memory Drive
Software to a Different Bootable Media).
In case the option of booting from the NVMe is not available, refer to the
Chapter 8 - Reload the Ultrastar Memory Software of this publication.
3. Save changes in BIOS configuration and reboot the system.
4. Re-enter the BIOS setup utility.
5. Navigate via the menu to Boot Options, then select Change Boot Order/
Priorities.
If the system is capable of booting directly from the NVMe (using UEFI), assign
to the UEFI Device and assign as the first/highest boot sequence priority.
In case the option of booting from the NVMe is not available, assign to the
USB Flash Drive highest boot sequence priority by moving it to the first
position on the list.
6. Select menu option Save & Exit and press Enter.
Figure 2-3. Change Boot Order / Priorities
NOTE: To enable boot device selection, save the BIOS
configuration and reboot the system.

Installing Ultrastar Memory Drive
- 12 -
2.5.2 Configuring BIOS to Support Intel®Virtualization Technology
Note, in most systems this feature is a default setting.
To enable Intel Virtualization Technology (VT), navigate via the menu and
proceed with the following steps:
1. Select Advanced, then select Socket Configuration.
2. Click on Processor Configuration, and set Intel Virtualization Technology
to Enabled.
Figure 2-4. Intel Virtualization Technology Configuration

Installing Ultrastar Memory Drive
- 13 -
To enable VT-D, navigate via the menu and proceed with the following steps:
1. Select Advanced, then select Socket Configuration.
2. Click on Integrated IO Configuration, and set Intel Virtualization Technology
for Direct I/O to Enabled.
Figure 2-5. Intel Virtualization Technology for Direct I/O Configuration

Ultrastar Memory Drive Boot, Settings and Diagnostics Menu
- 14 -
3.0 ULTRASTAR MEMORY DRIVE BOOT, SETTINGS AND DIAGNOSTICS
MENU
This chapter outlines the boot process of Ultrastar memory drives, as well as the
interactive user setting and diagnostics options available during the boot process.
It is important to note that for data centers in which KVM console access is not
available, the Ultrastar memory drive can be configured for those outputs and
controls to be available via Serial over LAN (SOL).
3.1 Boot Process
The following screens are displayed during the loading process, directly after the
Ultrastar memory drive system boot.
First visual the end-user sees after BIOS POST.
Figure 3-1. First Screen
Ultrastar memory drive boot start.
Figure 3-2. Ultrastar Memory Drive Boot

Ultrastar Memory Drive Boot, Settings and Diagnostics Menu
- 15 -
At this stage, the system displays configuration information. This specific example
shows:
1 x 2 Processor System (“Board”) with 768GiB RAM (of which 728GiB are
available for use).
2 x NVMe SSDs (“Boards”) with 1,788GiB each (of which 1,044GiB each are
available for use).
The system reports a total of 2,816GiB available for use.
Warning messages, if any (displayed in yellow font), notify the user about
deviations from best practices and recommendations.
A prompt is displayed to optionally enter the Ultrastar memory drive
settings and diagnostics menu (displayed in yellow font) by pressing <F5>
key. If no action is taken, after timeout, the Linux OS will be booted from
the device defined in the Ultrastar memory drive configuration.
Figure 3-3. Configuration Display
Guest OS Loads
Figure 3-4. Guest OS Loads

Ultrastar Memory Drive Boot, Settings and Diagnostics Menu
- 16 -
3.2 Guest OS
Once the guest operating system loads, it is possible to see all available system
memory using standard free command:
Figure 3-5. Performance Data Collection
Additionally, to check the Ultrastar memory drive version and configuration, users
may see the wmemversion command (Note: this requires installation of Ultrastar
memory tools; see Chapter 2 - Installing Ultrastar Memory Drive Tools):
Figure 3-6. Software Installer
Supported until: Unlimited this specific example shows the resources available for
use:
1 x 2 Processor System (“Board”) with 768GB RAM
(of which 728GB are available for use)
2 x NVMe SSDs (“Boards”) with 1,788GiB each
(of which 1,044GiB each are available for use).
The system reports total 2,816GB available for use.
NVMe® SMART attributes are accessible in-band using the command below. SMART
attributes are also accessible out-of-band using IPMI.
# free -g
total used free shared buff/cache available
Mem: 2771 21 2749 0 0 2746
# wmemversion --long
WDC Ultrastar DC ME200 memory: 8.6.2535.77 (Sep 20 2018 20:26:30)
System configuration:
Boards: 3
1 x Proc. + I/O + memory
2 x NVM devices (HGST HUSMR7616BDP3M1)
Processors: 2, Cores: 36, Threads: 72
Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz Stepping 04
Memory (GB): 2816 (of 4343), Cache: 730, Private: 797
1 x 728GB [ 768/ 1/ 39]
1 x 1044GB [1788/365/379] 86:00.0#1
1 x 1044GB [1788/365/379] 87:00.0#1
Boot device: [HDD0] ATA VK000240GWCFD

Ultrastar Memory Drive Boot, Settings and Diagnostics Menu
- 17 -
Figure 3-7. Software Installer
# wmemctl --pinfo
Device info:
Board number: 0
Device number: 0
Device type: 0
Device address: 0087:00.0#1
PCI VID:DID: 1c58:0023
PCI SVID:SDID: 1c58:0023
Controller ID:
Serial number (SN): CJH002000869
Model number (MN): HUSMR7632BHP301
Firmware number (FN): KNGNP090
Number of namespaces (NN): 128
Namespace ID:
Namespace size (NSZE): 543076758
Namespace capacity (NCAP): 543076758
Namespace utilization (NUSE): 543076758
SMART / HEALTH Information Log:
Critical Warning: 0x0
Composite Temperature: 48 C
Available Spare: 191%
Available Spare Threshold: 10%
Percentage Used: 3%
Data Units Read: 4021317382
Data Units Written: 2177033579
Host Read Commands: 452665252284
Host Write Commands: 272541820061
Controller Busy Time: 44577
Power Cycles: 751
Power On Hours: 2
Unsafe Shutdowns: 140
Media and Data
Integrity Errors: 0
Number of Error
Information Log Entries: 2
License Info:
Provisioned License

Ultrastar Memory Drive Boot, Settings and Diagnostics Menu
- 18 -
3.3 System Settings and Diagnostics
During the Ultrastar memory drive loading process, the user may go into the Settings
and Diagnostics menu by pressing <F5> when prompted.
In the System tab, the following can be configured:
The total system memory available to the operating system
(use the default or set it manually).
The physical board memory used (typically used in benchmarking,
otherwise keep as Default).
Set the system to auto-restart in case of critical failure of a system
component (restarting without that component or stay frozen with the
error message).
Define the console redirection and terminal type.
Toggle the display of the welcome screen between Always and Only After
Updates options.
Select the NUMA layout that Ultrastar memory drive will present to the
operating system.
Figure 3-8. Systems Settings and Diagnostics (1)

Ultrastar Memory Drive Boot, Settings and Diagnostics Menu
- 19 -
When setting the System Memory, the Default allows the Ultrastar memory drive to
set the number to the optimal value for performance.
Switch this setting to Manual and set any other value from the range allowed by the
system, if desired.
Setting a higher number than the default value would allow for larger system memory
capacity at the expense of performance.
Figure 3-9. Systems Settings and Diagnostics (2)

Ultrastar Memory Drive Boot, Settings and Diagnostics Menu
- 20 -
Ultrastar memory drives may present the added memory gain from the NVMe SSDs, as
per the following:
Unifying Remote Memory Domains - present a processor-less (memory-
only) NUMA domain, which is the default setting.
Expand Local Memory Domains - to expand each processor’s memory
domain with a portion of the memory made available by Ultrastar memory
drive.
The Expand Local Memory Domains option is recommended for applications like the
SAP HANA in-memory database.
Consult application-specific documentation for the right choice to a desired
application.
Figure 3-10. Systems Settings and Diagnostics (3)