Fusion FusionServer XH321L User manual

FusionServer XH321L V5 Liquid-Cooled Server
Node
Technical White Paper
Issue 07
Date 2023-11-30
xFusion Digital Technologies Co., Ltd.

Copyright © xFusion Digital Technologies Co., Ltd. 2023. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior written
consent of xFusion Digital Technologies Co., Ltd.
Trademarks and Permissions
and other xFusion trademarks are trademarks of xFusion Digital Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective
holders.
Notice
In this document, "xFusion" is used to refer to "xFusion Digital Technologies Co., Ltd." for concise description
and easy understanding, which does not mean that "xFusion" may have any other meaning. Any "xFusion"
mentioned or described hereof may not be understood as any meaning other than "xFusion Digital
Technologies Co., Ltd.", and xFusion Digital Technologies Co., Ltd. shall not bear any liability resulting from
the use of "xFusion".
The purchased products, services and features are stipulated by the contract made between xFusion and
the customer. All or part of the products, services and features described in this document may not be within
the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,
information, and recommendations in this document are provided "AS IS" without warranties, guarantees or
representations of any kind, either express or implied.
The information in this document is subject to change without notice. Every effort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, express or implied.
xFusion Digital Technologies Co., Ltd.
Address: 9th Floor, Building 1, Zensun Boya Square, Longzihu Wisdom Island
Zhengdong New District 450046
Zhengzhou, Henan Province
People's Republic of China
Website: https://www.xfusion.com
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. i

About This Document
Purpose
This document describes the FusionServer XH321L V5 liquid-cooled server node in
terms of features, structure, specifications, and component hardware and software
compatibility.
Intended Audience
This document is intended for pre-sales engineers.
Symbol Conventions
The symbols that may be found in this document are defined as follows.
Symbol Description
Indicates a hazard with a high level of risk which, if not
avoided, will result in death or serious injury.
Indicates a hazard with a medium level of risk which, if
not avoided, could result in death or serious injury.
Indicates a hazard with a low level of risk which, if not
avoided, could result in minor or moderate injury.
Indicates a potentially hazardous situation which, if not
avoided, could result in equipment damage, data loss,
performance deterioration, or unanticipated results.
NOTICE is used to address practices not related to
personal injury.
Supplements the important information in the main text.
NOTE is used to address information not related to
personal injury, equipment damage, and environment
deterioration.
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper About This Document
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. ii

Change History
Issue Date Description
07 2023-11-30 ● Added 11 Waste Product Recycling.
06 2023-02-28 ● Updated 5.7.1 Mainboard .
● Added A.5 Sensor List.
05 2022-11-11 Updated 8.1 Security.
04 2022-08-12 ● Optimized 6.3 Physical Specifications .
● Added A.1 Node Label.
03 2022-06-25 ● Added a figure that shows how to measure
dimensions.
● Updated 5.4.1 Drive Configuration and
5.4.2 Drive Numbering.
● Added 5.4.3 Drive Indicators.
02 2022-03-18 Added Certifications [XH321L V5-BP].
01 2021-12-20 This issue is the first official release.
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper About This Document
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. iii

Contents
About This Document.........................................................................................................ii
1 Product Introduction....................................................................................................... 1
2 Features............................................................................................................................ 4
3 Physical Structure........................................................................................................... 7
4 Logical Structure........................................................................................................... 10
5 Hardware Description.................................................................................................... 11
5.1 Front Panel.................................................................................................................................................11
5.1.1 Appearance.............................................................................................................................................11
5.1.2 Indicators and Buttons............................................................................................................................ 12
5.1.3 Ports .......................................................................................................................................................14
5.1.4 Installation Positions............................................................................................................................... 16
5.2 Processor...................................................................................................................................................16
5.3 Memory......................................................................................................................................................17
5.3.1 DDR4 Memory........................................................................................................................................ 18
5.3.1.1 Memory Identifier................................................................................................................................. 18
5.3.1.2 Memory Subsystem Architecture.........................................................................................................19
5.3.1.3 Memory Compatibility.......................................................................................................................... 20
5.3.1.4 DIMM Installation Rules.......................................................................................................................21
5.3.1.5 Memory Installation Positions.............................................................................................................. 22
5.3.1.6 Memory Protection Technologies.........................................................................................................23
5.4 Storage...................................................................................................................................................... 23
5.4.1 Drive Configuration................................................................................................................................. 24
5.4.2 Drive Numbering.....................................................................................................................................25
5.4.3 Drive Indicators.......................................................................................................................................26
5.4.4 RAID Controller Card..............................................................................................................................26
5.5 Network......................................................................................................................................................26
5.5.1 LOMs ..................................................................................................................................................... 26
5.6 I/O Expansion............................................................................................................................................ 29
5.6.1 PCIe Cards............................................................................................................................................. 29
5.6.2 PCIe Slots...............................................................................................................................................29
5.6.3 PCIe Slot Description..............................................................................................................................29
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper Contents
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. iv

5.7 Boards........................................................................................................................................................30
5.7.1 Mainboard ..............................................................................................................................................30
6 Product Specifications.................................................................................................. 32
6.1 Technical Specifications.............................................................................................................................32
6.2 Environmental Specifications.....................................................................................................................36
6.3 Physical Specifications ............................................................................................................................. 38
7 Software and Hardware Compatibility......................................................................... 39
8 Safety Instructions.........................................................................................................40
8.1 Security......................................................................................................................................................40
8.2 Maintenance and Warranty........................................................................................................................43
9 System Management..................................................................................................... 44
10 Certifications................................................................................................................ 46
11 Waste Product Recycling............................................................................................ 47
A Appendix........................................................................................................................ 48
A.1 Node Label................................................................................................................................................ 48
A.2 Product SN................................................................................................................................................ 48
A.3 Operating Temperature Limitations........................................................................................................... 50
A.3.1 Operating Temperature Limitations (Supercapacitor for the RAID Controller Card Configured)............50
A.3.2 Operating Temperature Limitations (Optical Modules Configured).........................................................51
A.3.3 Operating Temperature Limitations (Avago SAS3004iMR RAID Controller Card + M.2 FRU Configured)
.........................................................................................................................................................................54
A.4 RAS Features............................................................................................................................................55
A.5 Sensor List.................................................................................................................................................57
B Glossary......................................................................................................................... 67
B.1 A-E.............................................................................................................................................................67
B.2 F-J............................................................................................................................................................. 67
B.3 K-O............................................................................................................................................................ 68
B.4 P-T.............................................................................................................................................................68
B.5 U-Z.............................................................................................................................................................69
C Acronyms and Abbreviations...................................................................................... 70
C.1 A-E............................................................................................................................................................ 70
C.2 F-J............................................................................................................................................................. 71
C.3 K-O............................................................................................................................................................73
C.4 P-T.............................................................................................................................................................73
C.5 U-Z............................................................................................................................................................ 75
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper Contents
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. v

1 Product Introduction
The FusionServer XH321L V5 (XH321L V5) is a liquid-cooled server node designed
for X6000 servers. An X6000 can house up to four XH321L V5 server nodes in a 2U
chassis.
The XH321L V5 delivers supreme performance and high storage density in limited
space through innovative design. It supports liquid cooling and is easy to manage
and maintain.
The XH321L V5 supports six 2.5-inch SAS/SATA/NVMe drives, up to 16 DDR4
memory modules, and one PCIe standard card or screw-in RAID controller card.
The XH321L V5 is ideal for data center, cloud computing, Big Data, and Internet
applications.
The XH321L V5 nodes are available in two types of configurations: liquid cooling of
processors only and liquid cooling of processors and memory modules.
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 1 Product Introduction
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 1

Figure 1-1 XH321L V5 (liquid cooling of processors only)
Figure 1-2 XH321L V5 (liquid cooling of processors and memory modules)
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 1 Product Introduction
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 2

Table 1-1 Configuration description
Configuration Application description
Liquid cooling of
processors
● Liquid cold plates for processors are provided.
● It fits into our company's full liquid cooling cabinets or
customer's liquid-cooled cabinets.
● When used with our company's full liquid cooling
cabinets, the memory and other components are
cooled by the air/liquid heat exchanger in the cabinet.
Liquid cooling of
processors and
memory modules
● Liquid cold plates for processors and memory
modules are provided.
● It fits into our company's board-level liquid cooling
cabinets or customer's liquid-cooled cabinets.
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 1 Product Introduction
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 3

2 Features
Performance and Scalability
● Powered by two Intel® Xeon® Scalable Skylake or Cascade Lake processors,
the server provides up to 28 cores, 3.8 GHz frequency, a 38.5 MB L3 cache, and
two 10.4 GT/s UPI links between the processors, which deliver supreme
processing performance.
– It supports up to two processors with 56 cores and 112 threads to maximize
the concurrent execution of multithreaded applications.
– An L2 cache is added. Each core can exclusively use 1 MB of L2 cache and
at least 1.375 MB of L3 cache.
– Intel Turbo Boost Technology 2.0 allows processor cores to run faster than
the frequency specified in the Thermal Design Power (TDP) configuration if
they are operating below power, current, and temperature specification
limits.
– Intel Hyper-Threading Technology enables each processor core to run up to
two threads, improving parallel computation capability.
– The hardware-assisted Intel® Virtualization Technology (Intel® VT) allows
operating system (OS) vendors to better use hardware to address
virtualization workloads.
– Intel® Advanced Vector Extensions (Intel AVX 2.0 and Intel AVX-512)
significantly accelerate floating-point performance for computing-intensive
applications.
– The Cascade Lake processors support Intel® Deep Learning Boost vector
neural network instructions (VNNI) to improve the performance of deep
learning applications.
● The 1.2 V DDR4 memory modules consume 20% less power than 1.35 V
DDR3L memory modules used by the previous-generation platform.
● The server node supports double data rate 4 (DDR4) registered dual in-line
memory modules (RDIMMs) and load-reduced DIMMs (LRDIMMs) with error
checking and correcting (ECC). A server node fully configured with sixteen 2933
MT/s memory modules (only available with Cascade Lake processors) provides
1 TB memory space and the maximum theoretical memory bandwidth of 274.97
GB/s.
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 2 Features
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 4

● With Intel integrated I/O, the Intel® Xeon® Scalable processors integrate the
PCIe 3.0 controller to shortens I/O latency and improve overall system
performance.
● It supports six SATA/SAS HDDs and NVMe SSDs. The SSDs deliver higher I/O
performance than HDDs. An SSD supports approximately 100 times more I/O
operations per second (IOPS) than a typical HDD.
● One half-height half-length (HHHL) PCIe SSD card can be configured to support
high-bandwidth low-delay data access.
Availability and Serviceability
● Carrier-class components with process expertise ensure high system reliability
and availability.
● The server node supports six 2.5-inch or three 3.5-inch hot-swappable drives. It
supports RAID 0, 1, 1E, 5, 6, 10, 50, and 60, depending on the RAID controller
card used. It also uses a supercapacitor to protect the RAID cache data against
power failures.
● The server provides simplified O&M and efficient troubleshooting through the
UID/HLY LED indicators on the front panel, fault diagnosis LED, and iBMC
WebUI.
● The SSDs offer better reliability than HDDs, ensuring continued system
performance.
● The iBMC monitors system parameters in real time, triggers alarms, and
performs recovery actions in case of failures, minimizing system downtime.
● For more information about the warranty in the Chinese market, see Warranty.
Manageability and Security
● The built-in iBMC monitors server operating status and provides remote
management.
● The integrated Unified Extensible Firmware Interface (UEFI) improves setup,
configuration, and update efficiency and simplifies fault handling.
● The Advanced Encryption Standard–New Instruction (AES NI) algorithm allows
faster and stronger encryption.
● Intel Execute Disable Bit (EDB) function prevents certain types of malicious
buffer overflow attacks when working with a supported OS.
● The Network Controller Sideband Interface (NC-SI) allows a network port to
serve as a management port and a service port for maximized return on
investment (ROI) for customers. The NC-SI feature is disabled by default and
can be enabled through the iBMC or BIOS.
● Intel Trusted Execution Technology enhances security using hardware-based
defense against malicious software attacks, allowing applications to run
independently.
● The trusted platform module (TPM) 2.0 provides advanced encryption functions,
such as digital signatures and remote authentication.
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 2 Features
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 5

NO TE
The service port with NC-SI enabled supports the following configuration:
● Configuring the LOM port (NIC Mezz) of the server as the service port with NC-SI enabled.
LOM port 1 is configured by default.
● Enabling, disabling, and setting a virtual local area network (VLAN) ID for this port. The
VLAN ID is 0 and disabled by default.
● Configuring IPv4 addresses (IPv4 address, subnet mask, and gateway) and IPv6
addresses (IPv6 address, prefix length, and gateway) for this port.
Energy Efficiency
● The Intel® Xeon® Scalable processors outperform the previous-generation
processors while fitting into the same TDP.
● Efficient voltage regulator-down (VRD) power supplies for boards minimize the
energy loss from DC/DC power conversion.
● Intel® Intelligent Power Capability allows a processor to be powered on or off
based on requirements.
● Low-voltage Intel® Xeon® Scalable processors consume less energy, ideally
suited for data centers and telecommunications environments constrained by
power and thermal limitations.
● Staggered spin-up for drives reduces the server boot power consumption.
● The 1.2 V DDR4 RDIMMs consume 20% less energy than 1.5 V DDR3
RDIMMs.
● SSDs consume 80% less power than HDDs.
● The server supports intelligent frequency scaling of processors for energy
conservation and consumption reduction.
● The quadrilateral ventilation holes on the node panel provide higher ventilation
density than round holes, increasing the system cooling efficiency.
● The compute node uses liquid cooling cold plates to dissipate heat for major
components, such as the CPUs, VRD, and memory modules, which improves
heat dissipation efficiency, decreases the fan speed, and reduces power
consumption.
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 2 Features
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 6

3 Physical Structure
The XH321L V5 nodes are available in two types of configurations:
● Liquid cooling of processors only
● Liquid cooling of processors and memory modules
Figure 3-1 XH321L V5 (processor liquid cooling configurations) physical structure
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 3 Physical Structure
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 7

1 Liquid cold plate 2 PCIe card
3 Supercapacitor 4 RAID controller card
5 Mainboard 6 Processor
7 Memory 8 Node case
9 M.2 SSD card 10 TPM (with M.2 ports)
Figure 3-2 XH321L V5 (processor memory liquid cooling configurations) physical
structure
1Liquid cold plate 2 PCIe card
3 Supercapacitor 4 RAID controller card
5 Mainboard 6 Processor
7 Memory 8 Node case
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 3 Physical Structure
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 8

9 M.2 SSD card 10 TPM (with M.2 ports)
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 3 Physical Structure
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 9

4 Logical Structure
Figure 4-1 XH321L V5 logical structure
● The server supports two Intel® Xeon® Scalable processors.
● The server supports up to 16 memory modules.
● The CPUs (processors) interconnect with each other through two UPI links at a
speed of up to 10.4 GT/s.
● The PCIe cards connected to the processors provide service ports.
● The RAID controller card connects to the mainboard using a SAS high-speed
cable and connects to the system backplane through a connector.
● The BMC chipset interconnects with the PCH through PCIe and LPC buses to
provide a management interface.
● The BMC integrates the graphics card, video compression, and virtual media
component to provide device management functions, such as power control, slot
ID acquisition, power supply detection, and KVM over IP.
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 4 Logical Structure
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 10

5 Hardware Description
5.1 Front Panel
5.2 Processor
5.3 Memory
5.4 Storage
5.5 Network
5.6 I/O Expansion
5.7 Boards
5.1 Front Panel
5.1.1 Appearance
Figure 5-1 Front view
1PCIe card (RAID controller
card not configured)
2 Water outlet
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 5 Hardware Description
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 11

3 Water inlet 4 Slide-out label plate (with
an SN label)
5.1.2 Indicators and Buttons
Indicator and Button Positions
Figure 5-2 Indicators and buttons on the front panel
1LOM port indicator 2 Power button/indicator
3 UID button/indicator 4 Health status indicator
5 LOM port indicator 6 Data transmission status
indicator of the
management network port
7 Connection status indicator
of the management
network port
- -
Indicator and Button Descriptions
Table 5-1 Description of indicators and buttons on the front panel
Sign Indicator and
Button
Description
- LOM port
indicator
For details, see 5.5.1 LOMs .
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 5 Hardware Description
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 12

Sign Indicator and
Button
Description
Power button/
indicator
Power indicator:
● Off: The device is not powered on.
● Steady green: The device is powered on.
● Blinking yellow: The iBMC is starting. The
power button is locked and cannot be
pressed. The iBMC is started in about 1
minute, and then the power indicator is
steady yellow.
● Steady yellow: The device is standby.
Power button:
● When the device is powered on, you can
press this button to gracefully shut down the
OS.
NOTE
For different OSs, you may need to shut down the
OS as prompted.
● When the device is powered on, holding
down this button for 6 seconds will forcibly
power off the device.
● When the power indicator is steady yellow,
you can press this button to power on the
device.
Health status
indicator
● Off: The device is powered off or is faulty.
● Blinking red at 1 Hz: A major alarm has been
generated on the system.
● Blinking red at 5 Hz: A critical alarm has been
generated on the system.
● Steady green: The device is operating
properly.
UID button/
indicator
The UID button/indicator helps identify and
locate a device.
UID indicator:
● Off: The device is not being located.
● Blinking or steady blue: The device is being
located.
UID button description:
● You can control the UID indicator status by
pressing the UID button or using the iBMC.
● You can press this button to turn on or off the
UID indicator.
● You can press and hold down this button for 4
to 6 seconds to reset the iBMC.
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 5 Hardware Description
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 13

Sign Indicator and
Button
Description
- Data
transmission
status
indicator of
the
management
network port
● Off: No data is being transmitted.
● Blinking yellow: Data is being transmitted.
- Connection
status
indicator of
the
management
network port
● Off: The network port is not connected.
● Steady green: The network port is properly
connected.
5.1.3 Ports
Port Positions
● Ports on the front panel
Figure 5-3 Ports on the front panel
1Water outlet 2 Water inlet
3 Management network
port
4 LOM port 1
5 LOM port 2 6 Universal connector port
● Ports on the multi-port cable
FusionServer XH321L V5 Liquid-Cooled Server Node
Technical White Paper 5 Hardware Description
Issue 07 (2023-11-30) Copyright © xFusion Digital Technologies Co., Ltd. 14