xFusion Digital Technologies FusionServer G8600 V7 User manual

FusionServer G8600 V7 Server
Technical White Paper
Issue 03
Date 2023-09-15
xFusion Digital Technologies Co., Ltd.

Copyright © xFusion Digital Technologies Co., Ltd. 2023. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without prior written
consent of xFusion Digital Technologies Co., Ltd.
Trademarks and Permissions
and other xFusion trademarks are trademarks of xFusion Digital Technologies Co., Ltd.
All other trademarks and trade names mentioned in this document are the property of their respective
holders.
Notice
In this document, "xFusion" is used to refer to "xFusion Digital Technologies Co., Ltd." for concise description
and easy understanding, which does not mean that "xFusion" may have any other meaning. Any "xFusion"
mentioned or described hereof may not be understood as any meaning other than "xFusion Digital
Technologies Co., Ltd.", and xFusion Digital Technologies Co., Ltd. shall not bear any liability resulting from
the use of "xFusion".
The purchased products, services and features are stipulated by the contract made between xFusion and
the customer. All or part of the products, services and features described in this document may not be within
the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,
information, and recommendations in this document are provided "AS IS" without warranties, guarantees or
representations of any kind, either express or implied.
The information in this document is subject to change without notice. Every effort has been made in the
preparation of this document to ensure accuracy of the contents, but all statements, information, and
recommendations in this document do not constitute a warranty of any kind, express or implied.
xFusion Digital Technologies Co., Ltd.
Address: 9th Floor, Building 1, Zensun Boya Square, Longzihu Wisdom Island
Zhengdong New District 450046
Zhengzhou, Henan Province
People's Republic of China
Website: https://www.xfusion.com
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. i

About This Document
Overview
This document describes the appearance, features, performance parameters, and
hardware and software compatibility of FusionServer G8600 V7, so that users can
have an in-depth and detailed understanding of FusionServer G8600 V7.
Intended Audience
This document is intended for presales engineers.
Symbol Conventions
The symbols that may be found in this document are defined as follows.
Symbol Description
Indicates a hazard with a high level of risk which, if not
avoided, will result in death or serious injury.
Indicates a hazard with a medium level of risk which, if
not avoided, could result in death or serious injury.
Indicates a hazard with a low level of risk which, if not
avoided, could result in minor or moderate injury.
Indicates a potentially hazardous situation which, if not
avoided, could result in equipment damage, data loss,
performance deterioration, or unanticipated results.
NOTICE is used to address practices not related to
personal injury.
Supplements the important information in the main text.
NOTE is used to address information not related to
personal injury, equipment damage, and environment
deterioration.
FusionServer G8600 V7 Server
Technical White Paper About This Document
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. ii

Change History
Issue Date Description
03 2023-09-15 Updated the logical structure topology.
02 2023-08-11 Added the content related to high-performance
configuration.
01 2023-04-27 This issue is the first official release.
FusionServer G8600 V7 Server
Technical White Paper About This Document
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. iii

Contents
About This Document.........................................................................................................ii
1 Overview........................................................................................................................... 1
2 Features............................................................................................................................ 2
3 Physical Structure........................................................................................................... 5
4 Logical Structure............................................................................................................. 7
5 Hardware Description......................................................................................................9
5.1 Front Panel.................................................................................................................................................. 9
5.1.1 Appearance...............................................................................................................................................9
5.1.2 Indicators and Buttons............................................................................................................................ 10
5.1.3 Port......................................................................................................................................................... 14
5.2 Rear Panel.................................................................................................................................................17
5.2.1 Appearance.............................................................................................................................................17
5.2.2 Indicators and Buttons............................................................................................................................ 19
5.2.3 Port......................................................................................................................................................... 22
5.3 Processors.................................................................................................................................................24
5.4 Memory......................................................................................................................................................25
5.4.1 DDR5 Memory........................................................................................................................................ 25
5.4.1.1 Memory Identifier................................................................................................................................. 25
5.4.1.2 Memory Subsystem Architecture.........................................................................................................27
5.4.1.3 Memory Compatibility.......................................................................................................................... 28
5.4.1.4 DIMM Installation Rules.......................................................................................................................29
5.4.1.5 Positions of Memory Modules..............................................................................................................30
5.4.1.6 Memory Protection Technologies.........................................................................................................32
5.5 Storage...................................................................................................................................................... 32
5.5.1 Drive Configuration and Drive Numbering.............................................................................................. 32
5.5.1.1 8 x 2.5" Drive Configuration.................................................................................................................33
5.5.2 Drive Indicators.......................................................................................................................................38
5.5.3 RAID Controller Card..............................................................................................................................40
5.6 Network......................................................................................................................................................40
5.6.1 OCP 3.0 NIC...........................................................................................................................................40
5.7 I/O Expansion............................................................................................................................................ 40
FusionServer G8600 V7 Server
Technical White Paper Contents
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. iv

5.7.1 PCIe Card............................................................................................................................................... 40
5.7.2 PCIe Slots...............................................................................................................................................41
5.7.3 PCIe Slot Description..............................................................................................................................45
5.8 PSUs..........................................................................................................................................................48
5.9 Fan Modules.............................................................................................................................................. 53
5.10 Board....................................................................................................................................................... 55
5.10.1 Mainboard.............................................................................................................................................55
5.10.2 Drive Backplane....................................................................................................................................56
5.10.3 Fan Boards........................................................................................................................................... 58
5.10.4 M.2 SSD Adapter Board....................................................................................................................... 60
5.10.5 NVMe Adapter...................................................................................................................................... 61
5.10.6 BMC Adapter Board (Balanced)........................................................................................................... 62
5.10.7 BMC Adapter Board (High Performance)............................................................................................. 63
6 Product Specifications.................................................................................................. 64
6.1 Technical Specifications.............................................................................................................................64
6.2 Environmental Specifications.....................................................................................................................68
6.3 Physical Specifications.............................................................................................................................. 71
7 Software and Hardware Compatibility......................................................................... 73
8 Safety Instructions.........................................................................................................74
8.1 Security......................................................................................................................................................74
8.2 Maintenance and Warranty........................................................................................................................77
9 System Management..................................................................................................... 79
10 Certification.................................................................................................................. 81
A Appendix........................................................................................................................ 83
A.1 Chassis Label Information.........................................................................................................................83
A.1.1 Chassis Head Label............................................................................................................................... 83
A.1.1.1 Nameplate........................................................................................................................................... 84
A.1.1.2 Certificate............................................................................................................................................ 84
A.1.1.3 Quick Access Label.............................................................................................................................86
A.1.2 Chassis Internal Label............................................................................................................................87
A.1.3 Chassis Tail Label...................................................................................................................................87
A.2 Product SN................................................................................................................................................ 88
A.3 Operating Temperature Limitations........................................................................................................... 89
A.4 Nameplate................................................................................................................................................. 90
A.5 RAS Features............................................................................................................................................90
A.6 Sensor List.................................................................................................................................................90
B Glossary......................................................................................................................... 96
B.1 A-E.............................................................................................................................................................96
B.2 F-J............................................................................................................................................................. 97
B.3 K-O............................................................................................................................................................ 97
FusionServer G8600 V7 Server
Technical White Paper Contents
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. v

B.4 P-T.............................................................................................................................................................97
B.5 U-Z.............................................................................................................................................................98
C Acronyms and Abbreviations...................................................................................... 99
C.1 A-E............................................................................................................................................................ 99
C.2 F-J........................................................................................................................................................... 100
C.3 K-O..........................................................................................................................................................101
C.4 P-T...........................................................................................................................................................102
C.5 U-Z.......................................................................................................................................................... 104
FusionServer G8600 V7 Server
Technical White Paper Contents
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. vi

1 Overview
FusionServer G8600 V7 (G8600 V7) is a new-generation AI infrastructure designed
for the Internet, Internet Data Center (IDC), cloud computing, enterprise business,
and telecom.
The G8600 V7 is ideal for large-scale and ultra-large-scale model training and high-
performance computing scenarios.
It features high performance, high energy efficiency, and easy deployment.
NO TE
For details about the G8600 V7 nameplate, see A.4 Nameplate.
Figure 1-1 G8600 V7 front view
FusionServer G8600 V7 Server
Technical White Paper 1 Overview
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 1

2 Features
Performance
● Supports 4th-generation Intel® Xeon® Scalable processors (Sapphire Rapids).
The server provides up to 60 cores and 120 threads, up to 350 W TDP per
processor, a maximum of 4.1 GHz turbo frequency, 2 MB L2 cache and 1.875
MB L3 cache per core, and four groups of 16 GT/s UPI interconnect links
between the processors, which deliver supreme processing performance.
● Supports a maximum of 32 x DDR5 4800 MT/s registered dual-inline memory
modules (RDIMMs), delivering up to 8192 GB total memory capacity (calculated
using the maximum capacity of a single memory module: 256 GB). The memory
modules feature high speed and availability.
● Supports NVIDIA HGX 8-GPU NVLink modules (Ampere generation or Hopper
generation).
Scalability
● The server supports up to eight 2.5" front SAS/SATA drives and eight 2.5" rear
NVMe SSDs.
● The balanced configuration supports up to 12 PCIe 5.0 x16 standard expansion
slots.
● The high-performance configuration supports up to nine PCIe 5.0 x16 standard
expansion slots.
● The balanced configuration supports one OCP 3.0 GE/10GE/25GE/100GE NIC
which features PCIe 5.0 x16 and allows orderly hot swap. The high-performance
configuration does not support OCP 3.0 NICs.
NO TE
The hot swap of the OCP NIC depends on the OS driver. Ensure that the system enters
the OS state and the OCP hot swap-related drivers have been loaded.
● Two M.2 modules can be configured.
Availability and Serviceability
● Modular design for the server, so that the GPU module, system module, I/O
module, fan module, and power module can be maintained independently.
● Carrier-class components with process expertise ensure high system reliability.
FusionServer G8600 V7 Server
Technical White Paper 2 Features
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 2

● The server supports hot-swappable SAS/SATA drives for which RAID 0, 1, 1E,
10, 5, 50, 6, or 60 can be configured (the RAID level depends on the RAID
controller card used). It also uses a supercapacitor to protect the RAID cache
data against power failures.
● The panel provides UID or Healthy LED indicators and a fault diagnosis LED
indicator. The iBMC WebUI displays key component status, enabling technical
experts to quickly locate faulty or potentially faulty components, which simplifies
maintenance, speeds up troubleshooting, and improves system availability.
● The panel provides the iBMC direct connect management port to support local
iBMC O&M, improving O&M efficiency.
● Up to six 54 V PSUs (N+M redundancy supported) and two 12 V PSUs (N+N
redundancy supported) reduce 54 V to 12 V power conversion and power
consumption.
● The GPU heat dissipation system is decoupled from the system heat dissipation
system. The two computing planes are designed with independent air ducts to
provide an optimal heat dissipation solution.
● The onboard Intelligent Baseboard Management Controller (iBMC) can
continuously monitor system parameters, trigger alarms, and take recovery
measures to minimize shutdown.
● For more information about the warranty, see xFusion Warranty Policy.
Manageability and Security
● The built-in iBMC monitors system operating status and provides remote
management.
● A password is required for accessing the BIOS, ensuring system boot and
management security.
● The Network Controller Sideband Interface (NC-SI) feature allows a network port
to simultaneously serve as a management port and a service port. The NC-SI
feature is disabled by default and can be enabled on the iBMC or BIOS.
NO TE
The service port with NC-SI enabled supports the following configuration:
● Configuring any network port on the OCP 3.0 NIC or PCIe NIC (with NC-SI enabled).
● Enabling, disabling, and setting a virtual local area network (VLAN) ID for this port.
The VLAN ID is 0 and disabled by default.
● Configuring IPv4 addresses (IPv4 address, subnet mask, and default gateway) and
IPv6 addresses (IPv6 address, prefix length) for this port.
● The integrated Unified Extensible Firmware Interface (UEFI) improves setup,
configuration, and update efficiency and simplifies fault clearance.
● Intel Execute Disable Bit (EDB) function prevents certain types of malicious
buffer overflow attacks when working with a supported OS.
● Intel® Trusted Execution Technology prevents malicious software attacks based
on hardware, prevents the firmware on the device from being maliciously
modified, and prevents unauthorized boot block execution.
● Intel® Software Guard Extensions (Intel®SGX) technology allows applications to
run in their own independent space without being affected by other software
running in the system, thereby enhancing security.
FusionServer G8600 V7 Server
Technical White Paper 2 Features
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 3

● The secure boot based on the chip root of trust (RoT) implements level-by-level
firmware verification starting from the hardware RoT and builds a complete
secure boot chain.
● The trusted platform module (TPM) and trusted password module (TCM) provide
advanced encryption functions, such as digital signature and remote
authentication.
● The following requirements in NIST SP 800-147B are met:
– The BIOS firmware digital signature update mechanism is supported. During
the upgrade, the digital signature is verified to prevent unauthorized BIOS
firmware upgrade.
– The flash security protection mechanism is supported to prevent
unauthorized modification of the flash memory in the OS.
Energy Efficiency
● The 80 PLUS Platinum or Titanium PSUs of multiple energy efficiency levels
provide 96% power efficiency at 50% load.
● Supports active/standby power supply and high-voltage DC power supply to
improve the efficiency of the power supply system.
● Efficient voltage regulator-down (VRD) power supplies for boards minimize the
energy loss from DC/DC power conversion.
● Area-based, Proportional-Integral-Derivative (PID) intelligent fan speed
adjustment and intelligent CPU frequency scaling optimize heat dissipation and
reduce overall system power consumption.
● The improved thermal design with energy-efficient fans ensures optimal heat
dissipation and reduces system power consumption.
● Staggered spinup of drives reduces the server boot power consumption.
FusionServer G8600 V7 Server
Technical White Paper 2 Features
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 4

3 Physical Structure
Figure 3-1 Front view
1GPU module enclosure 2 System enclosure
3 54 V dual-input PSUs (for
GPU modules)
4 Chassis
FusionServer G8600 V7 Server
Technical White Paper 3 Physical Structure
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 5

Figure 3-2 Rear view
1 12 V PSUs (for the system
enclosure)
2 Composite enclosure
3 I/O module 1 4 I/O module 2
5 I/O module 3 6 I/O module 4
7 Rear fan modules (used for
heat dissipation of GPU
modules)
8 Rear upper subrack
9 Chassis - -
FusionServer G8600 V7 Server
Technical White Paper 3 Physical Structure
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 6

4 Logical Structure
Figure 4-1 Logical structure of the balanced configuration
FusionServer G8600 V7 Server
Technical White Paper 4 Logical Structure
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 7

Figure 4-2 Logical structure of the high-performance configuration
● A server supports two next-generation Intel® Xeon® Scalable processors
(Sapphire Rapids). It supports 32 DDR5 memory modules. The processors
interconnect with each other through three UltraPath Interconnect (UPI) links at
a speed of up to 16 GT/s.
● High-speed resources of the CPU are connected through PCBs, high-speed
connectors, and backplanes. PCIe 5.0 high-speed links are modularized and
cable-free.
● The Baseboard Management Controller (BMC) card integrates the BMC
management chip and provides external video graphic array (VGA),
management network port, and serial port.
FusionServer G8600 V7 Server
Technical White Paper 4 Logical Structure
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 8

5 Hardware Description
5.1 Front Panel
5.2 Rear Panel
5.3 Processors
5.4 Memory
5.5 Storage
5.6 Network
5.7 I/O Expansion
5.8 PSUs
5.9 Fan Modules
5.10 Board
5.1 Front Panel
5.1.1 Appearance
● 8 x 2.5" drive configuration
FusionServer G8600 V7 Server
Technical White Paper 5 Hardware Description
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 9

Figure 5-1 Front view
1 GPU module enclosure 2 Drives
3 System enclosure 4 54 V dual-input PSUs (for
GPU modules)
5 Slide-out label plate (with
an SN label)
- -
5.1.2 Indicators and Buttons
Indicator and Button Locations
● 8 x 2.5" drive configuration
FusionServer G8600 V7 Server
Technical White Paper 5 Hardware Description
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 10

Figure 5-2 Indicators and buttons on the front panel
1 Fault diagnosis LED 2 Power button/indicator
3 FlexIO card 1 presence
indicator
4 Health status indicator
5 FlexIO card 2 presence
indicator
6 UID button/indicator
7 iBMC direct connect
management port
indicator
8 54 V PSU indicators
Indicator and Button Descriptions
Table 5-1 Description of indicators and buttons on the front panel
Identifier Indicators
and Buttons
State Description
Fault
diagnosis LED
● ---: The device is operating properly.
● Error code: A component is faulty.
For details about the error codes, see the
FusionServer G8600 V7 Server iBMC Alarm
Handling.
FusionServer G8600 V7 Server
Technical White Paper 5 Hardware Description
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 11

Identifier Indicators
and Buttons
State Description
Health status
indicator
● Off: The device is powered off or faulty.
● Blinking red at 1 Hz: A major alarm has been
generated on the system.
● Blinking red at 5 Hz: A critical alarm has been
generated on the system.
● Steady green: The device is operating
properly.
FlexIO card
presence
indicator
Indicates whether the FlexIO card is detected.
● Off: The OCP 3.0 network adapter is not in
position.
● Blinking green at 0.5 Hz: The OCP 3.0
network adapter is in position but is not
powered on.
● Blinking green at 2 Hz: The OCP 3.0 network
adapter is in position and has just been
inserted.
● Steady green: The OCP 3.0 network adapter
is in position, and the power supply is normal.
Power button/
indicator
Power indicator:
● Off: The device is powered off.
● Steady green: The device is powered on.
● Blinking yellow: The iBMC is starting. The
power button is locked and cannot be
pressed. The iBMC is started in about 1
minute, and then the power indicator is
steady yellow.
● Steady yellow: The device is in the standby
state.
Power button:
● When the device is powered on, you can
press this button to gracefully shut down the
OS.
NOTE
For different OSs, you may need to shut down the
OS as prompted.
● When the device is powered on, you can hold
down this button for 6 seconds to forcibly
power off the device.
● When the power indicator is steady yellow,
you can press this button to power on the
server node.
FusionServer G8600 V7 Server
Technical White Paper 5 Hardware Description
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 12

Identifier Indicators
and Buttons
State Description
UID button/
indicator
The UID button/indicator helps identify and
locate a device.
UID indicator:
● Off: The device is not being located.
● Blinking or steady blue: The device is being
located.
UID button:
● You can control the UID indicator status by
pressing the UID button or using the iBMC.
● You can press this button to turn on or off the
UID indicator.
● You can press and hold down this button for 4
to 6 seconds to reset the iBMC.
iBMC direct
connect
management
port indicator
Indicates the status when the iBMC direct
connect management port connects to a
terminal (local PC or Android mobile phone):
● Off: No terminal is connected.
● Blinking green at short intervals for 3 seconds
and then off: The port is disabled.
● Steady green: The terminal is connected.
Indicates the status when the iBMC direct
connect management port connects to a USB
device:
● Blinking red at long intervals: The job fails or
an error is reported when the job is complete.
● Blinking green at short intervals: The job is
being executed.
● Blinking green at short intervals for 3 seconds
and then off: The port is disabled.
● Steady green: The server configuration file is
being copied from the USB device or the job
is successfully completed.
FusionServer G8600 V7 Server
Technical White Paper 5 Hardware Description
Issue 03 (2023-09-15) Copyright © xFusion Digital Technologies Co., Ltd. 13