Dolphin MXS824 User manual

Dolphin PCI Express MXS824 Switch
MXS824 24 ports PCI Express Gen3 switch users guide
Version 1.0.2
Date: 5th November 2018

MXS824 Users Guide –Dolphin Interconnect Solutions Page 1
Table of Contents
Table of Contents.......................................................................................................................................................................1
Terms and Acronyms .................................................................................................................................................................3
Overview....................................................................................................................................................................................4
Software support...................................................................................................................................................................4
Revision history..........................................................................................................................................................................5
Hardware revisions................................................................................................................................................................5
Management firmware revisions ..........................................................................................................................................5
PCI Express Gen3 Switch - MXS824 ...........................................................................................................................................6
MXS824 high level specification............................................................................................................................................6
Configuration examples.............................................................................................................................................................7
6 node PCIe x16 Gen3 NTB Cluster........................................................................................................................................7
12 node PCIe x8 Gen3 NTB Cluster........................................................................................................................................7
24 node PCIe xx Gen3 NTB Cluster........................................................................................................................................7
10 node PCIe x16 Gen3 NTB Cluster......................................................................................................................................8
20 node PCIe x8 Gen3 NTB Cluster........................................................................................................................................8
Larger NTB Cluster configurations.........................................................................................................................................8
Transparent PCIe IO expansion –Multiple expansion chassis connected to one server ....................................................10
Cables ..................................................................................................................................................................................11
Management processor ......................................................................................................................................................11
Use cases summary and settings.........................................................................................................................................11
LEDs .........................................................................................................................................................................................12
MXS824 Switch Status LEDs ................................................................................................................................................12
Port and Link status LEDs ....................................................................................................................................................12
Port status LEDs ..............................................................................................................................................................12
Link status LEDs...............................................................................................................................................................13
USB Port...................................................................................................................................................................................13
SD Card Port.............................................................................................................................................................................13
Installation ...............................................................................................................................................................................14
Safety instructions...............................................................................................................................................................14
Installing the Switch in a rack..............................................................................................................................................14
Switch desktop use..............................................................................................................................................................14
Ethernet...............................................................................................................................................................................14
Installing and removing PCIe cables....................................................................................................................................14
MXS824 operating modes...................................................................................................................................................15

MXS824 Users Guide –Dolphin Interconnect Solutions Page 2
Transparent mode port usage.............................................................................................................................................15
NTB mode port usage..........................................................................................................................................................15
Cascading switches..............................................................................................................................................................15
Partitioning –mixed transparent and NTB usage ...............................................................................................................15
Configuring the MXS824 switch...............................................................................................................................................16
Status section ......................................................................................................................................................................16
PCIe Configuration section ..................................................................................................................................................16
Change configuration......................................................................................................................................................17
Upload configuration ......................................................................................................................................................17
Delete configuration group.............................................................................................................................................18
Reset switch ....................................................................................................................................................................18
Firmware section.................................................................................................................................................................18
Software installation, PXH830, PXH820...................................................................................................................................19
Software installation, MXH830................................................................................................................................................19
Support ....................................................................................................................................................................................19
Compliance ..............................................................................................................................................................................20
EMC compliance..................................................................................................................................................................20
Safety compliance ...............................................................................................................................................................20
RoHS compliance.................................................................................................................................................................20
Flammability standard.........................................................................................................................................................20
WEEE Notice........................................................................................................................................................................20
Limited warranty .....................................................................................................................................................................21
Warranty period..................................................................................................................................................................21
Coverage..............................................................................................................................................................................21
Service procedure ...............................................................................................................................................................21
Limitations...........................................................................................................................................................................21
DISCLAIMER
DOLPHIN INTERCONNECT SOLUTIONS RESERVES THE RIGHT TO MAKE CHANGES WITHOUT FURTHER NOTICE TO ANY OF
ITS PRODUCTS TO IMPROVE RELIABILITY, FUNCTION, OR DESIGN. DOLPHIN INTERCONNECT SOLUTIONS DOES NOT
ASSUME ANY LIABILITY ARISING OUT OF THE APPLICATION OR USE OF ANY PRODUCT.
LIFE SUPPORT POLICY
DOLPHIN INTERCONNECT SOLUTIONS’ PRODUCTS ARE NOT AUTHORIZED FOR USE AS CRITICAL COMPONENTS IN LIFE
SUPPORT DEVICES.

MXS824 Users Guide –Dolphin Interconnect Solutions Page 3
ENVIRONMENTAL POLICY
Dolphin is minimizing the amount of printed documentation and software CDs in its shipments, please download
additional documentation and software from www.dolphinics.com/support/.
Terms and Acronyms
Important terms and acronyms used in this manual
AOC Active Optical Cable. PCIe fiber cable assembly.
CMI Cable Management Interface. The 2-wire management interface for communication between
subsystems connected by a PCIe 3.0 cable. Details can be found in the PCI-SIG External Cabling
Specification 3.0. The CMI functionality is optional with the MXS824 switch.
CMI Controller Microcontroller on the MXS824 used to manage and implement the CMI communications. CMI
communication require PCIe 3.0 cables.
eXpressWare Dolphins software stack for PCIe clustering and IO. www.dolphinics.com for more information.
Lane One PCI Express Lane contains a differential pair for transmission and a differential pair for reception.
Link A collection of PCI Express Lanes providing the communication path between an Upstream and
Downstream Port. The MXS824 support links of x4, x8 or x16 width by utilizing lanes from 1,2 or 4 ports.
MiniSAS-HD Standard MiniSAS-HD cable with SFF-8644 connectors.
PCIe 3.0 cable PCIe 3.0 cable with SFF-8644 connectors. Complaint to the new PCI-SIG External Cabling Specification
3.0. Support for CMI.
Port The MXS824 has 24 x4 Gen3 ports, named P1, P2, P3 …, P24. The port numbers are printed on the front
panel below the connectors.
Fabric Port MXS824 ports configured to be used as a link to another MXS824 switch. For standard dual switch
configurations, port P1, P2, P3 and P4 are normally the fabric ports.

MXS824 Users Guide –Dolphin Interconnect Solutions Page 4
Overview
This document describes the Dolphin MXS824 PCI Express Gen3 switch used in combination with other Dolphin PCI
Express products. Dolphin is currently providing the following compliant PCI Express products:
•PXH820 PCI Express Gen3 x8 NTB XMC MiniSAS-HD / SFF-8644 cable adapter card
•PXH822 PCI Express Gen3 x8 Transparent Host and Target XMC MiniSAS-HD / SFF-8644 cable adapter card
•PXH830 PCI Express Gen3 x16 NTB MiniSAS-HD / SFF-8644 cable adapter card
•PXH832 PCI Express Gen3 x16 Transparent Host and Target MiniSAS-HD / SFF-8644 cable adapter card
•MXH830 PCI Express Gen3 x16 NTB MiniSAS-HD / SFF-8644 cable adapter card
•MXH832 PCI Express Gen3 x16 Transparent Host and Target MiniSAS-HD / SFF-8644 cable adapter card
•PXH820 PCI Express Gen3 x8 XMC NTB MiniSAS-HD / SFF-8644 cable adapter card
•PXH822 PCI Express Gen3 x8 XMC Transparent Host and Target MiniSAS-HD / SFF-8644 cable adapter card
The MXS824 switch can be used to create a PCIe fabric or PCIe expansion using the above products or to any product
compliant to the PCI-SIG External Cabling Specification 3.0.
Software support
All Dolphin PXH and MXH PCI Express NTB adapter cards support the complete eXpressWare suite of Dolphin PCI Express
Software including Dolphin SuperSockets™, optimized TCP/IP drivers, SmartIO and SISCI Embedded software.
Dolphin SuperSockets is a Berkeley Sockets compliant library that provides a socket latency below 1 microsecond, and
close to the wire speed streaming bandwidth for networked applications. SuperSockets™is currently available on Linux
and Windows. It is a 100% transparent plug-and-play solution for commercial and embedded applications.
Dolphin’s optimized TCP/IP driver enables PCI Express to be used as a traditional 10G Ethernet / 40G Ethernet
replacement such as for NFS sharing and legacy networking that does not need the low latency provided by SuperSockets.
The TCP/IP driver supports gateway functionality.
The SISCI software provides a well-defined, easy to use shared memory / reflective memory programming API for PCI
Express over cable.
The SmartIO software enables dynamic sharing of and access to PCIe devices through a PCIe fabric.
More information about eXpressWare at http://www.dolphinics.com/products/dolphin_pci_express_software.html.

MXS824 Users Guide –Dolphin Interconnect Solutions Page 5
Revision history
Hardware revisions
The following table gives a general overview of the hardware capabilities.
Switch revision
Capabilities
MXS824-BC
•Initial product version
MXS824-CC
•Minor cosmetic and manufacturing improvements
Management firmware revisions
The following table gives a general overview of the management software capabilities. All versions contain various bug
fixes, security patches, and stability improvements.
Revision
Capabilities
1.0.0
•Initial functionality, 6x16 and 12x8 NTB configurations
1.1.0
•Added configurations for 24x4 NTB configurations, Dual switch configurations,
Transparent configurations. Updated web interface.

MXS824 Users Guide –Dolphin Interconnect Solutions Page 6
PCI Express Gen3 Switch - MXS824
The MXS824 is a 24 port PCI Express Gen3 x4 switch available from Dolphin providing an easy to use, multi-functional
solution.
MXS824 high level specification
•PCI Express Gen1 2.5 Gbps / Gen2 5.0 Gbps / Gen3 8.0 Gbps per lane signaling –4 lanes –up to 32 Gbps total
signaling per port.
•Compliant with PCI Express Gen1, Gen2, and Gen3 computers and IO systems, auto detection. Each port operates
independently at Gen1, Gen2, or Gen3 speeds
•Lane width auto-training to connected cable width (x16, x8, x4, x2, x1)
•19-Inch 1U rack mountable chassis.
•Physical dimensions 440mm x 357mm x 45mm (17” x 14” x 1.75”) (WxDxH).
•Supports NTB connections to other hosts and transparent connections to IO systems as a transparent P2P device
•Based on Microsemi PFX Switchtec chipset
•PCI Express Base Specification Rev 3.1
•PCI Express External Cabling Specification, Rev. 3.0
•Twenty-four PCI Express Gen3 x4 SFF-8644 Cable connection
oDurability max total 250 mating cycles
•Link clock isolation. Automatic support for hosts running CFC or SSC mode.
•Support for hot plugging of the PCI Express cable (NTB mode, SISCI and SuperSockets)
•Built in management processor for monitoring and firmware upgrade and management
oRJ45 Ethernet (10/100) connector (optional use)
oUSB port for firmware upgrade
oSD-card boot medium
•Front operational status and alert LEDs
•No PCI Express power domain isolation
•4 Redundant fans
•Operating systems supported:
oNTB mode: Linux, Windows, RTX and VxWorks.
oTransparent mode: All (No drivers required)
•Power supply input 110 - 240VAC 50/60 Hz –2.8A
•Power Consumption: Max 80 Watt (including active optical transceivers)
•Operating temperature 0°C to 40°C, Relative Humidity 5% - 95% non-condensing
•Storage temperature -25°C to 70°C, Relative Humidity 10% - 90% non-condensing
•Weight: 4.6 Kilo (10 lbs.)
•RoHS compliant
•FCC & CE Mark
Picture 1: MXS824 front view

MXS824 Users Guide –Dolphin Interconnect Solutions Page 7
Configuration examples
The MXS824 switch can be used as a standalone PCI Express switch, or several switches can be cascaded to build larger
PCIe expansion systems or PCIe fabrics. It can be combined with compliant PCIe cards available from Dolphin or compliant
customer designs. Some examples can be found below. Please consult the chapter Configuring the MXS824 switch on page
16 for more details. Please note that the Dolphin eXpressWare currently doesn’t support mixing MXH830 and PXH8x0 in a
single cluster.
6 node PCIe x16 Gen3 NTB Cluster
Each node has a Dolphin Express MXH830, PXH830 or PXH820
adapter. Up to six systems can be connected to the Dolphin
MXS824 switch using the default firmware configuration. All nodes
connect directly to the switch using up to 4 x4 MiniSAS-HD or PCIe
3.0 cables (x16 link). This is the switch default configuration.
12 node PCIe x8 Gen3 NTB Cluster
Each node has a Dolphin Express MXH830, PXH830 or PXH820
adapter. Up to twelve systems can be connected to the Dolphin
MXS824 switch. All nodes connect directly to the switch using up to
2 x4 MiniSAS-HD or PCIe 3.0 cables (x8 link).
24 node PCIe xx Gen3 NTB Cluster
Each node has a Dolphin Express MXH830, PXH830 or PXH820 adapter. Up to twenty-four systems can be connected to
the Dolphin MXS824 switch. All nodes connect directly to the switch using a single x4 MiniSAS-HD or PCIe 3.0 cables (x4
link).
Figure 1: 6 Node PCIe x16 Cluster
Figure 2: 12 Node PCIe x8 Cluster

MXS824 Users Guide –Dolphin Interconnect Solutions Page 8
10 node PCIe x16 Gen3 NTB Cluster
Each node has a Dolphin Express MXH830, PXH830 or PXH820 adapter. Up to
10 systems can be connected to the Dolphin MXS824 switch. All nodes
connect directly to the switch using up to 2 x4 MiniSAS-HD or PCIe 3.0 cables.
The two switches are interconnected using 4 x4 MiniSAS-HD or PCIe 3.0
cables (x16 link).
The cables between the two switches should be connected to port P1, P2, P3
and P4 on both switches.
20 node PCIe x8 Gen3 NTB Cluster
Each node has a Dolphin Express MXH830, PXH830 or PXH820 adapter. Up to
twenty systems can be connected to the Dolphin MXS824 switch. All nodes
connect directly to a switch using up to 2 x4 MiniSAS-HD or PCIe 3.0 cables.
The two switches are interconnected using 4 x4 MiniSAS-HD or PCIe 3.0
cables (x16 link).
The cables between the two switches should be connected to port P1, P2, P3
and P4 on both switches.
Larger NTB Cluster configurations
Firmware and software to support larger configurations are being developed and tested. Scalability up to 64/128 nodes is
planned, A general configuration example can be found below. Please contact Dolphin to learn about the latest test results
and release plans.
Figure 4: 20 Node PCIe x8 Cluster
Figure 3: 10 Node PCIe x16 Cluster

MXS824 Users Guide –Dolphin Interconnect Solutions Page 9