Dolphin MXH94 Series User manual

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 1
Dolphin
MXH94x / MXH95x NTB Adapter
Users Guide
Version 1.0
Date: 20th January 2021

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 2

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 3
Table of Contents
DISCLAIMER................................................................................................................................................................ 4
TERMS AND ACRONYMS............................................................................................................................................. 4
HIGH LEVEL SPECIFICATION ........................................................................................................................................ 5
PCIE GEN4LINK NTB PRODUCT VARIANTS......................................................................................................................... 5
PCIE GEN3LINK NTB PRODUCT VARIANTS......................................................................................................................... 5
DETAILED SPECIFICATIONS............................................................................................................................................... 5
MXH94x Maximum DC current across input voltage ranges................................................................................... 6
MXH95x Maximum DC current across input voltage ranges................................................................................... 6
PACKAGING................................................................................................................................................................ 7
PRE-INSTALLATION QUESTIONS.................................................................................................................................. 7
PCIE SLOT DETERMINATION............................................................................................................................................ 7
ADAPTER CARD CONFIGURATION...................................................................................................................................... 7
OPERATING ENVIRONMENT ............................................................................................................................................ 7
CABLE CONNECTIONS .................................................................................................................................................... 7
INSTALLATION ............................................................................................................................................................ 8
STEP 1-UNPACK BOARD................................................................................................................................................ 8
STEP 3-CONFIGURE THE BOARD FOR PROPER OPERATION ..................................................................................................... 8
STEP 4-INSTALL THE ADAPTER CARD ................................................................................................................................ 8
STEP 5-INSTALLING AND REMOVING CABLES...................................................................................................................... 8
Connecting Cables ................................................................................................................................................ 8
Disconnecting Cables............................................................................................................................................ 8
STEP 6-INSTALLING THE DOLPHIN EXPRESSWARE SOFTWARE................................................................................................. 8
STEP 7–VERIFY INSTALLATION &LEDS............................................................................................................................. 9
OPERATION ...............................................................................................................................................................10
CONFIGURATION AND DIP SWITCHES ...............................................................................................................................10
DIP Switch Bank –Configuration..........................................................................................................................10
DIP Switch settings for NTB operation..................................................................................................................10
USE CASE A-2NODE CONFIGURATION –NTB MODE .........................................................................................................11
Connecting the cables for 2 Node Configurations, x16 link....................................................................................11
EEPROM AND FIRMWARE UPGRADE...............................................................................................................................12
IDENTIFYING THE CARD .................................................................................................................................................12
SUPPORT...................................................................................................................................................................13
TECHNICAL INFORMATION ........................................................................................................................................14
BOARD REVISION HISTORY..............................................................................................................................................14
SIMPLIFIED SCHEMATICS................................................................................................................................................14
PENDING COMPLIANCE AND REGULATORY TESTING .................................................................................................15
LIMITED WARRANTY..................................................................................................................................................16
WARRANTY PERIOD .....................................................................................................................................................16
COVERAGE.................................................................................................................................................................16
SERVICE PROCEDURE ....................................................................................................................................................16

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 4
DISCLAIMER
DOLPHIN INTERCONNECT SOLUTIONS RESERVES THE RIGHT TO MAKE CHANGES WITHOUT FURTHER NOTICE TO ANY OF ITS
PRODUCTS TO IMPROVE RELIABILITY, FUNCTION, OR DESIGN.
TO THE FULLEST EXTENT PERMITTED BY LAW, DOLPHIN WILL NOT BE LIABLE FOR ANY INDIRECT, INCIDENTAL, SPECIAL OR
CONSEQUENTIAL DAMAGES (INCLUDING LOST PROFITS, LOST DATA, OR LOSS OF USE) ARISING OUT OF ANY USE OF DOLPHIN’S
PRODUCTS, SOFTWARE OR SERVICE PROVIDED. DOLPHIN’S MAXIMUM LIABILITY WILL NOT EXCEED THE TOTAL AMOUNT PAID FOR
THE PRODUCT BY THE PURCHASER.
LIFE SUPPORT POLICY
DOLPHIN INTERCONNECT SOLUTIONS’ PRODUCTS ARE NOT AUTHORIZED FOR USE AS CRITICAL COMPONENTS IN LIFE SUPPORT
DEVICES.
ENVIRONMENTAL POLICY
Dolphin is minimizing the amount of printed documentation and software CDs in its shipments; please download additional
documentation and software from www.dolphinics.com.
Terms and acronyms
Important terms and acronyms used in this manual
BMC Board Management Microcontroller on the adapter card.
eXpressWare Dolphins software stack for PCIe clustering and IO. Please visit http://www.dolphinics.com/software
for more information.
FireFly™ Optical fiber solution for PCIe over cable from Samtec Inc
Transparent Host mode The card is installed in a root complex / Host PC and configured for Transparent operation.
Transparent Hot Add Attaching an IO Expansion system after the server is booted.
Target mode The card is installed in an expansion chassis or backplane for Transparent IO expansion.
Lane One PCI Express Lane contains a differential pair for transmission and a differential pair for reception.
Link A collection of one or more PCI Express Lanes providing the communication path between an
Upstream and Downstream Port.
NTB mode The card is configured for Non Transparent Bridge operations. Requires Dolphin eXpressWare or
custom NTB drivers.
Port PCIe Cable port. The adapter card, depending on the model, has up four x4 ports, named P1, P2, P3,
P4 or up to two x8 ports, named P1 and P2. The physical ports are identified by text on the PCIe
brackets.

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 5
High Level Specification
The MXH94x and MXH95x are a family of low profile, half-length PCIe adapters utilizing the Samtec FireFly™ optical transceivers. It
is available in several variants to support various use cases and configurations. All boards are using a Gen4 PCIe switch from
Microchip. The MXH94x boards are equipped with Samtec FireFly™ PCIe Gen4 optical transceivers. The MXH95x boards are
equipped with the Samtec FireFly™ PCIe Gen3 optical transceivers.
Some of these variants come with NTB enabled firmware and are bundled with Dolphins eXpressWare. Please reference the tables
below for details. This manual covers all NTB enabled variants. Please reference the Dolphin
MXH94x / MXH95x Transparent Adapter Users Guide if you have one of the Transparent only variants.
The cards operate at the highest common speed shared between the slot and the card (Gen4) and the widest common link-width
(x16).
PCIe Gen4 Link NTB Product variants
The variants in the table below utilize Gen4 FireFly™ modules, support NTB configurations and come with a full eXpressWare
software license. Each Model has different FireFly™ optical engines mounted depending on your use case and requirements:
SKU / Model
#FireFly™
Config
Required Fiber cable
MXH940
2 x8
NTB x16 / NTB 2 x8
2 units 24 fiber MPO
MXH941
1 x8
NTB x8
1 unit 24 fiber MPO
MXH943
2 x4
NTB x8 / NTB 2 x4
2 units 12 fiber MPO
MXH944
1 x4
NTB x4
1 unit 12 fiber MPO
MXH949
4 x4
NTB x16, NTB 2 x8, NTB 4x 4
4 units 12 fiber MPO
NOTE: Please note that cards utilizing 24 fiber MPO cannot be connected to cards utilizing 12 fiber MPO.
PCIe Gen3 Link NTB Product variants
The variants in the table below utilize the Gen3 FireFly™ modules, support NTB configurations and come with a full eXpressWare
software license. Each Model has different FireFly™ optical engines mounted depending on your use case and requirements:
SKU / Model
#FireFly™
Config
Required Fiber cable
MXH950
2 x8
NTB x16 / NTB 2 x8
2 units 24 fiber MPO
MXH951
1 x8
NTB x8
1 unit 24 fiber MPO
MXH953
2 x4
NTB x8 / NTB 2 x4
2 units 12 fiber MPO
MXH954
1 x4
NTB x4
1 unit 12 fiber MPO
MXH959
4 x4
NTB x16, NTB 2 x8, NTB 4x 4
4 units 12 fiber MPO
NOTE: Please note that cards utilizing 24 fiber MPO cannot be connected to cards utilizing 12 fiber MPO.
Detailed specifications
•PCI Express Base Specification, Rev. 4.0.
•PCI Express CEM Specification, Rev. 4.0.
•PCI Express Gen4 16.0 GT/s per lane signaling –256 GT/s total signaling. Variants utilizing Gen3 FireFly™ modules
are limited to PCIe Gen3 8 GT/S signaling rate on the optical link.
•PCI Express Gen4 x16 edge connector. The card installs in any PCI Express slot that has a physical x16 connector.
•Compliant with PCI Express Gen1 through Gen4 computers and IO systems, auto detection.
•Cable port configurations (depends on SKU number)
oOne x16 NTB port
oTwo x8 NTB ports
oFour x4 NTB ports
oOne x8 NTB port
oTwo x4 NTB ports
oOne x4 NTB port
•MPO cable connector
•Microchip / Microsemi Switchtec™PM40036B PFX PCI Express Gen4 chipset. (PM40052B used in Rev A and B cads)

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 6
•Pre-loaded firmware configurations, DIP switch selectable. Please reference the adapter card firmware release note
for details.
•100 nanosecond cut-through latency port to port.
•PCIe DMA engine for optimized transfers.
•Support for optical patch cables up to 100 meters.
•Low profile, Half-length - PCI Express Electromechanical Specification, Rev 4.0.
•Dimensions 167.65mm (6.600 inches) x 68.90 mm (2.731 inches).
•Host clock isolation. Automatic support for host running CFC or SSC mode.
•VAUX powered board management controllers for flexible configuration and cable management.
•Flash recovery option. PFX Multi configuration support.
•Operating Temperature: 0°C - 55°C (32°F - 131°F), Air Flow: 700 LFM
•Relative Humidity: 5% - 95% (non- condensing)
•Regulatory
oRoHS
oEMC testing for CE and FCC in progress
•One year warranty
MXH94x Maximum DC current across input voltage ranges
MXH94X
#FireFly™
12V
3.3V
VAUX
1
1.97 A
0.84 A
0.35 A
2
2.30 A
0.84 A
0.35 A
3
2.62 A
0.84 A
0.35 A
4
2.95 A
0.84 A
0.35 A
MXH95x Maximum DC current across input voltage ranges
MXH95X
#FireFly™
12V
3.3V
VAUX
1
1.76 A
0.84 A
0.35 A
2
1.88 A
0.84 A
0.35 A
3
2.00 A
0.84 A
0.35 A
4
2.12 A
0.84 A
0.35 A
Note:
25 W Slot specification for +12V rail is 2.1 A max.
75 W slot specification for +12 V rail is 5.5 A max.

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 7
Packaging
The products are delivered includes the following components.
•Adapter Board with pre mounted FireFly™ modules and standard profile bracket.
•Anti-static bag
•Getting started guide with serial number for quick download of Dolphins PCIe software.
Pre-Installation Questions
Certain steps should be taken prior to installing the adapter card. You should determine the following configuration requirements.
•Which PCIe slot in the system will the card be installed?
•What is the speed and link width of the installation slot?
•What is the operating environment around the installed card?
•What quantity, type, and length of patch cable will be used?
•Are the operational conditions proper - temperature and air-flow?
PCIe Slot Determination
The adapter card supports PCIe Gen1, Gen2, Gen3 and Gen4 speeds. The adapter card must be installed in physical x16 connector
but supports x1, x2, x4, x8 and x16 electrical slot-widths. The slot width and speed will affect the performance of the adapter card.
The adapter card will auto configure to the slot speed and width.
NOTE:
•Install the adapter card in a slot connecting directly to the CPU for optimal shared memory performance.
•Install the adapter card behind a PCIe switch in the IO system for optimal PCIe peer to peer performance (e.g. between
GPUs, FPGAs).
TIP: After installing the eXpressWare software, run the “sisci_benchmarks” benchmark suite to verify the expected performance.
Please contact Dolphin support if you need assistance to reach the expected performance.
Adapter card Configuration
The adapter card has a DIP switch bank to control the main configuration of the card. The DIP switch labeled SW1 can be found
close to the upper edge of the board. The default DIP switch setting is NTB Host operation, single, full width link operations.
Operating Environment
To maximize lifespan for the product and maintain the warranty, please honor the specified operating temperature and make sure
the specified air flow is present. Special care should be considered when the adapter card is used in office type cabinets in
combination with other high energy consuming PCIe devices, e.g. not active cooled GPUs:
•Operating Temperature: 0°C - 55°C (32°F - 131°F), Air Flow: 700 LFM
•Relative Humidity: 5% - 95% (non- condensing)
TIP: After installing the Dolphin eXpressWare software, you can use the tool dis_diag to determine the actual board temperatures
Cable Connections
The adapter cards are designed for 12 or 24 fiber MPO optical patch cables. Please carefully select the 12 or 24 fiber path cables
specified for each product variant.
NOTE: Please note that all cards utilizing 24 fiber MPO cannot be connected to cards utilizing 12 fiber MPO.

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 8
Installation
Step 1 - Unpack board
The adapter card is shipped in an anti-static bag to prevent static electricity damage. The card should only be removed from the
bag after ensuring that anti-static precautions are taken. Static electricity from your clothes or work environment can
damage your PCI Express adapter card or your PC. Always wear a grounded anti-static wrist strap while opening the
PC and when the adapter card is removed from the anti-static bag.
Unpack the adapter card from the anti-static bag using proper anti-static procedures.
Step 3 - Configure the Board for Proper Operation
Set the DIP switch settings for proper operation depends on the firmware. Please refer to the section Configuration and DIP
Switches on page 10 for details.
Step 4 - Install the Adapter Card
Before installing the adapter card, make sure you are properly grounded to avoid static discharges that may destroy your computer
or the adapter card. Ensure you are properly grounded before opening your computer or the anti-static bag
containing the adapter card. Please follow your computer’s or expansion chassis manual on how to install a PCI
Express card.
The adapter card can only be installed into any PCI Express x16 physical slot. The adapter supports PCI Express Gen1,
Gen2, Gen3 and Gen4 signaling. The adapter supports hosts using either spread spectrum or constant frequency clocking.
The card implements clock isolation.
Step 5 - Installing and Removing Cables
Installing and removing cables in NTB or Transparent Hot Add mode can be done while the systems are running. Hot
plugging / removing cables are fully supported. The fiber optic cable(s) are connected to the MPO connector(s) found on
the PCIe bracket. The bracket will have one or two, single or dual MPO connectors depending on the board configuration.
Connecting Cables
Please carefully install the fiber patch cable connector into the connector housing on the adapter card. To install the
cable, remove the protective cover on both the cable and the card, match the cable house keying with the MPO
connector on the adapter card and use light pressure to insert the connector until it is latched. Cables should always use
strain relief to protect the connected equipment from excessive force on the cable. This is especially important for cables
between racks. Please always save the protective covers for future use if the cable is disconnected from the card.
Disconnecting Cables
Please carefully pull the release tab to release the cable from the locking. Always replace the protective cover on both the
card and the cable after disconnecting the cable.
Step 6 - Installing the Dolphin eXpressWare Software
The adapter card is compliant with Dolphin’s eXpressWare software package for the MX product line. Please visit
http://www.dolphinics.com/mx
to register and download the latest documentation and software. It is recommended to follow the quick installation guide found
on the web page above to install the software for your operating system.
Note to PXH84x users:
If you are migrating from any of the PCIe Gen3 PXH84x adapter cards to the MXH94x or MXH95x cards, you need to uninstall the PX
software and install the MX software.

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 9
TIP: Dolphin provides software and documentation for several product families; please remember to select the MX product family
before downloading. The adapter card requires Dolphin software version DIS 5.18.0 or higher to operate.
The software download requires a password to log in. The password will automatically be emailed to you if you follow the
instructions found on the getting started document bundled with the adapter card. If you fail to provide the correct serial number
found on the getting started document, your request will be managed manually.
Step 7 –Verify Installation & LEDs
The adapter card comes with 4 bi-color LEDs which show the corresponding cable port status according to Table 1: LED below.
The LEDs are visible through cut-outs in the PCIe bracket on each side of the cable connector block.
LED color
Function
Off
No cable installed
Yellow
Cable installed, no link
Green
Cable installed, link operational
Table 1: LED behavior

MXH94x / MXH95x NTB Adapter User’s Guide – Dolphin Interconnect Solutions Page 10
Operation
Configuration and DIP Switches
The adapter card has one bank of 8 DIP switches for setting special modes or operations, the meaning of each DIP switch depends
on the loaded firmware. Please carefully read the documentation shipping with the card before modifying any DIP switch settings.
Please pay close attention to ON and OFF positions written on the DIP switch.
The default factory setting for the NTB enabled adapter cards is a single NTB link connection suitable for a two-node configuration.
DIP Switch Bank –Configuration
Figure 1: DIP Switch shows the configuration DIP switch for the adapter card. Please leave all
undocumented DIP switches in the default position. Table 2: DIP Switch settings shows all
the various DIP switch NTB settings for the adapter card.
DIP no.
Name
Description
ON
OFF
Default
1-5
OPT1-5
Configuration selector, details
below
Opt4: ON
6
OPT6
OFF
7
URES
Holds the management processors
in reset
Board management is
held in reset
Normal operation
OFF
8
SAFE
Enables the card to boot if the
EEPROM has been corrupted
Safe EEPROM
Normal operation
OFF
Table 2: DIP Switch settings
Note: DIP switch configuration options may be changed in the future versions. Please always consult the latest user
guide for details. The table above is valid for firmware version 1 prerelease.
NTB Operation
The default configuration is NTB operation.
DIP Switch settings for NTB operation
The following DIP Switch settings should be considered when configuring the adapter card for NTB operation:
Use Case
Configuration Transparent Host
DIP ON
DIP switch view
A
NTB Host
One Link
256MB BAR
(P1+P2+P3+P4)
OPT4
(Shipping
Default)
A
Transparent Host
Two x8 downstream ports
32GB BAR
(P1+P2, P3+P4)
OPT1
+
OPT4
Figure 1: DIP Switch