Magma ExpressBox EB3600-P User manual

EB3600-P
User Manual
PCIe to PCIe Expansion
MODEL
EB3600-P

Magma
EB3600-P |
2
Table of Contents
Preface ................................................................................................................................... 6
Advisories ......................................................................................................................................................................... 6
Safety Instructions........................................................................................................................................................... 7
Chapter 1 Introduction..................................................................................................... 9
EB3600-P........................................................................................................................................................................... 9
OVERVIEW........................................................................................................................................................................ 9
FEATURES .......................................................................................................................................................................... 9
BENEFITS............................................................................................................................................................................ 9
EB3600-P Technical Specification.............................................................................................................................. 10
Pre-Installation Information......................................................................................................................................... 11
Parts List .......................................................................................................................................................................... 11
Components of EB3600-P unit.................................................................................................................................... 12
Pre-Installation Information......................................................................................................................................... 14
Tools Required for Installation..................................................................................................................................... 14
Chapter 2 Hardware Installation .................................................................................... 15
Installation-Procedures Overview.............................................................................................................................. 15
Open Enclosure ............................................................................................................................................................ 16
Expansion Interface card Installation ....................................................................................................................... 16
Host Interface Card Installation................................................................................................................................. 19
Host Card installation in a Slot-Carrier housing ....................................................................................................... 21
Cable Installation ......................................................................................................................................................... 22
Connect cables to expansion unit....................................................................................................................... 23
Install Clamp ............................................................................................................................................................. 23
Connect cables to computer / server................................................................................................................. 24
Cable Configurations .................................................................................................................................................. 25
Configuration for EB3600-P ......................................................................................................................................... 26
Two Host computers attached ............................................................................................................................. 26
Four Host computers attached ............................................................................................................................. 26
PCIe Card Installation.................................................................................................................................................. 27
PCIe card Memory Requirements ........................................................................................................................ 28
PCIe card Power Requirements............................................................................................................................ 28
K80 Installation .............................................................................................................................................................. 29
Placement of GPUs on the backplane .............................................................................................................. 31
Connect Power Cord / Cable ................................................................................................................................... 34
Connecting to Electrical Outlets ............................................................................................................................... 34

Magma
EB3600-P |
3
Is it ok to connect four 1200W Power Supply modules to 120V? ................................................................... 35
Use two separate circuit lines................................................................................................................................ 36
Do not use multiple electrical outlets on one circuit line. ................................................................................ 36
Power on EB3600-P....................................................................................................................................................... 38
Power on Computer .................................................................................................................................................... 39
Hardware Check.......................................................................................................................................................... 39
Host & Expansion Interface cards LEDs................................................................................................................ 40
Check Expansion and Host Cards LEDs ............................................................................................................... 40
Check Front Panel Status LEDs .............................................................................................................................. 40
Check Backplane LEDs........................................................................................................................................... 41
Verify Installation .......................................................................................................................................................... 42
Windows .................................................................................................................................................................... 42
Linux ................................................................................................................................................................................ 43
Chapter 3 Advanced Technical Information .................................................................... 49
EB3600-P Chassis Layout.............................................................................................................................................. 49
Backplane and LEDs.................................................................................................................................................... 50
Card Slot Link LEDs........................................................................................................................................................ 51
Interface Card and LEDs............................................................................................................................................. 52
Reset & Power Status LEDs .......................................................................................................................................... 53
Display Functions of Interface card LEDs ................................................................................................................. 54
Interface card Dimensions ......................................................................................................................................... 54
Front Panel Status LEDs................................................................................................................................................ 55
Fan LEDs ......................................................................................................................................................................... 55
Temperature Sensor Alarm LEDs ............................................................................................................................... 55
Power Supply Indicator LEDs ...................................................................................................................................... 56
Control Switches........................................................................................................................................................... 56
SW1 and SW2............................................................................................................................................................ 57
SW3............................................................................................................................................................................. 57
SW4............................................................................................................................................................................. 57
Fan .................................................................................................................................................................................. 58
Fan Replacement.................................................................................................................................................... 59
Fan Specs.................................................................................................................................................................. 59
SNMP / Netburner......................................................................................................................................................... 60
NetBurner Module ........................................................................................................................................................ 60
Replacing the NetBurner........................................................................................................................................ 60
NetBurner Installation .............................................................................................................................................. 61
Replacing the Interface Board ............................................................................................................................. 62

Magma
EB3600-P |
4
NetBurner Interface Board Installation................................................................................................................. 63
Power Supply unit......................................................................................................................................................... 64
How to replace the power supply module........................................................................................................ 64
How to Add Power Module to EB3600................................................................................................................. 66
Power Supply Specifications.................................................................................................................................. 67
Chapter 4 Express I/O Manager ...................................................................................... 71
Web Server and SNMP Features ................................................................................................................................ 71
Web Access and SNMP Configuration..................................................................................................................... 71
Supported Internet Browser ................................................................................................................................... 71
To Access Express IO Manager.................................................................................................................................. 72
Connect Ethernet Cable........................................................................................................................................ 72
Configure Network Setting..................................................................................................................................... 72
How to configure your network setting in Windows OS .................................................................................... 72
How to configure your network setting in Linux / Ubuntu................................................................................. 73
Launch Internet Browser......................................................................................................................................... 74
Authentication Log-in ............................................................................................................................................. 74
Default Home Webpage ....................................................................................................................................... 75
Management Console / Web page ....................................................................................................................... 75
HOME ......................................................................................................................................................................... 75
ADMINISTRATOR ....................................................................................................................................................... 76
SNMP SETTINGS ......................................................................................................................................................... 76
ALARM SETTINGS....................................................................................................................................................... 77
NETWORK................................................................................................................................................................... 77
REMOTE MANAGEMENT.......................................................................................................................................... 78
PCI-E INFO ................................................................................................................................................................. 78
Reset the chassis to default parameters.................................................................................................................. 79
Retrieving the MIB File.................................................................................................................................................. 80
Accessing the various SNMP functions..................................................................................................................... 81
Chapter 5 Functional / Benchmark Tests ........................................................................ 82
SNMP Verification Test ................................................................................................................................................. 82
Replaceable Fan Test.................................................................................................................................................. 83
Power Supply Test......................................................................................................................................................... 83
Express IO Manager “radio button” Test.................................................................................................................. 84
Express IO Manager Administration Test .................................................................................................................. 84
Chapter 6 Rack Slide Installation .................................................................................... 85
Installing the EB3600-P Chassis in a Rack.................................................................................................................. 85
Rack Mounting ............................................................................................................................................................. 85

Magma
EB3600-P |
5
Rack Slides..................................................................................................................................................................... 85
Slide Rail Dimension Information................................................................................................................................ 85
Specifications ........................................................................................................................................................... 86
Safety Instructions......................................................................................................................................................... 87
Materials handling................................................................................................................................................... 87
Rack Stability ............................................................................................................................................................ 87
Tools and Supplies Needed........................................................................................................................................ 88
Parts List .......................................................................................................................................................................... 88
Slide Rail Installation Instructions ................................................................................................................................ 89
Prepare Slide Rails for Assembly (STEP 1) ............................................................................................................. 89
Attach Rail to Chassis (STEP 2) ............................................................................................................................... 90
Assemble Slide Rails (STEP 3) .................................................................................................................................. 91
Add Screws to Rack Post (STEP 4) ......................................................................................................................... 93
Attach Slide Rail to Rack (STEP 5) ......................................................................................................................... 93
Attach Chassis to Slide Rail / Rack (STEP 6)......................................................................................................... 94
Secure the Chassis to the Rack (STEP 7) .............................................................................................................. 96
Earth Leakage Current ........................................................................................................................................... 96
Grounding the EB3600-P Expansion Chassis........................................................................................................ 96
Regulatory Safety Requirements .......................................................................................................................... 97
Chapter 7 Troubleshooting ............................................................................................. 98
Locate the Problem..................................................................................................................................................... 98
Computer Hangs During Power Up........................................................................................................................... 98
Computer Doesn’t Detect Expansion Chassis ........................................................................................................ 99
Chapter 8 Chassis Maintenance .................................................................................... 100
General Chassis Cleaning ........................................................................................................................................ 100
Cleaning the Air Filter ................................................................................................................................................ 101
Chapter 9 Frequently Asked Questions (FAQ) ............................................................... 102
Chapter 10 How to Get More Help ................................................................................. 104
Contacting Technical Support ................................................................................................................................ 104
Returning Merchandise to MAGMA ....................................................................................................................... 104
Appendix A Compliance .................................................................................................................................... 105

Magma
EB3600-P | Preface
6
Preface
Advisories
Five types of advisories are used throughout this manual to provide helpful information, or to alert you to
the potential for hardware damage or personal injury.
NOTE
Used to amplify or explain a comment related to procedural steps or text.
IMPORTANT
Used to indicate an important piece of information or special “tip” to help you
CAUTION
Used to indicate and prevent the following procedure or step from causing damage to the
equipment.
WARNING
Used to indicate and prevent the following step from causing injury.
DANGER or STOP
Used to indicate and prevent the following step from causing serious injury or significant
data loss
Disclaimer: We have attempted to identify most situations that may pose a danger, warning, or caution
condition in this manual. However, Magma does not claim to have covered all situations that might require
the use of a Caution, Warning, or Danger indicator.

Magma
EB3600-P | Preface
7
Safety Instructions
Always use caution when servicing any electrical component. Before handling the Magma Expansion
chassis, read the following instructions and safety guidelines to prevent damage to the product and to
ensure your own personal safety. Refer to the “Advisories” section for advisory conventions used in this
manual, including the distinction between Danger, Warning, Caution, Important, and Note.
Always use caution when handling/operating the computer. Only qualified, experienced, authorized
electronics personnel should access the interior of the computer and expansion chassis.
WARNING
Never modify or remove the radio frequency interference shielding from your workstation or
expansion unit. To do so may cause your installation to produce emissions that could
interfere with other electronic equipment in the area of your system.
When Working Inside a Computer
1. Before taking covers off a computer, perform the following steps:
2. Turn off the computer and any peripheral devices.
3. Disconnect the computer and peripheral power cords from their AC outlets or inlets in order to
prevent electric shock or system board damage.
In addition, take note of these safety guidelines when appropriate:
To help avoid possible damage to systems boards, wait five seconds after turning off the computer before
removing a component, removing a system board, or disconnecting a peripheral device from the
computer.
When you disconnect a cable, pull on its connector or on its strain-relief loop, not on the cable itself. Some
cables have a connector with locking tabs. If you are disconnecting this type of cable, press in on the
locking tabs before disconnecting the cable. As you pull connectors apart, keep them evenly aligned to
avoid bending any connector pins. Also, before connecting a cable, make sure both connectors are
correctly oriented and aligned.
CAUTION
Do not attempt to service the system yourself except as explained in this manual. Follow
installation instructions closely.

Magma
EB3600-P | Preface
8
Protecting Against Electrostatic Discharge
Electrostatic Discharge (ESD) Warning
Electrostatic Discharge (ESD) is the enemy of semiconductor devices. You should always
take precautions to eliminate any electrostatic charge from your body and clothing before
touching any semiconductor device or card by using an electrostatic wrist strap and/or
rubber mat.
Static electricity can harm system boards. Perform service at an ESD workstation and follow proper ESD
procedures to reduce the risk of damage to components. Magma strongly encourages you to follow
proper ESD procedures, which can include wrist straps and smocks, when servicing equipment.
You can also take the following steps to prevent damage from electrostatic discharge (ESD):
When unpacking a static-sensitive component from its shipping carton, do not remove the component’s
anti-static packaging material until you are ready to install the component in a computer. Just before
unwrapping the anti-static packaging, be sure you are at an ESD workstation or are grounded.
When transporting a sensitive component, first place it in an anti-static container or packaging.
Handle all sensitive components at an ESD workstation. If possible, use anti-static floor pads and workbench
pads.
Handle components and boards with care. Do not touch the components or contacts on a board. Hold a
board by its edges or by its metal mounting bracket.

Magma
EB3600-P | Chapter 1 Introduction
9
Chapter 1 Introduction
EB3600-P
Magma’s ExpressBox 3600-P provides partitioned support for server grade GPUs and other HPC peripherals
to existing host computer systems. Designed to support full Gen 3 x16 PCIe connectivity to all devices, the
EB3600-P supports 128 Gbps communication between the host and all peripherals. The EB3600-Psupports
eight total peripherals with four host links for support of one to four independent hosts.
OVERVIEW
EB3600-P-P: Eight total peripherals; one to four hosts
PCIe Gen 3 x16 (128Gbps) for all connections
Optimized for server class (passively coded) GPUs
Magma Express I/O System Management is included in all configurations
FEATURES
Dedicated connectivity for up to four independent hosts
Passive host interface; no PCIe switch on host link cards provides lowest latency possible.
Two 1200W/1500W hot swappable power supplies provided standard. Up to two, more can be
configured, for support of up to 4800W/6000W.
Direct flow through air-cooling is provided for passively cooled server class GPUs.
BENEFITS
Design optimized for GPU applications - concentrate your HPC investment in a chassis that will
support full GPU performance utilization.
Each partition supports better ratio of Peripheral: Host connection of 2:1.
Low-profile host card allows for easy installation into low-profile computers.
Supports peer-to-peer transfers between cards in the expansion chassis to provide full bandwidth
potential among I/O cards.
Supports daisy-chaining of fan-out configuration of multiple expansion chassis.

Magma
EB3600-P | Chapter 1 Introduction
10
EB3600-P Technical Specification
BASE MODEL
EB3600-P
-Two partitioned backplanes composed of two segments per backplane with
two peripheral slots per segment. Four total segments, eight total peripherals;
Supports up to 8 GPUs.
- Two 1200W (2400W total) PSUs
TECHNOLOGY
PCI Express Bus Specification
PCI Express Bus Specification 3.0, 2.3; | PCI Bridge Architecture Revision 1.2
HARDWARE
Backplane
Partitioned backplane: Four segments with two peripheral slots (GPUs) per
segment; Four host links (1 per segment) (All Gen 3 x16).
Interconnect Bandwidth
PCIe Gen 3 x16: 128 Gbits/sec to all peripherals and host link
ENCLOSURE
Material
Steel Chassis
Dimension
16.8” Wide by 7” High (4U) by 22.125” Deep | 50lbs/2.7kg
HOST CONNECTION
Host side: Any PCIe based host-
equipped computer (preferably Gen 3
x16)
Expansion side:
All host links are Gen 3 x16 capable and will train down to Gen 2 or 1 as
needed.
Switchless host link adapter is designed
for minimal host/expansion latency 1m
or 2m cable lengths are supported for
host interface.
POWER SUPPLY
1200W PSU
1200W@120V/ 1500W@240V AC power PSU. Can be added to base models
above. Up to four total (4800W@120V/ 6000W@240V)
ENVIRONMENT
Ambient Temperature
0º to 50º C
Storage Temperature
-55º to 125º C
Relative Humidity
0% to 90% non-condensing
REGULATORY COMPLIANCE
FCC Class A Verified
RoHS Compliant
CE Certified
SUPPORTED OPERATING SYSTEMS
Windows
Mac OS
Linux