Magma EB3600-AB User manual

EB3600-AB
User Manual
PCIe to PCIe Expansion
MODEL
EB3600-AB

Magma
EB3600-AB |
2
Table of Contents
Preface.................................................................................................................................. 7
Advisories ......................................................................................................................................................................... 7
Safety Instructions........................................................................................................................................................... 8
Chapter 1 Introduction.................................................................................................... 10
EB3600-AB Features...................................................................................................................................................... 11
Technical Specification............................................................................................................................................... 11
The core of ExpressBox 3600....................................................................................................................................... 12
Block Diagram .............................................................................................................................................................. 13
Configuration Model ................................................................................................................................................... 13
Parts List .......................................................................................................................................................................... 15
Components ................................................................................................................................................................. 16
Pre-Installation Information......................................................................................................................................... 18
Tools Required for Installation..................................................................................................................................... 18
Chapter 2 Hardware Installation ..................................................................................... 19
Installation-Procedures Overview.............................................................................................................................. 20
Open Enclosure ............................................................................................................................................................ 21
Install Expansion Interface Card................................................................................................................................ 21
Install Host Interface Card .......................................................................................................................................... 24
Host Card installation in a Slot-Carrier ...................................................................................................................... 26
Connect Interface Cables ......................................................................................................................................... 28
Proper handling of cables ..................................................................................................................................... 28
How to insert the cable correctly ......................................................................................................................... 30
Connect cables to expansion unit....................................................................................................................... 32
Secure all the cables with a retainer clamp. ..................................................................................................... 32
Connect cables to computer / server................................................................................................................. 34
Proper Placement of Cables ................................................................................................................................. 35
Standard Cable vs. Daisy Chain Configurations................................................................................................ 35
Install PCIe card / GPU ................................................................................................................................................ 36
Installation of Singe-wide GPU .............................................................................................................................. 36
Installation of Double-wide GPU ........................................................................................................................... 36
Connect Auxiliary power........................................................................................................................................ 38
PCIe card Memory Requirements ........................................................................................................................ 39
PCIe card Power Requirements............................................................................................................................ 40
K-80 / GPU Installation ................................................................................................................................................. 41
Connect Auxiliary power adapter ....................................................................................................................... 43

Magma
EB3600-AB |
3
Double-wide GPUs vs. Single-slot GPUs................................................................................................................ 44
Connect Power Cord / Cable ................................................................................................................................... 45
Connect to Electrical Outlets..................................................................................................................................... 45
Is it ok to connect four 1200W Power Supply modules to 120V? .................................................................... 45
Use two separate circuit lines................................................................................................................................ 46
Do not use multiple electrical outlets on one circuit line ................................................................................. 46
Power ON the expansion unit .................................................................................................................................... 48
Power ON Computer................................................................................................................................................... 49
Hardware Check.......................................................................................................................................................... 49
Host & Expansion Interface cards LEDs................................................................................................................ 50
Front Panel Status LEDs ........................................................................................................................................... 51
Backplane LEDs........................................................................................................................................................ 51
Completing Hardware Set up .............................................................................................. 52
Chapter 3 Software Installation ..................................................................................... 54
Install Software.............................................................................................................................................................. 54
Verify Installation .......................................................................................................................................................... 54
Windows .................................................................................................................................................................... 54
Linux............................................................................................................................................................................ 55
Check / Verify PCIe cards / GPU............................................................................................................................... 61
Chapter 4 General & Technical Information .................................................................... 62
EB3600-AB ...................................................................................................................................................................... 62
Placement of GPUs on the expansion backplane ................................................................................................ 63
EB3600-AB with 10 “single-wide” GPUs................................................................................................................ 64
EB3600-AB with 9 “double-wide” GPUs................................................................................................................ 64
Proper Placement of GPUs .................................................................................................................................... 65
Backplane PCIe slots and Ports assignment............................................................................................................ 69
Backplane and LEDs.................................................................................................................................................... 70
Card Slot Link LEDs........................................................................................................................................................ 71
Interface Card and LEDs............................................................................................................................................. 72
Display Functions of Interface card LEDs................................................................................................................. 73
Reset & Power Status LEDs.......................................................................................................................................... 74
Front Panel Status LEDs................................................................................................................................................ 75
Fan LEDs..................................................................................................................................................................... 75
Temperature Sensor Alarm LEDs........................................................................................................................... 75
Power Supply Indicator LEDs.................................................................................................................................. 76
Control Switches........................................................................................................................................................... 76
SW1 and SW2............................................................................................................................................................ 77

Magma
EB3600-AB |
4
SW3............................................................................................................................................................................. 77
SW4............................................................................................................................................................................. 77
Fan .................................................................................................................................................................................. 78
Fan Specs.................................................................................................................................................................. 78
Fan Replacement.................................................................................................................................................... 79
SNMP / Netburner......................................................................................................................................................... 80
NetBurner Module ........................................................................................................................................................ 80
Replacing the NetBurner........................................................................................................................................ 80
Netburner Installation.............................................................................................................................................. 82
Replacing the Interface Board ............................................................................................................................. 83
Netburner Interface Board Installation ................................................................................................................ 84
Power Supply unit......................................................................................................................................................... 85
How to replace the power supply module........................................................................................................ 85
How to Add Power Module to EB3600................................................................................................................. 87
Power Supply Specifications.................................................................................................................................. 87
Chapter 5 Management Console I/O Manager ................................................................. 91
SNMP Features .............................................................................................................................................................. 91
Web Access and SNMP Configuration..................................................................................................................... 91
Supported Internet Browser ................................................................................................................................... 91
To Access Express IO Manager / SNMP.................................................................................................................... 92
Connect Ethernet Cable........................................................................................................................................ 92
Configure Network Setting..................................................................................................................................... 92
How to configure your network setting in Windows OS.................................................................................... 92
How to configure your network setting in Linux / Ubuntu................................................................................. 93
Launch Internet Browser......................................................................................................................................... 94
Authentication Log-in ............................................................................................................................................. 94
Default Home Webpage ....................................................................................................................................... 95
Management Console / Web GUI........................................................................................................................... 95
HOME ......................................................................................................................................................................... 95
ADMINISTRATOR ....................................................................................................................................................... 96
SNMP SETTINGS ......................................................................................................................................................... 96
ALARM SETTINGS....................................................................................................................................................... 97
NETWORK................................................................................................................................................................... 97
REMOTE MANAGEMENT.......................................................................................................................................... 98
PCI-E INFO ............................................................................................................................................................... 101
Servicing the Side A or Side B.................................................................................................................................. 104
Turning OFF 3.3v when servicing the chassis ....................................................................................................... 104

Magma
EB3600-AB |
5
Deactivating Server A/B (side A/B) .................................................................................................................... 105
Reactivating Server A/B (side A/B)..................................................................................................................... 106
Fan Control ............................................................................................................................................................. 107
Fan Configuration.................................................................................................................................................. 108
Reset the chassis to default parameters................................................................................................................ 109
Pressing the front power push button ................................................................................................................ 109
Flashing the Netburner.......................................................................................................................................... 109
Configure your SNMP Agent .................................................................................................................................... 110
Retrieving the MIB File................................................................................................................................................ 110
Accessing the various SNMP functions................................................................................................................... 110
Chapter 6 Functional / Benchmark Tests ...................................................................... 112
SNMP Verification Test ............................................................................................................................................... 112
Replaceable Fan Test................................................................................................................................................ 113
Replaceable Power Supply Test .............................................................................................................................. 113
Express IO Manager “radio button” Test................................................................................................................ 114
Express IO Manager Administration Test ................................................................................................................ 114
Chapter 7 Rack Slide Installation .................................................................................. 115
Installing the EB3600 Chassis in a Rack ................................................................................................................... 115
Rack Mounting ........................................................................................................................................................... 115
Rack Slides................................................................................................................................................................... 115
Slide Rail Dimension Information.............................................................................................................................. 115
Specifications ......................................................................................................................................................... 116
Safety Instructions....................................................................................................................................................... 117
Rack Stability .......................................................................................................................................................... 117
Tools and Supplies Needed...................................................................................................................................... 118
Parts List ........................................................................................................................................................................ 118
Slide Rail Installation Instructions .............................................................................................................................. 120
Prepare Slide Rails for Assembly (STEP 1) ........................................................................................................... 120
Attach Rail to Chassis (STEP 2)............................................................................................................................. 121
Assemble Slide Rails (STEP 3) ................................................................................................................................ 121
Add Screws to Rack Post (STEP 4)....................................................................................................................... 123
Attach Slide Rail to Rack (STEP 5) ....................................................................................................................... 123
Attach Chassis to Slide Rail / Rack (STEP 6)....................................................................................................... 125
Secure the Chassis to the Rack (STEP 7) ............................................................................................................ 126
Earth Leakage Current ......................................................................................................................................... 126
Grounding the EB3600-AB Expansion Chassis ....................................................................................................... 127
Regulatory Safety Requirements ........................................................................................................................ 127

Magma
EB3600-AB |
6
Chapter 8 Updating Netburner / SNMP ......................................................................... 129
How to Update the Netburner on Linux ................................................................................................................. 129
Requirements.......................................................................................................................................................... 129
Procedures Overview............................................................................................................................................ 129
Download Files ...................................................................................................................................................... 129
Connect the Ethernet Cable ................................................................................................................................... 129
Launch the AutoUpdate.jar ..................................................................................................................................... 130
Check and verify the update .................................................................................................................................. 132
Chapter 9 Troubleshooting ............................................................................................ 133
Chapter 10 Chassis Maintenance ................................................................................... 135
General Chassis Cleaning ........................................................................................................................................ 135
Cleaning the Air Filter ................................................................................................................................................ 136
Chapter 11 Frequently Asked Questions (FAQ) ............................................................. 137
What are the differences between EB3600-10 and EB3600-AB? .................................................................. 139
Chapter 12 How to Get More Help ................................................................................. 140
Contacting Technical Support ................................................................................................................................ 140
Returning Merchandise to MAGMA ....................................................................................................................... 140
Online Support Resources......................................................................................................................................... 141
Appendix A Compliance .................................................................................................................................... 142

Magma
EB3600-AB | Preface
7
Preface
Advisories
Five types of advisories are used throughout this manual to provide helpful information, or to alert you to
the potential for hardware damage or personal injury.
NOTE
Used to amplify or explain a comment related to procedural steps or text.
IMPORTANT
Used to indicate an important piece of information or special “tip” to help you
CAUTION
Used to indicate and prevent the following procedure or step from causing damage to the
equipment.
WARNING
Used to indicate and prevent the following step from causing injury.
DANGER or STOP
Used to indicate and prevent the following step from causing serious injury or significant
data loss
Disclaimer: We have attempted to identify most situations that may pose a danger, warning, or caution
condition in this manual. However, Magma does not claim to have covered all situations that might require
the use of a Caution, Warning, or Danger indicator.

Magma
EB3600-AB | Preface
8
Safety Instructions
Always use caution when servicing any electrical component. Before handling the Magma Expansion
chassis, read the following instructions and safety guidelines to prevent damage to the product and to
ensure your own personal safety. Refer to the “Advisories” section for advisory conventions used in this
manual, including the distinction between Danger, Warning, Caution, Important, and Note.
Always use caution when handling/operating the computer. Only qualified, experienced,
authorized electronics personnel should access the interior of the computer and expansion
chassis.
WARNING
Never modify or remove the radio frequency interference shielding from your workstation or
expansion unit. To do so may cause your installation to produce emissions that could
interfere with other electronic equipment in the area of your system.
When Working Inside a Computer
1. Before taking covers off a computer, perform the following steps:
2. Turn off the computer and any peripheral devices.
3. Disconnect the computer and peripheral power cords from their AC outlets or inlets in order to
prevent electric shock or system board damage.
In addition, take note of these safety guidelines when appropriate:
To help avoid possible damage to systems boards, wait five seconds after turning off the computer
before removing a component, removing a system board, or disconnecting a peripheral device
from the computer.
When you disconnect a cable, pull on its connector or on its strain-relief loop, not on the cable
itself. Some cables have a connector with locking tabs. If you are disconnecting this type of cable,
press in on the locking tabs before disconnecting the cable. As you pull connectors apart, keep
them evenly aligned to avoid bending any connector pins. Also, before connecting a cable,
make sure both connectors are correctly oriented and aligned.
CAUTION
Do not attempt to service the system yourself except as explained in this manual. Follow
installation instructions closely.

Magma
EB3600-AB | Preface
9
Protecting Against Electrostatic Discharge
Electrostatic Discharge (ESD) Warning
Electrostatic Discharge (ESD) is the enemy of semiconductor devices. You should always
take precautions to eliminate any electrostatic charge from your body and clothing before
touching any semiconductor device or card by using an electrostatic wrist strap and/or
rubber mat.
Static electricity can harm system boards. Perform service at an ESD workstation and follow proper ESD
procedures to reduce the risk of damage to components. Magma strongly encourages you to follow
proper ESD procedures, which can include wrist straps and smocks, when servicing equipment.
You can also take the following steps to prevent damage from electrostatic discharge (ESD):
When unpacking a static-sensitive component from its shipping carton, do not remove the
component’s anti-static packaging material until you are ready to install the component in a
computer. Just before unwrapping the anti-static packaging, be sure you are at an ESD
workstation or are grounded.
When transporting a sensitive component, first place it in an anti-static container or packaging.
Handle all sensitive components at an ESD workstation. If possible, use anti-static floor pads and
workbench pads.
Handle components and boards with care. Do not touch the components or contacts on a board.
Hold a board by its edges or by its metal mounting bracket.

Magma
EB3600-AB | Chapter 1 Introduction
10
Chapter 1 Introduction
EB3600-AB
Magma ExpressBox 3600-AB is a 10-Slot PCI Express® Expansion System optimized for High Performance
Computing (HPC) applications. The expansion system consists of a PCIe® host interface card, High
Speed/HD IO cables, a 4U rack-mount chassis containing a 10 Slot backplane and an expansion interface
card, a replaceable power supply, and high CFM replaceable cooling fans.
Provides the optimum features for supporting world class High Performance Computing (HPC)
applications that demand ‘mission critical’ features such as N+1 redundant power and cooling
and high speed PCIe Gen 3 x16 connectivity.
It comes with a network management interface, including a web based GUI and SNMP interface
for monitoring the health of the chassis. The design is optimized for HPC applications built with
leading edge GPUs such as NVIDIA K-Series/GRID, AMD and coprocessors.
It is ideal for HPC applications such as oil and gas exploration, financial modeling, medical
imaging, genome sequencing, and simulation –any application requiring big data set analysis and
processing.
With the capacity for up to 9 state-of-the-art GPUs (e.g. NVIDIA K40), support for up to 12.9 Tflops (double
precision) or 38.6 Tflops (single precision) can be obtained within one ExpressBox 3600. For Big Data
applications, this level of performance is a game changer in the field of data mining and other
applications that could not be addressed just a short number of years ago.
Up to 38.6 Teraflops (single precision) or 12.9 Teraflops (double precision) of performance possible
(data representative of nine NVIDIA K40s).
Provides expansion for up to ten PCIe Gen 3 x16 peripherals including up to nine double width GPU
modules.
Passive host interface: no PCIe switch on host link cards, providing lowest latency possible.
Full 300W per slot power support (with included PCI AUX power).
Redundant N+1 air cooling for up to nine GPU modules.
19” Rack-mountable metal, 4U chassis 16.8” Wide x 7” High (4U) x 22.125” Deep.
Configurable and replaceable N+1 capable power supply solutions of 1200W, 2400W, 3600W or
4800W
Product Name
Description
EB3600-AB
Ten peripheral slots and two host link slots (All Gen 3 x16)