Graphcore IPU-POD64 reference design Manual

IPU-POD64 reference design | build and test guide
IPU-POD64 REFERENCE DESIGN
Build and test guide

IPU-POD64 reference design | build and test guide
2
Table of contents
Overview.............................................................................................5
1.1 Acronyms and abbreviations ..................................................................... 5
IPU-POD64 reference design components............................................6
2.1 IPU-M2000 ................................................................................................ 6
Overview 6
QR code label 7
2.2 Server........................................................................................................ 7
2.3 Switches.................................................................................................... 7
100GbE RoCE/RDMA switch (ToR switch) 7
1GbE management switch 7
2.4 Power distribution units ............................................................................ 7
2.5 Rack .......................................................................................................... 8
2.6 Supplementary mounting components...................................................... 8
2.7 Cables ....................................................................................................... 8
RJ45 cables 8
OSFP cables 8
QSFP cables 8
Rack assembly.....................................................................................9
3.1 Equipment checklist................................................................................. 11
3.2 Preparing the rack ................................................................................... 12
Rail distance 12
Unpackaging the rack 12
Removing the side panels and doors 13
Removing the vertical accessory channels 14
Adjusting the rear accessory channels 14
Adjusting the rear vertical rails 15
Adjusting the front vertical rails 15
Installing the rack rails 16
Installing PDU brackets 18
3.3 Installing the equipment.......................................................................... 20
Installing the IPU-M2000s 20
Installing the management switch 23
Installing the ToR switch 23
Installing the PDUs 24
Installing the Dell R6525 server(s) 24
3.4 Wiring the rack........................................................................................ 26
IPU-M2000 to IPU-M2000 IPU-Link connectivity (OSFP) 27
IPU-M2000 to IPU-M2000 Sync-Link cabling 29
IPU-M2000 to management switch cabling (RJ45) 31
Management switch –BMC wiring 32
3.4.4.1. Management switch –BMC + GW SoC wiring 34
IPU-M2000 to ToR switch cabling (QSFP) 36

IPU-POD64 reference design | build and test guide
3
Dell R6525 server(s) wiring 39
ToR switch to Dell server(s) 40
Management switch to Dell server(s) - iDRAC 41
Management switch to Dell server(s) –network connector 41
3.4.9.1. Management switch to Dell server(s) –switch management 42
Management switch to PDUs 42
3.5 Power cabling.......................................................................................... 43
IPU-M2000 power cabling 44
Server power cabling –Dell R6525 45
Switch power cabling 45
3.6 Completing the rack................................................................................. 46
Blanking panels 46
Front and rear doors 46
Side panels 46
IPU-POD64 server and switch configuration .......................................47
4.1 Server configuration ................................................................................ 47
Hardware recommendations 47
Storage configuration recommendations 47
Operating system recommendations 48
User accounts and groups 48
Ubuntu OS Installation and packages 49
CentOS OS installation and packages 50
Python packages 50
4.2 Network configuration............................................................................. 51
Overview 51
IPU-POD64 network interfaces 52
Management switch configuration 53
ToR switch configuration 53
IPU-POD64 VLAN assignments 54
Server network configuration 56
Services: DHCP (Dynamic Host Configuration Protocol) 57
Services: NTP (Network Time Protocol) 59
Services: syslog 60
IPU-POD64 software installation and configuration ............................61
5.1 Management server ................................................................................ 61
5.2 V-IPU software installation and configurations ........................................ 61
5.3 IPU-M2000 software installation and configuration ................................. 62
Download IPU-M2000 software update bundle 63
Software update of all IPU-M2000s 63
IPU-M2000 GW’s root file system config files 64
5.4 rack_tool ........................................................................................... 65
IPU-POD64 manual installation tests...................................................66
6.1 Running system BISTs .............................................................................. 66
6.2 Troubleshooting ...................................................................................... 66
BMC BISTs 66

IPU-POD64 reference design | build and test guide
4
V-IPU built in self tests 66
Automatic IPU-POD64 configuration...................................................71
7.1 Devices and preparation.......................................................................... 71
7.2 Scanning and test .................................................................................... 71
Description of each QR code step 75
Scan troubleshooting 76
Installation troubleshooting 76
Example output for Sync-Link or Traffic test failure 77
Other useful commands 77
Document revisions...........................................................................78
8.1 Revision history....................................................................................... 78
Legal notices .....................................................................................79
Warranties & licences 79

IPU-POD64 reference design | build and test guide
5
Overview
The IPU-POD64 reference design is a rack solution containing 16 IPU-M2000s, 1 to 4 host
servers (default 1 host server in reference configuration), network switches and IPU-POD
software. There are 64 Mk2 GC200 IPUs in total with four IPUs in each IPU-M2000.
For more information on IPU-POD systems available from Graphcore see
https://www.graphcore.ai/products.
This guide is for properly trained service personnel and technicians who are required to install
the IPU-POD64.
If you have any questions then please contact your Graphcore representative or use the
resources on the Graphcore support portal: https://www.graphcore.ai/support.
1.1 Acronyms and abbreviations
This is a short list that describes some of the most commonly used terms in this document.
BMC
Baseboard Management Controller –standby power domain
service processor doing system hardware management
BOM
Bill of Materials
GW
Short for IPU-Gateway, a co-processor to the four IPUs in the
IPU-M2000. It enables scaling with multiple IPU-M2000 units
IPU-Link
High speed communication links that interconnects IPUs within
and between IPU-M2000 units. Special cables are required for
IPU-Links between IPU-M2000 units
GW-Link
High speed communication link(s) that interconnect IPUs and
IPU-GWs horizontally between IPU-M2000 units. Special cables
are required for GW-Links between IPU-M2000 units
PDU
Power Distribution Unit
RDMA
Remote DMA
RNIC
RDMA Network Interface Controller
RoCE
RDMA over converged Ethernet
ToR
Top of Rack. Often used in combination with the ToR RDMA
switch that is placed on top of the IPU-M2000 stacked units.
Warning: Only qualified personnel should install, service, or replace IPU-POD64 equipment.

IPU-POD64 reference design | build and test guide
6
IPU-POD64 reference design components
This section describes the components in the IPU-POD64. Each IPU-POD64 contains:
•16 IPU-M2000s
•1 server (default configuration is 1 host server, up to 4 can be supported)
•1 1GbE management switch
•1 100GbE ToR switch
•2 power distribution units
•1 rack
•Supplementary mounting components
•Cables
2.1 IPU-M2000
Overview
There are 16 IPU-M2000s in each IPU-POD64 making a total of 64 IPUs: 4 IPUs per IPU-M2000.
The IPU-M2000 front panel contains:
•2 RNIC ports
•8 IPU-Link ports
•2 management GbE ports (BMC/GW SoC management ports)
•2 GW-Link ports
•8 IPU-Link ports
Front panel
The IPU-M2000 back panel contains:
•2 power connectors per IPU-M2000
•Fan units
•Unit QR code
Back panel
QR
code

IPU-POD64 reference design | build and test guide
7
QR code label
There is a QR code label on the back panel of each IPU-M2000. The QR code contains the
following information for each IPU-M2000:
•Company name (Graphcore)
•Serial number
•Part number
•BMC Ethernet MAC address
•GW Ethernet MAC address
•Graphcore support web URL (https://www.graphcore.ai/support)
2.2 Server
The default configuration of the IPU-POD64 uses a single PowerEdge R6525 server but up to
four servers can be connected. Contact Graphcore sales for details of other supported server
types. This document describes the default server (PowerEdge R6525) installation only –other
servers may have different installation requirements.
The default server configuration is described in section 4.1.
2.3 Switches
Each IPU-POD64 contains two switches serving different purposes.
100GbE RoCE/RDMA switch (ToR switch)
The 100GbE RoCE/RDMA switch (also referred to as the ToR switch) is used by the end user’s
machine learning (ML) jobs as a data-plane, connecting the host servers running thePoplar®
SDK with the IPUs running the ML model in the IPU-M2000s. The default ToR switch is an Arista
DCS-7060CX-32S-F. Contact Graphcore sales for details of other supported switch types. This
document describes the default switch (7060CX) installation only –other switches may have
different installation requirements.
1GbE management switch
The 1GbE management switch is used for connecting the management ports together inside
the rack. The default management switch is an Arista DCS-7010T-48-F. Contact Graphcore
sales for details of other supported switch types. This document describes the default switch
(7010T) installation only –other switches may have different installation requirements.
2.4 Power distribution units
Two power distribution units (PDUs) are installed in each IPU-POD64.The default unit is an
APC AP8886.

IPU-POD64 reference design | build and test guide
8
2.5 Rack
The IPU-M2000s, servers, switches, and PDUs are installed in an APC AR3300SP rack. This rack
has a packing system designed to safely transport and unload the rack.
It is important to follow the instructions carefully when packing or unpacking the rack.
2.6 Supplementary mounting components
The supplementary components listed below also need to be installed.
•Cable organizer
•Blanking panel
2.7 Cables
Each IPU-POD64 has three types of cabling:
•RJ45 cables
•OSFP cables
•QSFP cables
RJ45 cables
•Red: IPU-M2000 to IPU-M2000 within-rack IPU-Link connectivity
•Blue: Connecting IPU-M2000s to the management switch (BMC +GWmanagement)
•Blue: Connecting servers to the management switch
•Yellow for connecting IPU-M2000s to the management switch (BMC only
management)
OSFP cables
•IPU-M2000 to IPU-M2000 (IPU-Link) connectivity
QSFP cables
•IPU-M2000 to ToR switch connectivity
•Server to ToR switch connectivity
All cable connections are described in Section 0.

IPU-POD64 reference design | build and test guide
9
Rack assembly
Please note the correct orientation of the IPU-M2000, server and switch units in the rack to
ensure correct airflow.
The front interface of the IPU-M2000 units (connectivity ports) should be matched with the
front door of the rack (cold aisle). The rear interface of the server and switches (power and
fans) should be matched with the rear door of the rack (hot aisle).
Completed rack - cold aisle (four-server version)
Note that this photo shows a four-server version of the IPU-POD64.The default reference
design has one server which would be the server in the lowest position, closest to the
switches.

IPU-POD64 reference design | build and test guide
10
Completed rack –hot aisle (four server version)
Note that this photo shows 3x blue RJ45 cables in each R6525 server. In the default build,
servers 2 - 4 only have 2x blue RJ45 cables. See later sections for more information about
server cabling.
Note also that this photo shows a four-server version of the IPU-POD64.The default reference
design has one server which would be the server in the lowest position, closest to the
switches.