Cray XD1 User manual

Cray XD1™ RapidArray
Interconnect Topologies
Private
HR6–6425–F

© 2006 Cray Inc. All Rights Reserved. Unpublished Private Information. This unpublished work is protected to trade secret,
copyright and other laws. Except as permitted by contract or express written permission of Cray Inc., no part of this work or its
content may be used, reproduced or disclosed in any form.
U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE
The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR 252.227-7014.
All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted
Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14
or DFARS 48 CFR 252.227-7014, as applicable.
Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the
U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable.
Autotasking, Cray, Cray Channels, Cray Y-MP, GigaRing, LibSci, UNICOS and UNICOS/mk are federally registered
trademarks and Active Manager, CCI, CCMT, CF77, CF90, CFT, CFT2, CFT77, ConCurrent Maintenance Tools, COS, Cray Ada,
Cray Animation Theater, Cray APP, Cray Apprentice2, Cray C++ Compiling System, Cray C90, Cray C90D, Cray CF90, Cray EL,
Cray Fortran Compiler, Cray J90, Cray J90se, Cray J916, Cray J932, Cray MTA, Cray MTA-2, Cray MTX, Cray NQS, Cray Research,
Cray SeaStar, Cray S-MP, Cray SHMEM, Cray SSD-T90, Cray SuperCluster, Cray SV1, Cray SV1ex, Cray SX-5, Cray SX-6,
Cray T3D, Cray T3D MC, Cray T3D MCA, Cray T3D SC, Cray T3E, Cray T90, Cray T916, Cray T932, Cray UNICOS, Cray X1,
Cray X1E, Cray XD1, Cray X-MP, Cray XMS, Cray XT3, Cray XT4, Cray Y-MP EL, Cray-1, Cray-2, Cray-3, CrayDoc, CrayLink,
Cray-MP, CrayPacs, Cray/REELlibrarian, CraySoft, CrayTutor, CRInform, CRI/TurboKiva, CSIM, CVT, Delivering the power...,
Dgauss, Docview, EMDS, Gigaring, HEXAR, HSX, IOS, ISP/Superlink, Libsci, MPP Apprentice, ND Series Network Disk Array,
Network Queuing Environment, Network Queuing Tools, OLNET, RapidArray, RQS, SEGLDR, SMARTE, SSD, SUPERLINK,
System Maintenance and Remote Testing Environment, Trusted UNICOS, TurboKiva, UNICOS MAX, UNICOS/lc, and
UNICOS/mp are trademarks of Cray Inc.
Linux is a registered trademark of Linus Torvalds. All other trademarks are the property of their respective owners.

New Features
Cray XD1™ RapidArray Interconnect Topologies HR6–6425–F
This revision includes the WEEE compliance statement. This revision also incorporates information on the
following new topics:
•Cabling 36-chassis fat tree systems
•Cabling 48-chassis fat tree systems
•Configuring nonstandard fat tree topologies by depopulating standard base topologies

Record of Revision
Version Description
F May 2006
Updates to support the general availability of release 1.4 of the Cray XD1 product.
Includes the Waste Electrical and Electronic Equipment (WEEE) compliance
statement.
E October 2005
Supports the generally available 1.3 release of the Cray XD1 product. Minor changes
only.
D July 2005
Supports the limitedly available 1.3 release of the Cray XD1 product. Minor changes
only.
C May 2005
Supports the generally available release 1.2 of the Cray XD1 product. Minor changes
only.
A, B April 2005
Initial versions. Support the limitedly available release 1.2 of the Cray XD1 product.
HR6–6425–F Cray Private i

Contents
Page
Preface xi
Accessing Product Documentation . . . . . . . . . . . . . . . . . . . xi
Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Reader Comments . . . . . . . . . . . . . . . . . . . . . . . . xiii
Cray User Group . . . . . . . . . . . . . . . . . . . . . . . . xiii
Cray XD1 Support . . . . . . . . . . . . . . . . . . . . . . . . xiv
Introduction [1] 1
Who Should Read This Manual . . . . . . . . . . . . . . . . . . . . 1
Scope of This Manual . . . . . . . . . . . . . . . . . . . . . . . 1
How This Manual is Organized . . . . . . . . . . . . . . . . . . . . 1
Related Publications . . . . . . . . . . . . . . . . . . . . . . . 2
WEEE Compliance Statement . . . . . . . . . . . . . . . . . . . . . 3
Interconnecting the RapidArray Network [2] 5
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Handling RapidArray Cables . . . . . . . . . . . . . . . . . . . . . 5
ESD Precautions . . . . . . . . . . . . . . . . . . . . . . . . 5
Bend Radii . . . . . . . . . . . . . . . . . . . . . . . . . 6
Connecting RapidArray Cables . . . . . . . . . . . . . . . . . . . 6
Disconnecting RapidArray Cables . . . . . . . . . . . . . . . . . . 7
Procedure 1: To disconnect a RapidArray cable . . . . . . . . . . . . . 7
Cabling the Topology . . . . . . . . . . . . . . . . . . . . . . . 8
Preparing to Cable Your System . . . . . . . . . . . . . . . . . . . 8
Routing and Securing the Cables for Your System . . . . . . . . . . . . . . 9
Procedure 2: To route and secure RapidArray cables . . . . . . . . . . . . 9
HR6–6425–F Cray Private iii

Cray XD1™RapidArray Interconnect Topologies
Page
Cabling Direct-connect Topologies [3] 11
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Direct-connect Topology . . . . . . . . . . . . . . . . . . . . . 11
Equipment Required . . . . . . . . . . . . . . . . . . . . . . 12
Time Required . . . . . . . . . . . . . . . . . . . . . . . . 12
Installation Order . . . . . . . . . . . . . . . . . . . . . . . 12
Direct-connect Topology: Two Chassis . . . . . . . . . . . . . . . . . . 12
Direct-connect Topology: Three Chassis . . . . . . . . . . . . . . . . . . 14
Direct-connect Topology: Four Chassis . . . . . . . . . . . . . . . . . . 17
Direct-connect Topology: Five Chassis . . . . . . . . . . . . . . . . . . 20
Direct-connect Topology: Six Chassis . . . . . . . . . . . . . . . . . . 25
Direct-connect Topology: Seven Chassis . . . . . . . . . . . . . . . . . 30
Direct-connect Topology: Eight to Thirteen Chassis . . . . . . . . . . . . . . 35
Eight- to Thirteen-Chassis Connections . . . . . . . . . . . . . . . . . 35
Set A Connections . . . . . . . . . . . . . . . . . . . . . . 35
Set B Connections . . . . . . . . . . . . . . . . . . . . . . . 43
Cabling Fat Tree Topologies: Introduction [4] 51
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Base Configurations and Supported Configurations . . . . . . . . . . . . . . 52
External Switch Chassis . . . . . . . . . . . . . . . . . . . . . . 52
Number of Switch Chassis . . . . . . . . . . . . . . . . . . . . . 53
Switch Chassis Port Numbering . . . . . . . . . . . . . . . . . . . 53
Equipment Required . . . . . . . . . . . . . . . . . . . . . . . 54
Cabling a 12-Chassis Fat Tree [5] 55
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Port Groups (Fabrics) . . . . . . . . . . . . . . . . . . . . . . . 57
Installation Order . . . . . . . . . . . . . . . . . . . . . . . . 57
Port Mapping Details . . . . . . . . . . . . . . . . . . . . . . . 57
Fabric A . . . . . . . . . . . . . . . . . . . . . . . . . . 57
iv Cray Private HR6–6425–F

Contents
Page
Fabric B . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Fabric C . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Fabric D . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Cabling a 24-Chassis Fat Tree [6] 65
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Port Groups (Fabrics) . . . . . . . . . . . . . . . . . . . . . . . 67
Installation Order . . . . . . . . . . . . . . . . . . . . . . . . 67
Port Mapping Details . . . . . . . . . . . . . . . . . . . . . . . 67
Fabric A . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Fabric B . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Fabric C1 . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Fabric C2 . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Fabric D1 . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Fabric D2 . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Cabling a 36-Chassis Fat Tree [7] 79
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Cable Inventory . . . . . . . . . . . . . . . . . . . . . . . . . 81
Port Groups (Fabrics) . . . . . . . . . . . . . . . . . . . . . . . 82
Installation Order . . . . . . . . . . . . . . . . . . . . . . . . 83
Port Mapping Details . . . . . . . . . . . . . . . . . . . . . . . 83
Fabric A . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Fabric B . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Fabric C . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Fabric D . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Fabric E . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Fabric F . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Cabling a 48-Chassis Fat Tree [8] 103
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Cable Inventory . . . . . . . . . . . . . . . . . . . . . . . . . 105
HR6–6425–F Cray Private v

Cray XD1™RapidArray Interconnect Topologies
Page
Port Groups (Fabrics) . . . . . . . . . . . . . . . . . . . . . . . 105
Installation Order . . . . . . . . . . . . . . . . . . . . . . . . 107
Port Mapping Details . . . . . . . . . . . . . . . . . . . . . . . 107
Fabric A . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Fabric B . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Fabric C . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Fabric D . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Fabric E . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Fabric F . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Fabric G . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Fabric H . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Configuring Nonstandard Fat Trees [9] 133
Choosing the Base Configuration . . . . . . . . . . . . . . . . . . . . 133
Example 1: Base configuration for a 28-chassis system . . . . . . . . . . . . 133
Depopulation Rules . . . . . . . . . . . . . . . . . . . . . . . . 133
Depopulating Cray XD1 Chassis . . . . . . . . . . . . . . . . . . . 133
Example 2: 28-chassis system: removing chassis from 48-chassis configuration . . . . 134
Depopulating Switch Chassis: General . . . . . . . . . . . . . . . . . 134
Example 3: 8 switch chassis selected from 24-chassis configuration . . . . . . . 134
Depopulating Cables . . . . . . . . . . . . . . . . . . . . . . 134
Depopulating 24-port Switch Chassis . . . . . . . . . . . . . . . . . 134
Minimum Number of 144-port Switch Chassis . . . . . . . . . . . . . . . 134
Depopulating Leaf Blades in 144-port Switch Chassis . . . . . . . . . . . . . 134
Minimum Number of Cray XD1 Chassis . . . . . . . . . . . . . . . . . 135
Systems with 24-port Switches . . . . . . . . . . . . . . . . . . . 135
Example 4: Minimum number of chassis with 24-port switches . . . . . . . 135
Systems with 144-port Switches . . . . . . . . . . . . . . . . . . 136
Example 5: Minimum number of chassis with 144-port switches . . . . . . . 136
Cabling Remaining Components . . . . . . . . . . . . . . . . . . . 136
vi Cray Private HR6–6425–F