Cray ClusterStor H-6167 User manual

ClusterStor™ MMU/AMMU Addition
(3.3)
H-6167

Contents
About ClusterStor™ MMU and AMMU Addition......................................................................................................... 3
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System........................................................................ 5
MMU/AMMU Addition Process........................................................................................................................ 8
Prerequisites for MMU/AMMU Addition...........................................................................................................8
Cable MMU/AMMU Hardware to the Storage System.................................................................................. 11
Unmount Lustre and Auto-Discovery.............................................................................................................13
Verify MDS Node Discovery.......................................................................................................................... 17
Complete the MMU/AMMU Addition Procedure............................................................................................ 21
Sample Output for configure_mds........................................................................................................... 27
Tips and Tricks.........................................................................................................................................................32
Failed Recovery or MMU/AMMU Added Out of Sequence........................................................................... 32
Node Stuck in Discovery Prompt...................................................................................................................33
Reconnect a Screen Session........................................................................................................................ 34
Check Puppet Certificate...............................................................................................................................35
SSH Connection Refused / Puppet Certificates............................................................................................ 35
SSH Connection Refused-- MDS/OSS Node Not Fully Booted.................................................................... 35
No Free Arrays / Wrong md Assignment.......................................................................................................36
Check ARP, Local Hosts, and DHCP.............................................................................................................37
Post Upgrade Check..................................................................................................................................... 37
Test Mount Lustre File System on Node 001.................................................................................................37
Contents
H-6167 2

About ClusterStor™ MMU and AMMU Addition
ClusterStor™ MMU and AMMU Addition (3.3) H-6167 includes instructions for adding additional Metadata
Management Units (MMUs) or Advanced Metadata Management Units (AMMUs) to a storage system in the field.
MMUs and AMMUs allow the system to take advantage of the Lustre Distributed Namespace (DNE) feature
introduced with Lustre 2.5. An AMMU is intended for systems that have a requirement for a high single-directory
file create and metadata operation rate.
MMUs are supported for ClusterStor L300 and L300N systems running releases 2.1.0, 3.0.0, 3.1, 3.2, and 3.3.
AMMUs are only supported for L300 and L300N systems running releases 3.0.0, 3.1, 3.2, and 3.3. Only one (1)
AMMU, located in the base rack, is supported in release 3.0.0. Beginning with release 3.1, systems support up to
two (2) AMMUs (one [1] in the base rack).
Record of Revision: Publication H-6167
Table 1. Record of Revision
Publication Title Date Updates
ClusterStor MMU/AMMU Addition (3.3) H-6167 December 2019 Release of 3.3 version
ClusterStor MMU/AMMU Addition (3.2) H-6167 August 2019 Release of 3.2 version
ClusterStor MMU/AMMU Addition (3.1) H-6167 February 2019 Release of 3.1 version
Scope and Audience
The procedure in this publication is intended to be performed only by qualified Cray personnel.
Typographic Conventions
Monospace A Monospace font indicates program code, reserved words or library functions,
screen output, file names, path names, and other software constructs
Monospaced Bold A bold monospace font indicates commands that must be entered on a command
line.
Oblique or Italics An oblique or italics font indicates user-supplied values for options in the
syntax definitions
Proportional Bold A proportional bold font indicates a user interface control, window name, or
graphical user interface button or control.
\ (backslash) At the end of a command line, indicates the Linux® shell line continuation character
(lines joined by a backslash are parsed as a single line).
About ClusterStor™ MMU and AMMU Addition
H-6167 3

Other Conventions
Sample commands and command output used throughout this publication are shown with a generic filesystem
name of cls12345.
Trademarks
© 2019, Cray Inc. All rights reserved. All trademarks used in this document are the property of their respective
owners.
About ClusterStor™ MMU and AMMU Addition
H-6167 4

Add an MMU or AMMU to a ClusterStor™ L300 and L300N
System
The topics to follow describe the process to add additional Metadata Management Units (MMUs) or Advanced
Metadata Management Units (AMMUs) to a storage system in the field. MMUs and AMMUs allow the system to
take advantage of the Lustre Distributed Namespace (DNE) feature introduced with Lustre 2.5.
MMUs are supported for ClusterStor L300 and L300N systems running releases 2.1.0, 3.0.0, 3.1, 3.2, and 3.3.
AMMUs are only supported for L300 and L300N systems running releases 3.0.0, 3.1, 3.2, and 3.3. Only one (1)
AMMU, located in the base rack, is supported in release 3.0.0. Beginning with release 3.1, systems support up to
two (2) AMMUs (one [1] in the base rack).
CAUTION: This procedure is intended to be performed only by qualified Cray personnel.
Lustre 2.5 and later releases support Phase 1 of the Lustre DNE feature, which allows multiple MDTs, operating
through multiple MDS nodes, to be configured and to operate as part of a single file system. This feature allows
the number of metadata operations per second within a cluster to scale beyond the capabilities of a standard
storage system's single MDS. To achieve this capability requires that the file system namespace be configured
manually so that file system operations are evenly distributed across the MDS/MDT resources.
The function of the standard MDS node in the MMU or AMMU does not change when additional MMUs and
AMMUs are added. Additional MDS servers are added to the system via the procedures in this publication.
A system running one of the supported software releases supports up to eight (8) total MMUs. One MMU must be
installed in the base rack. Up to seven (7) additional MMUs can be installed into the storage racks, with no more
than one (1) MMU per storage rack. Each MMU is connected to the Local Management Network (LMN) and the
Local Data Network (LDN), as illustrated in the following figure.
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 5

Figure 1. MMU Network Overview
For systems using an AMMU instead of an MMU in the base rack, a maximum of one additional AMMU may be
added to a storage rack.
MMU and AMMU Hardware
An MMU is a 2U storage enclosure that consists of the following:
● Two (2) Embedded Application Controllers (EACs)
● 22, 10K RPM, HDDs (ten [10] for each MDT and two [2] global hot spares)
● Two (2) shared Power and Cooling Modules (PCMs)
An AMMU is a 4U enclosure assembly of servers and storage that consists of the following:
● Two (2) discrete 1U servers, each server with dual power supplies and a local disk for HA state
● One (1) 2U24 EBOD, consisting of
○ 22, 10K RPM, HDDs (ten for each MDT and two global hot spares)
○ Two (2) PCMs
MMUs and AMMUs support 900GB or 1.8TB HDDs
Lustre MDS Functionality
Each MMU/AMMU presents as two (2) Lustre Metadata Server (MDS) nodes:
● One (1) MDS running on each of the two EACs, or two discrete 1U servers
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 6

● The two (2) MDSes operate as an HA pair
● Each MDS, under normal operation, owns and operates one Lustre Metadata Target (MDT) of two MDTs
provided by the MMU/AMMU
● During failover, the HA partner of the failed MDS operates both MDTs
● Each MDT is configured as a single RAID 10 (5+5) array, using a total of 20 HDDs for both MDTs
Abbreviations Used in this Publication
Abbreviation Meaning
ADU Deprecated term meaning Additional DNE Unit. Has been replaced by
MMU and AMMU.
AMMU Advanced Metadata Management Unit
BMC Baseboard Management Controller
CLI Command Line Interface
CSCLI ClusterStor Command Line Interface
DHCP Dynamic Host Configuration Protocol
DNE Distributed Namespace
EBOD Expanded Bunch of Disks
FGR Fine-grained Routing
GUI Graphical User Interface
HA High Availability
HPC High Performance Computing
IBS0 / IBS1,
OPA0 / OPA 1,
40G0 / 40G1 or 100G0 / 100G1
Notation for the LDN switches (InfiniBand, Omni-Path, or high speed
Ethernet)
IPMI Intelligent Platform Management Interface
LCN Lustre Client Network
LDN Local Data Network
LMN Local Management Network
MDS Metadata Server
MDT Metadata Target
MGMT Management node (that is, MGMT0 or MGMT1)
MGS Management Server
MMU Metadata Management Unit
OSS Object Storage Server
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 7

Abbreviation Meaning
PCM Power Cooling Module
PDU Power Distribution Unit
SSU Scalable Storage Unit
SW0, SW1 Notation for the LMN switches
MMU/AMMU Addition Process
The process of adding additional MMUs to a system, and then configuring the file system cluster to use those
MMUs is called MMU Addition. It is called AMMU Addition when adding one AMMU to a system that already has
an AMMU in the base rack.
The process is designed to be run for one additional MMU or AMMU at a time and consists of the following
general steps:
● Install the new MMU/AMMU hardware in the proper position in a storage rack. There is a maximum of one (1)
MMU/AMMU per storage rack. ClusterStor 3.1, 3.2, and 3.3 support only one (1) additional AMMU in a
system.
● Connect the new hardware's power cables, and cable the MMU's EACs or the AMMU 1U servers to the LMN
and the LDN.
● Power on the MMU/AMMU nodes, and run the discovery procedure, which provisions MMU/AMMU network
interfaces and storage volumes.
● Register MDS/MDT components with the Lustre Management Server (MGS), and configure them as part of
the file system.
To add more than one (>1) MMU at a time, it is recommended that the entire addition procedure be performed
and completed sequentially for each new MMU. Do not proceed to adding subsequent MMUs until the previous
MMU addition operation is complete.
MMUs and AMMUs are added to storage racks only. Only one (1) MMU/AMMU may be added per storage rack,
up to a maximum of eight (8) total MMUs or two (2) total AMMUs for a system.
Contact Cray support to add an MMU or AMMU to a storage rack.
Prerequisites for MMU/AMMU Addition
The following prerequisites must be met before performing this procedure.
Table 2. Required Firmware Levels
Firmware Version
MMU OSS firmware Minimum version 3022
I/O controller firmware in
MMU storage
Minimum version 3022
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 8

Firmware Version
To check the minimum version on L300 and L300N systems, run this command
from the MGS or MDS node:
[admin@n002]$ sudo sg_map –x –i | grep SP-
To check the minimum version on ClusterStor 9000 systems that have been
upgraded to 3.1, run this command from the MGS or MDS node:
[admin@n002]$ sudo sg_map –x –i | grep XYRATEX
Updated GEM GOBI
firmware
1.33 is a minimal version for L300 and L300N systems; the current version is 1.36.
Not applicable for CS9000 systems that have been upgraded to release 3.1.
Updated USM (GEM)
firmware
● 3.26c for ClusterStor 9000 systems that have been upgraded to release 3.1.
● For L300 and L300N systems:
○ 4.1.16a for system enclosures hosted by E6EBD controllers (5U84 and
2U24 6Gbit/s)
○ 5.1.6c for system enclosures hosted by E12EBD controllers (5U84 and
2U24 12Gbit/s)
NOTE: The USM firmware update for L300 and L300N may be run
using the ebod_update.sh script (packaged with GOBI 1.33).
Alternatively, the USM update may be specified to to run when
updating to GEM GOBI 1.33.
● Newly installed systems must have had the Customer Setup Wizard run. Follow the procedures in the latest
revision of the Field Installation Guide to complete the initial installation setup, if necessary.
●Site Preparation Guide (Site Survey): the Site Preparation Guide is required to complete the Customer
Wizard installation prior to performing MMU/AMMU Addition.
● The storage system must be in Daily Mode before the MMU/AMMU Addition procedure is performed.
● This procedure requires the use of a single Lustre client system to perform end-to-end verification of some
procedures. Lustre client systems are not provided as part of the storage system. Usually, a compute node in
an HPC cluster has Lustre client software installed and can be used for this purpose. However, the site may
require using a reservation system to use a compute node.
●Inspection Report: the Inspection Report (also referred to as Concessions Document or Customer Ship
Summary) provides details about the complete system, including rack and component serial numbers,
software installed, and in what order the racks were installed. This document can be obtained from the
Account Manager, or in the case of an end user deployment, the document will be delivered with the system.
●System access requirements: may require the technician to have root (super user) access. Root access is
required to perform this procedure on a ClusterStor system.
●Service interruption level: Interrupt (requires taking the Lustre file system offline)
●Rack position in a storage rack: It is recommended that the MMU be installed at the same approximate
rack position that the base-rack MMU occupies in a typical system: directly below the top-of-rack
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 9

management and network switches and before any SSUs. In a standard 42U storage rack, the MMU
enclosure should occupy rack positions 37 and 38. An AMMU enclosure should occupy rack positions 35
through 38. See the following diagram.
Figure 2. Storage Racks with MMU and AMMU Enclosures Added
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 10

Required Tools and Equipment
● SSU OSS Reset Tool (P/N 101150300), which is used to press the power on/off button on the MMU MDS
nodes without damaging the EAC
● For additional information about updating GEM GOBI firmware on systems after release 2.1.0 SU-004, please
contact Cray Support.
● For additional information about updating USM firmware on systems running releases prior to 2.1.0 SU-005,
please contact Cray Support. The publication Sonexion® USM Firmware Update Guide H-6137, available on
http://pubs.cray.com, may also be helpful for ClusterStor 9000 and Sonexion 2000 systems that have been
upgraded to release 3.1 and higher.
Estimated Service Interval
The following table shows time estimates for each MMU/AMMU being added to the system. Schedule an
appropriate service interval with the customer.
Table 3. Time Needed to Install One MMU/AMMU
Task Required Time
Install and cable hardware 20-30 minutes
Perform discovery 5-10 minutes
Assign and apply 15-20 minutes
Assorted tasks (keyboard time, powering up, etc.) 5-10 minutes
For example, adding one (1) MMU takes approximately 45 to 70 minutes.
The timings above do not take into consideration additional time that is needed when adding an MMU/AMMU to a
running system. In such a case, before the new MDS nodes can be used on the system, Lustre must be stopped
and beSystemNetConfig should be re-run to ensure that proper NIDs and parameters are applied to the newly
added MDS nodes. The time required for this is approximately 15 to 20 minutes.
Cable MMU/AMMU Hardware to the Storage System
An MMU/AMMU requires the following power and cable connections in the storage rack:
● 2 (MMU) or 6 (AMMU) power connections: one (MMU) or 3 (AMMU) to each of the rack's two PDUs
● 4 connections to the rack's management switches: one cable from each of the MMU's EACs or the AMMU's
1U servers to each of the management switches (SW0 / SW1)
● 2 connections to the rack's network switches: one cable from each of the MMU's EACs or the AMMU's 1U
servers to one of the network switches (IBS0/IBS1 or OPA0/OPA1 or 40G0/40G1 or 100G0/100G1)
In addition, the two (2) 1U servers in the AMMU are cabled to the 2U24 EBOD with 12Gb/sec SAS connections.
The following figure illustrates how the MMU is cabled to the management and network switches. The port
assignments are examples only.
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 11

Figure 3. MMU Cabling
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
X1 X2 X3 X4 X5 X6 X7 X8
1 2 3 456 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
X1 X2 X3 X4 X5 X6 X7 X8
Eth SW1
IB SW1
IB SW0
Eth SW0
A
B
MMU EAC Module
MMU
1 3 5 7 9 11 13 15 17 19 21 23
14 16 18 20 22 24
25 27 29 31 33 35
26 28 30 32 34 36
2 4 8 10 126
1 3 5 7 9 11 13 15 17 19 21 23
14 16 18 20 22 24
25 27 29 31 33 35
26 28 30 32 34 362 4 8 10 126
Figure 4. AMMU Server Cabling
For recommended port assignments for both the MMU and AMMU, contact Cray Support to obtain internal
cabling documentation.
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 12

Figure 5. AMMU SAS Connections
Unmount Lustre and Auto-Discovery
Procedure
1. Log in to the primary MGMT node via SSH:
[Client]$ ssh –l admin primary_MGMT_node
2. In this and following steps, verify that client systems have unmounted the Lustre file system and that Lustre
has been stopped.
Use the MDS node to confirm that all Lustre clients are unmounted. From the primary MGMT node, run:
[MGMT0]$ ssh mds_nodename "lctl get_param '*.*.exports.*.uuid'"
● Example output showing one client mounted and Lustre still running:
[admin@cls12345n000 ~]$ ssh cls12345n003 "lctl get_param
'*.*.exports.*.uuid'"
mdt.testfs-MDT0000.exports.[email protected]=2412630a-
db8d-806a-09eb-0690c8e1e86b
It is recommended that every effort be made to identify and stop Lustre clients before performing the
MMU/AMMU Addition procedure. In the sample output above, 172.18.1.188@o2ib (shown
underscored) represents a client that still has the Lustre file system mounted and needs to be addressed.
See Step 3 on page 14 if it is necessary to unmount a client manually.
● Example output showing no clients mounted and Lustre still started:
[admin@cls12345n000 ~]$ ssh cls12345n003 "lctl get_param
'*.*.exports.*.uuid'"
[admin@cls12345n000 ~]$
● Example output showing no clients mounted and Lustre stopped:
[admin@cls12345n000 ~]$ ssh cls12345n003 "lctl get_param
'*.*.exports.*.uuid'"
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 13

error: get_param: /proc/
{fs,sys}
/
{lnet,lustre}
///exports/*/uuid: Found no match
3. If a client system needs to be unmounted manually, run the following command:
[Client]$ sudo umount lustre_mountpoint
Example:
[Client]$ sudo umount /mnt/lustre
4. If the system is NXD enabled, perform the following steps. Otherwise, go to step 5 on page 14
a. Stop NXD caching on the system.
[MGMT0]$ cscli nxd disable
This command initiates a process to flush the NXD cache to the main array. If the cache is completely full,
it may take more than 40 minutes to flush all cached data out to drives. Note that there is no output from
the cscli nxd disable command.
b. Monitor the NXD caching state by running the cscli nxd list command repeatedly and checking the
output. Note that while the cache is being flushed, the value displayed in the Caching State field (of the
cscli nxd list command output) for the OSS nodes and OSTs is disabling. The displayed values
change to disabled once the cache is completely flushed.
c. Wait until the value displayed in the Caching State field is disabled for all OSS nodes and OSTs before
proceeding to the next step.
[admin@cls12345n000 ~]$ cscli nxd list
----------------------------------------------------------------------------------------
Host Cache Caching Total Cache Cache Cache Bypass
Group State Cache Size Block Window IO
Size In Use Size Size Size
-----------------------------------------------------------------------------------
-----
cls12345n004 nxd_cache_0 disabled 1.406 TB 699.875 MB 8(4 KiB) 128(64 KiB) 64(32 KiB)
cls12345n005 nxd_cache_1 disabled 1.406 TB 745.500 MB 8(4 KiB) 128(64 KiB) 64(32 KiB)
----------------------------------------------------------------------------------------
5. To stop Lustre on the storage system, first verify that the Lustre file system is mounted:
[MGMT0]$ cscli fs_info
For example:
[admin@cls12345n000 ~]$ cscli fs_info
-------------------------------------------------------------------
OST Redundancy style: Declustered Parity (GridRAID)
Disk I/O Integrity guard (ANSI T10-PI) is not supported by hardware
-------------------------------------------------------------------
Information about "cls12345" file system:
-------------------------------------------------------------------
Node Role Targets Failover partner Devices
-------------------------------------------------------------------
cls12345n002 mgs 0 / 0 cls12345n003
cls12345n003 mds 1 / 1 cls12345n002 /dev/md66
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 14

cls12345n004 oss 2 / 2 cls12345n005 /dev/md0, /dev/md2
cls12345n005 oss 2 / 2 cls12345n004 /dev/md1, /dev/md3
In the output, under the heading "Targets," look for "0" in the first character position. For example, 0 / 4 or 0 /
0. If any value is not "0", you must stop the Lustre file system in the next step. If all values are "0" you may
skip to step 7 on page 15.
6. If the file system is mounted, stop the Lustre file system:
[admin@cls12345n000 ~]$ cscli unmount -f cls12345
unmount: cls12345 is stopped on cls12345n[402-403]!
unmount: cls12345 is stopped on cls12345n[404-405]!
unmount: MGS is stopping...
unmount: MGS is stopped!
unmount: File system cls12345 is unmounted.
7. Verify that the file system has been unmounted:
[admin@cls12345n000 ~]$ cscli fs_info
----------------------------------------------------------------------
OST Redundancy style: Declustered Parity (GridRAID)
Disk I/O Integrity guard (ANSI T10-PI) is not supported by hardware
----------------------------------------------------------------------
Information about "cls12345" file system:
----------------------------------------------------------------------
Node Role Targets Failover partner Devices
----------------------------------------------------------------------
cls12345n002 mgs 0 / 0 cls12345n003
cls12345n003 mds 0 / 1 cls12345n002 /dev/md66
cls12345n004 oss 0 / 2 cls12345n005 /dev/md0, /dev/md2
cls12345n005 oss 0 / 2 cls12345n004 /dev/md1, /dev/md3
8. Following addition of the MMU/AMMU, it is possible for the pool of IP addresses that are available for LDN
use to become exhausted. To prevent this issue, check the range of permitted IP addresses and, if it is too
small, increase it.
First, determine the number of nodes on the system after MMU/AMMU addition. Add to the number of pre-
existing nodes, 1 + the number of MMU EACs or AMMU 1U servers to be added through the MMU/AMMU
addition. Or as a formula:
[number of pre-existing nodes] + (1 + [number of MMU EACs or AMMU 1U servers added through the MMU/
AMMU addition])
Then, in the following steps, compare the result to the size of the pool of IP addresses that are available for
LDN use.
9. List the range of IP addresses that are allocated for LDN use and determine which of those IP addresses are
available for use:
[MGMT0]$ cscli lustre_network find_gaps
For example:
[admin@cls12345n000]$ cscli lustre_network find_gaps
find_gaps:
assigned IPs in InfiniBand fabric:
172.31.172.1 - 172.31.172.10
id 1:
range: 172.31.172.1 - 172.31.172.254 # size: 254, free: 244
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 15

free IPs:
172.31.172.11 - 172.31.172.254
In the above example, the IP address pool has 254 IP addresses in its range (starting with 172.31.172.1 up to
and including 172.31.172.254). Of those 254 IP addresses, 244 are currently available for LDN use (starting
with 172.31.172.11 up to and including 172.31.172.254).
The underscored text in the above example shows the range ID of the IP address pool that is available for
LDN use. If the pool needs to be extended, this ID number will be needed in the following step.
If the number of IP addresses needed for LDN use following the MMU/AMMU addition is greater than the
available supply, continue to the following step. If not, go to Step 12 on page 16.
10. Change the upper limit of the range to increase the supply of IP addresses that are available for LDN use:
[MGMT0]$ cscli lustre_network extend_range -i range_ID_to_extend -t
new_range_end
The new_range_end should be large enough to accommodate all the new MDS nodes.
The following example builds on the example from the preceding step, illustrating how to extend the range of
that IP address pool from 172.31.172.1 - 172.31.172.254 to 172.31.172.1 up to and including 172.31.173.254.
[admin@cls12345n000 ~]$ cscli lustre_network extend_range -i 1 -t 172.31.173.254
11. Verify the new range settings:
[admin@cls12345n000]$ cscli lustre_network find_gaps
The range maximum value should match the value from Step 8 on page 15.
12. Validate the new address range to ensure it is compatible with the netmask and does not overlap with other
networks:
[admin@cls12345n000]$ sudo su -
[root@cls12345n000]# /opt/xyratex/bin/beSystemNetConfig.sh -V $(nodeattr -VU
cluster)
If the script prints nothing, the new settings are valid. Exit root and proceed to Step 13 on page 16.
If the script prints an error, the new settings are invalid. Please contact Cray Support.
13. Power on the MMU/AMMU for discovery, in increasing node number order starting with the lowest numbered
node, as follows (see the following figure).
a. Power on the new MMU/AMMU. Place the power switches in the ON position or plug in enclosures that
do not have a power switch. In a 2U24 MMU, the power switches are on the PCM. For the AMMU nodes
(1U servers), power is applied when the servers are plugged in. For the AMMU 2U24 EBOD, the power
switches are on the PCM.
Wait at least 50 seconds before powering on the MMU MDS nodes from their EACs, as it can take that
long for the BMC to start.
b. On an MMU, use the SSU OSS Reset Tool (PN 101150300) to press the Power On/Off button (hold for 3
seconds, and then release) on the rear panel of both MDS node's EACs.
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 16

Figure 6. MDS Node Layout
The management software's auto discovery feature will discover the new MDS nodes and prepare them
(the order of the individual controllers within an MMU/AMMU will be handled by the management
software). After powering up the MMU/AMMU, check that the auto discovery feature has discovered the
new MDS nodes. For more information, refer to Verify MDS Node Discovery on page 17.
Verify MDS Node Discovery
About this task
Use the commands in this section to check for successful discovery of the new MDS nodes. As these commands
are run on the system, compare the output with the examples shown.
Procedure
1. Show the list of new nodes; that is, the nodes booted into discovery mode:
[MGMT0]$ cscli show_new_nodes
For example:
[admin@cls12345n000 ~]$ cscli show_new_nodes
---------------------------------------------------------------------------
Hostname/MAC IPMI Free arrays Assigned arrays Pass/Fail
---------------------------------------------------------------------------
/ 00:50:CC:79:0A:66 172.16.0.111 Passed
/ 00:50:CC:79:20:E0 172.16.0.112 Pending
New nodes listed with a Pass/Fail status of Passed have booted into discovery mode and are ready. New
nodes with a Pass/Fail status of Pending have not booted into discovery mode and are not ready.
Once all new MDS nodes have booted into discovery mode and are listed with a Pass/Fail status of Passed,
proceed.
2. Determine what host names are already in use in the storage system:
[MGMT0]$ cscli show_nodes
Example command output showing Lustre stopped on a typical GridRAID system:
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 17

[admin@cls12345n000 ~]$ cscli show_nodes
-------------------------------------------------------------------------------
Hostname Role Power Service Targets Partner HA Resources
state state
-------------------------------------------------------------------------------
cls12345n000 MGMT On N/a 0 / 0 cls12345n001 None
cls12345n001 (MGMT) On N/a 0 / 0 cls12345n000 None
cls12345n002 (MDS),(MGS) On Stopped 0 / 1 cls12345n003 Local
cls12345n003 (MDS),(MGS) On Stopped 0 / 1 cls12345n002 Local
cls12345n004 (OSS) On Stopped 0 / 1 cls12345n005 Local
cls12345n005 (OSS) On Stopped 0 / 1 cls12345n004 Local
-------------------------------------------------------------------------------
3. Assign unique host names, that are not currently in use, to each new MDS node pair. Using the MAC
addresses that were displayed by the cscli show_new_nodes command, run:
[MGMT0]$ cscli configure_hosts –m 1st_node_MAC_addr --hostname
1st_node_hostname && cscli configure_hosts –m 2nd_node_MAC_addr
--hostname 2nd_node_hostname
In the following example, the new MDS nodes are given the host names cls12345n006 and cls12345n007:
[admin@cls12345n000 ~]$ cscli configure_hosts -m 00:50:CC:79:0A:66 --hostname
cls12345n006 && cscli configure_hosts -m 00:50:CC:79:20:E0 --hostname
cls12345n007
configure_hosts: discovered node hostname: discovery-0050cc790a66
configure_hosts: Syncing puppet information
configure_hosts: Waiting until node reboots ...
configure_hosts: done
configure_hosts: discovered node hostname: discovery-0050cc7920e0
configure_hosts: Syncing puppet information
configure_hosts: Waiting until node reboots ...
configure_hosts: done
The cscli configure_hosts command allows any host name to be set, as long as it is unique and not
already in use. The host names do not have to be contiguous numbers.
For each new MDS node pair, it is important that the host name assignment and subsequent reboot occur
within a short period of time of each other. Chaining the host name assignments into a single command helps
to ensure this.
4. Verify that the new MDS nodes have rebooted and they are up:
[MGMT0]$ pdsh –a uptime
For example:
[admin@cls12345n000 ~]$ pdsh -a uptime
cls12345n006: ssh: connect to host cls12345n006 port 22: Connection refused
pdsh@cls12345n000: cls12345n006: ssh exited with exit code 255
cls12345n000: 11:35:03 up 5:19, 3 users, load average: 0.52, 0.59, 0.53
cls12345n005: 11:35:03 up 3:25, 0 users, load average: 1.15, 1.05, 1.28
cls12345n001: 11:35:03 up 5:11, 0 users, load average: 0.06, 0.08, 0.02
cls12345n002: 11:35:03 up 5:05, 0 users, load average: 0.03, 0.11, 0.12
cls12345n004: 11:35:03 up 3:12, 0 users, load average: 0.34, 0.63, 0.63
cls12345n003: 11:35:03 up 5:05, 0 users, load average: 0.11, 0.12, 0.12
cls12345n007: 11:35:03 up 29 min, 0 users, load average: 1.10, 1.05, 1.93
In this sample output, the new MDS node, cls12345n007, is up and has booted into the regular appliance
prompt. The other new MDS node, cls12345n006, is booting but is not completely up yet.
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 18

If, after the new MDS nodes reboot, there is an "SSH connection refused" message, follow the procedures in
SSH Connection Refused / Puppet Certificates on page 35 for Puppet cleanup.
5. Once the new MDS nodes are up, verify that the hostnames from Step 2 on page 17 have been assigned to
the new nodes:
[MGMT0]$ cscli show_new_nodes
For example:
[admin@cls12345n000 ~]$ cscli show_new_nodes
----------------------------------------------------------------------
Hostname/MAC IPMI Free arrays Assigned Pass/Fail
arrays
----------------------------------------------------------------------
cls12345n006 172.16.0.111 md0 Passed
cls12345n007 172.16.0.112 md1 Passed
----------------------------------------------------------------------
6. After all of the new MDS nodes have been successfully assigned hostnames, use the following commands to
verify that new node information is listed or to check HSN connectivity. Refer to the examples that follow the
list of commands.
●cat /etc/hosts
●pdsh -a date | sort
●sudo ibnodes or:
pdsh -a ibstat mlx4_0 1 | grep Physical | sort
NOTE: The sudo ibnodes and pdsh commands in this bullet will not work on a system that
uses Intel Omni-Path switches.
●sudo ibhosts
NOTE: The sudo ibhosts command in this bullet will not work on a system that uses Intel
Omni-Path switches.
●pdsh -w new_nodes ethtool eth20 | grep Speed or:
pdsh -w new_nodes ibstat | grep Link
Notes:
● Run all commands from the primary MGMT node. For example:
[MGMT0]$ cat /etc/hosts
● The IP addresses for BMC, IB, and eth0 will increment in numeric order.
● If the proper entries in all references below are not listed, or any entry is out of sequence, please stop and
contact Cray Support. Do not add any additional MMUs or AMMUs.
If the proper entries are listed and they are in the correct sequence, please continue the MMU/AMMU
Addition procedure. See Section Complete the MMU/AMMU Addition Procedure on page 21.
Example output for command: cat /etc/hosts
[admin@cls12345n000 ~]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 19

localhost6.localdomain6
172.16.2.1 puppet
172.16.2.2 nfsserv
172.16.2.3 cls12345n000 cls12345n000-eth0
172.16.0.101 cls12345n000-ipmi
172.18.1.1 cls12345n000 cls12345n000-ib0
169.254.0.1 cls12345n000-ha cls12345n000-eth10
172.16.2.4 cls12345n001 cls12345n001-eth0
172.16.0.102 cls12345n001-ipmi
172.18.1.2 cls12345n001 cls12345n001-ib0
169.254.0.2 cls12345n001-ha cls12345n001-eth10
172.16.2.5 cls12345n002 cls12345n002-eth0
172.16.0.104 cls12345n002-ipmi
172.18.1.3 cls12345n002 cls12345n002-ib0
172.16.2.6 cls12345n003 cls12345n003-eth0
172.16.0.103 cls12345n003-ipmi
172.18.1.4 cls12345n003 cls12345n003-ib0
172.16.2.7 cls12345n004 cls12345n004-eth0
172.16.0.109 cls12345n004-ipmi
172.18.1.5 cls12345n004 cls12345n004-ib0
172.16.2.8 cls12345n005 cls12345n005-eth0
172.16.0.110 cls12345n005-ipmi
172.18.1.6 cls12345n005 cls12345n005-ib0
172.16.2.9 cls12345n006 cls12345n006-eth0
172.16.0.111 cls12345n006-ipmi
172.18.1.7 cls12345n006 cls12345n006-ib0
172.16.2.10 cls12345n007 cls12345n007-eth0
172.16.0.112 cls12345n007-ipmi
172.18.1.8 cls12345n007 cls12345n007-ib0
Example output for command: pdsh –a date | sort
[admin@cls12345n000 ~]$ pdsh -a date | sort
cls12345n000: Wed Sep 4 12:11:32 PDT 2013
cls12345n001: Wed Sep 4 12:11:32 PDT 2013
cls12345n002: Wed Sep 4 12:11:32 PDT 2013
cls12345n003: Wed Sep 4 12:11:32 PDT 2013
cls12345n004: Wed Sep 4 12:11:32 PDT 2013
cls12345n005: Wed Sep 4 12:11:32 PDT 2013
cls12345n006: Wed Sep 4 12:11:32 PDT 2013
cls12345n007: Wed Sep 4 12:11:32 PDT 2013
Example output for command: sudo ibnodes
NOTE: The sudo ibnodes command will not work on a system that uses Intel Omni-Path
switches.
[admin@cls12345n000 ~]$ sudo ibnodes
Ca : 0x001e6703003e36f7 ports 1 "cls12345n002 HCA-1"
Ca : 0x001e67030039b77c ports 1 "cls12345n001 HCA-1"
Ca : 0x0050cc0300792a37 ports 2 "cls12345n005 HCA-1"
Ca : 0x0050cc030079093e ports 2 "cls12345n004 HCA-1"
Ca : 0x001e6703003e3d9f ports 1 "cls12345n003 HCA-1"
Ca : 0x0050cc0300790a6a ports 2 "cls12345n006 HCA-1"
Ca : 0x0050cc03007920e2 ports 2 "cls12345n007 HCA-1"
Ca : 0x0002c901300ed3c2 ports 1 "cls12345n0-oem HCA-1"
Ca : 0x001e6703003e4017 ports 1 "cls12345n000 HCA-1"
Switch : 0x0002c903006769b0 ports 36 "SwitchX - Mellanox Technologies"
base port 0 lid 21 lmc 0
Switch : 0x0002c90300672330 ports 36 "SwitchX - Mellanox Technologies"
base port 0 lid 8 lmc 0
Add an MMU or AMMU to a ClusterStor™ L300 and L300N System
H-6167 20