NetApp AFF A400 User manual

Controller module
Install and maintain
NetApp
February 26, 2023
This PDF was generated from https://docs.netapp.com/us-en/ontap-systems/a400/controller-replace-
overview.html on February 26, 2023. Always check docs.netapp.com for the latest.

Table of Contents
Controller module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê1
Overview of controller module replacement - AFF A400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê1
Shut down the impaired controller - AFF A400. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê1
Replace the controller module hardware - AFF A400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê5
Restore and verify the system configuration - AFF A400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê14
Recable the system and reassign disks - AFF A400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê16
Complete system restoration - AFF A400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê19

Controller module
Overview of controller module replacement - AFF A400
You must review the prerequisites for the replacement procedure and select the correct
one for your version of the ONTAP operating system.
•All drive shelves must be working properly.
•If your system is in a MetroCluster configuration, you must review the section Choosing the correct
recovery procedure to determine whether you should use this procedure.
If this is the procedure you should use, note that the controller replacement procedure for a controller in a
four or eight node MetroCluster configuration is the same as that in an HA pair. No MetroCluster-specific
steps are required because the failure is restricted to an HA pair and storage failover commands can be
used to provide nondisruptive operation during the replacement.
•You must replace the failed component with a replacement FRU component you received from your
provider.
•You must be replacing a controller module with a controller module of the same model type. You cannot
upgrade your system by just replacing the controller module.
•You cannot change any drives or drive shelves as part of this procedure.
•In this procedure, the boot device is moved from the impaired controller to the replacement controller so
that the replacement controller will boot up in the same version of ONTAP as the old controller module.
•It is important that you apply the commands in these steps on the correct systems:
◦The impaired controller is the controller that is being replaced.
◦The replacement node is the new controller that is replacing the impaired controller.
◦The healthy controller is the surviving controller.
•You must always capture the controller’s console output to a text file.
This provides you a record of the procedure so that you can troubleshoot any issues that you might
encounter during the replacement process.
Shut down the impaired controller - AFF A400
Shut down or take over the impaired controller using the appropriate procedure for your
configuration.
1

Option 1: Most systems
To shut down the impaired controller, you must determine the status of the controller
and, if necessary, take over the controller so that the healthy controller continues to
serve data from the impaired controller storage.
About this task
•If you are using NetApp Storage Encryption, you must have reset the MSID using the instructions in
the Returning SEDs to unprotected mode.
•If you have a SAN system, you must have checked event messages (cluster kernel-service
show) for impaired controller SCSI blade. The cluster kernel-service show command
displays the node name, quorum status of that node, availability status of that node, and operational
status of that node.
Each SCSI-blade process should be in quorum with the other nodes in the cluster. Any issues must
be resolved before you proceed with the replacement.
•If you have a cluster with more than two nodes, it must be in quorum. If the cluster is not in quorum or
a healthy controller shows false for eligibility and health, you must correct the issue before shutting
down the impaired controller; see Synchronize a node with the cluster.
Steps
1. If AutoSupport is enabled, suppress automatic case creation by invoking an AutoSupport message:
system node autosupport invoke -node * -type all -message
MAINT=number_of_hours_downh
The following AutoSupport message suppresses automatic case creation for two hours: cluster1:>
system node autosupport invoke -node * -type all -message MAINT=2h
2. Disable automatic giveback from the console of the healthy controller: storage failover modify
–node local -auto-giveback false
When you see Do you want to disable auto-giveback?, enter y.
3. Take the impaired controller to the LOADER prompt:
If the impaired controller is
displaying…
Then…
The LOADER prompt Go to Remove controller module.
Waiting for giveback… Press Ctrl-C, and then respond ywhen prompted.
System prompt or password
prompt
Take over or halt the impaired controller from the healthy
controller: storage failover takeover -ofnode
impaired_node_name
When the impaired controller shows Waiting for giveback…, press
Ctrl-C, and then respond y.
2

Option 2: Controller is in a two-node MetroCluster
To shut down the impaired controller, you must determine the status of the controller
and, if necessary, switch over the controller so that the healthy controller continues to
serve data from the impaired controller storage.
About this task
•If you are using NetApp Storage Encryption, you must have reset the MSID using the instructions in
the "Return a FIPS drive or SED to unprotected mode" section of NetApp Encryption overview with
the CLI.
•You must leave the power supplies turned on at the end of this procedure to provide power to the
healthy controller.
Steps
1. Check the MetroCluster status to determine whether the impaired controller has automatically
switched over to the healthy controller: metrocluster show
2. Depending on whether an automatic switchover has occurred, proceed according to the following
table:
If the impaired controller… Then…
Has automatically switched over Proceed to the next step.
Has not automatically switched
over
Perform a planned switchover operation from the healthy
controller: metrocluster switchover
Has not automatically switched
over, you attempted switchover
with the metrocluster
switchover command, and
the switchover was vetoed
Review the veto messages and, if possible, resolve the issue and
try again. If you are unable to resolve the issue, contact technical
support.
3. Resynchronize the data aggregates by running the metrocluster heal -phase aggregates
command from the surviving cluster.
controller_A_1::> metrocluster heal -phase aggregates
[Job 130] Job succeeded: Heal Aggregates is successful.
If the healing is vetoed, you have the option of reissuing the metrocluster heal command with
the -override-vetoes parameter. If you use this optional parameter, the system overrides any soft
vetoes that prevent the healing operation.
4. Verify that the operation has been completed by using the metrocluster operation show command.
3

controller_A_1::> metrocluster operation show
Ê Operation: heal-aggregates
Ê State: successful
Start Time: 7/25/2016 18:45:55
Ê End Time: 7/25/2016 18:45:56
Ê Errors: -
5. Check the state of the aggregates by using the storage aggregate show command.
controller_A_1::> storage aggregate show
Aggregate Size Available Used% State #Vols Nodes
RAID Status
--------- -------- --------- ----- ------- ------ ----------------
------------
...
aggr_b2 227.1GB 227.1GB 0% online 0 mcc1-a2
raid_dp, mirrored, normal...
6. Heal the root aggregates by using the metrocluster heal -phase root-aggregates
command.
mcc1A::> metrocluster heal -phase root-aggregates
[Job 137] Job succeeded: Heal Root Aggregates is successful
If the healing is vetoed, you have the option of reissuing the metrocluster heal command with
the -override-vetoes parameter. If you use this optional parameter, the system overrides any soft
vetoes that prevent the healing operation.
7. Verify that the heal operation is complete by using the metrocluster operation show command
on the destination cluster:
mcc1A::> metrocluster operation show
Ê Operation: heal-root-aggregates
Ê State: successful
ÊStart Time: 7/29/2016 20:54:41
Ê End Time: 7/29/2016 20:54:42
Ê Errors: -
8. On the impaired controller module, disconnect the power supplies.
4

Replace the controller module hardware - AFF A400
To replace the controller module hardware, you must remove the impaired controller,
move FRU components to the replacement controller module, install the replacement
controller module in the chassis, and then boot the system to Maintenance mode.
Step 1: Remove the controller module
To access components inside the controller module, you must remove the controller module from the chassis.
You can use the following , illustration, or the written steps to remove the controller module from the chassis.
Animation - Remove the controller module
1. If you are not already grounded, properly ground yourself.
2. Release the power cable retainers, and then unplug the cables from the power supplies.
3. Loosen the hook and loop strap binding the cables to the cable management device, and then unplug the
system cables and SFPs (if needed) from the controller module, keeping track of where the cables were
connected.
Leave the cables in the cable management device so that when you reinstall the cable management
device, the cables are organized.
4. Remove the cable management device from the controller module and set it aside.
5. Press down on both of the locking latches, and then rotate both latches downward at the same time.
The controller module moves slightly out of the chassis.
6. Slide the controller module out of the chassis.
5

Make sure that you support the bottom of the controller module as you slide it out of the chassis.
7. Place the controller module on a stable, flat surface.
8. On the replacement controller module, open the air duct and remove the empty risers from the controller
module using the animation, illustration, or the written steps:
Animation - Remove the empty risers from the replacement controller module
a. Press the locking tabs on the sides of the air duct in toward the middle of the controller module.
b. Slide the air duct toward the back of the controller module, and then rotate it upward to its completely open
position.
c. Rotate the riser locking latch on the left side of riser 1 up and toward air duct, lift the riser up, and then set it
aside.
d. Repeat the previous step for the remaining risers.
Step 2: Move the power supplies
You must move the power supply from the impaired controller module to the replacement controller module
when you replace a controller module.
You can use the following animation, illustration, or the written steps to move the power supplies to the
replacement controller module.
Animation - Move the power supplies
1. Remove the power supply:
6

a. Rotate the cam handle so that it can be used to pull the power supply out of the chassis.
b. Press the blue locking tab to release the power supply from the chassis.
c. Using both hands, pull the power supply out of the chassis, and then set it aside.
2. Move the power supply to the new controller module, and then install it.
3. Using both hands, support and align the edges of the power supply with the opening in the controller
module, and then gently push the power supply into the controller module until the locking tab clicks into
place.
The power supplies will only properly engage with the internal connector and lock in place one way.
To avoid damaging the internal connector, do not use excessive force when sliding the
power supply into the system.
4. Repeat the preceding steps for any remaining power supplies.
Step 3: Move the NVDIMM battery
To move the NVDIMM battery from the impaired controller module to the replacement controller module, you
must perform a specific sequence of steps.
You can use the following animation, illustration, or the written steps to move the NVDIMM battery from the
impaired controller module to the replacement controller module.
Animation - Move the NVDIMM battery
7

1. Open the air duct:
a. Press the locking tabs on the sides of the air duct in toward the middle of the controller module.
b. Slide the air duct toward the back of the controller module, and then rotate it upward to its completely
open position.
2. Locate the NVDIMM battery in the controller module.
3. Locate the battery plug and squeeze the clip on the face of the battery plug to release the plug from the
socket, and then unplug the battery cable from the socket.
4. Grasp the battery and press the blue locking tab marked PUSH, and then lift the battery out of the holder
and controller module.
5. Move the battery to the replacement controller module.
6. Align the battery module with the opening for the battery, and then gently push the battery into slot until it
locks into place.
Do not plug the battery cable back into the motherboard until instructed to do so.
Step 4: Move the boot media
You must locate the boot media, and then follow the directions to remove it from the impaired controller module
and insert it into the replacement controller module.
8

You can use the following animation, illustration, or the written steps to move the boot media from the impaired
controller module to the replacement controller module.
Animation - Move the boot media
1. Locate and remove the boot media from the controller module:
a. Press the blue button at the end of the boot media until the lip on the boot media clears the blue button.
b. Rotate the boot media up and gently pull the boot media out of the socket.
2. Move the boot media to the new controller module, align the edges of the boot media with the socket
housing, and then gently push it into the socket.
3. Check the boot media to make sure that it is seated squarely and completely in the socket.
If necessary, remove the boot media and reseat it into the socket.
4. Lock the boot media in place:
a. Rotate the boot media down toward the motherboard.
b. Press the blue locking button so that it is in the open position.
c. Placing your fingers at the end of the boot media by the blue button, firmly push down on the boot
media end to engage the blue locking button.
9

Step 5: Move the PCIe risers and mezzanine card
As part of the controller replacement process, you must move the PCIe risers and mezzanine card from the
impaired controller module to the replacement controller module.
You can use the following animations, illustrations, or the written steps to move the PCIe risers and mezzanine
card from the impaired controller module to the replacement controller module.
Moving PCIe riser 1 and 2 (left and middle risers):
Animation - Move PCI risers 1 and 2
Moving the mezzanine card and riser 3 (right riser):
Animation - Move the mezzanine card and riser 3
1. Move PCIe risers one and two from the impaired controller module to the replacement controller module:
a. Remove any SFP or QSFP modules that might be in the PCIe cards.
b. Rotate the riser locking latch on the left side of the riser up and toward air duct.
The riser raises up slightly from the controller module.
c. Lift the riser up, and then move it to the replacement controller module.
d. Align the riser with the pins to the side of the riser socket, lower the riser down on the pins, push the
riser squarely into the socket on the motherboard, and then rotate the latch down flush with the sheet
metal on the riser.
e. Repeat this step for riser number 2.
2. Remove riser number 3, remove the mezzanine card, and install both into the replacement controller
10

module:
a. Remove any SFP or QSFP modules that might be in the PCIe cards.
b. Rotate the riser locking latch on the left side of the riser up and toward air duct.
The riser raises up slightly from the controller module.
c. Lift the riser up, and then set it aside on a stable, flat surface.
d. Loosen the thumbscrews on the mezzanine card, and gently lift the card directly out of the socket, and
then move it to the replacement controller module.
e. Install the mezzanine in the replacement controller and secure it with the thumbscrews.
f. Install the third riser in the replacement controller module.
Step 6: Move the DIMMs
You need to locate the DIMMs, and then move them from the impaired controller module to the replacement
controller module.
You must have the new controller module ready so that you can move the DIMMs directly from the impaired
controller module to the corresponding slots in the replacement controller module.
You can use the following animation, illustration, or the written steps to move the DIMMs from the impaired
controller module to the replacement controller module.
Animation - Move the DIMMs
11

1. Locate the DIMMs on your controller module.
2. Note the orientation of the DIMM in the socket so that you can insert the DIMM in the replacement
controller module in the proper orientation.
3. Verify that the NVDIMM battery is not plugged into the new controller module.
4. Move the DIMMs from the impaired controller module to the replacement controller module:
Make sure that you install the each DIMM into the same slot it occupied in the impaired
controller module.
a. Eject the DIMM from its slot by slowly pushing apart the DIMM ejector tabs on either side of the DIMM,
and then slide the DIMM out of the slot.
Carefully hold the DIMM by the edges to avoid pressure on the components on the
DIMM circuit board.
12

b. Locate the corresponding DIMM slot on the replacement controller module.
c. Make sure that the DIMM ejector tabs on the DIMM socket are in the open position, and then insert the
DIMM squarely into the socket.
The DIMMs fit tightly in the socket, but should go in easily. If not, realign the DIMM with the socket and
reinsert it.
d. Visually inspect the DIMM to verify that it is evenly aligned and fully inserted into the socket.
e. Repeat these substeps for the remaining DIMMs.
5. Plug the NVDIMM battery into the motherboard.
Make sure that the plug locks down onto the controller module.
Step 7: Install the controller module
After all of the components have been moved from the impaired controller module to the replacement controller
module, you must install the replacement controller module into the chassis, and then boot it to Maintenance
mode.
You can use the following animation, illustration, or the written steps to install the replacement controller
module in the chassis.
Animation - Install the controller module
1. If you have not already done so, close the air duct.
2. Align the end of the controller module with the opening in the chassis, and then gently push the controller
module halfway into the system.
Do not completely insert the controller module in the chassis until instructed to do so.
3. Cable the management and console ports only, so that you can access the system to perform the tasks in
the following sections.
13

You will connect the rest of the cables to the controller module later in this procedure.
4. Complete the installation of the controller module:
a. Plug the power cord into the power supply, reinstall the power cable locking collar, and then connect
the power supply to the power source.
b. Using the locking latches, firmly push the controller module into the chassis until the locking latches
begin to rise.
Do not use excessive force when sliding the controller module into the chassis to avoid
damaging the connectors.
c. Fully seat the controller module in the chassis by rotating the locking latches upward, tilting them so
that they clear the locking pins, gently push the controller all the way in, and then lower the locking
latches into the locked position.
The controller module begins to boot as soon as it is fully seated in the chassis. Be prepared to
interrupt the boot process.
d. If you have not already done so, reinstall the cable management device.
e. Interrupt the normal boot process and boot to LOADER by pressing Ctrl-C.
If your system stops at the boot menu, select the option to boot to LOADER.
f. At the LOADER prompt, enter bye to reinitialize the PCIe cards and other components.
g. Interrupt the boot process and boot to the LOADER prompt by pressing Ctrl-C.
If your system stops at the boot menu, select the option to boot to LOADER.
Restore and verify the system configuration - AFF A400
After completing the hardware replacement and booting to Maintenance mode, you verify
the low-level system configuration of the replacement controller and reconfigure system
settings as necessary.
Step 1: Set and verify system time after replacing the controller
You should check the time and date on the replacement controller module against the healthy controller
module in an HA pair, or against a reliable time server in a stand-alone configuration. If the time and date do
not match, you must reset them on the replacement controller module to prevent possible outages on clients
due to time differences.
About this task
It is important that you apply the commands in the steps on the correct systems:
•The replacement node is the new node that replaced the impaired node as part of this procedure.
•The healthy node is the HA partner of the replacement node.
Steps
14

1. If the replacement node is not at the LOADER prompt, halt the system to the LOADER prompt.
2. On the healthy node, check the system time: show date
The date and time are given in GMT.
3. At the LOADER prompt, check the date and time on the replacement node: show date
The date and time are given in GMT.
4. If necessary, set the date in GMT on the replacement node: set date mm/dd/yyyy
5. If necessary, set the time in GMT on the replacement node: set time hh:mm:ss
6. At the LOADER prompt, confirm the date and time on the replacement node: show date
The date and time are given in GMT.
Step 2: Verify and set the HA state of the controller module
You must verify the HA state of the controller module and, if necessary, update the state to match your system
configuration.
1. In Maintenance mode from the new controller module, verify that all components display the same HA
state: ha-config show
The HA state should be the same for all components.
2. If the displayed system state of the controller module does not match your system configuration, set the HA
state for the controller module: ha-config modify controller ha-state
The value for HA-state can be one of the following:
◦ha
◦mcc
◦mcc-2n
◦mccip
◦non-ha
3. If the displayed system state of the controller module does not match your system configuration, set the HA
state for the controller module: ha-config modify controller ha-state
4. Confirm that the setting has changed: ha-config show
Step 3: Run diagnostics
After you have replaced a component in your system, you should run diagnostic tests on that component.
Your system must be at the LOADER prompt to start diagnostics.
All commands in the diagnostic procedures are issued from the controller where the component is being
replaced.
15

1. If the controller to be serviced is not at the LOADER prompt, reboot the controller: system node halt
-node node_name
After you issue the command, you should wait until the system stops at the LOADER prompt.
2. At the LOADER prompt, access the special drivers specifically designed for system-level diagnostics to
function properly: boot_diags
3. Select Scan System from the displayed menu to enable running the diagnostics tests.
4. Select Test system from the displayed menu to run diagnostics tests.
5. Select the test or series of tests from the various sub-menus.
6. Proceed based on the result of the preceding step:
◦If the test failed, correct the failure, and then rerun the test.
◦If the test reported no failures, select Reboot from the menu to reboot the system.
During the boot process, you might see the following prompts:
▪A prompt warning of a system ID mismatch and asking to override the system ID.
▪A prompt warning that when entering Maintenance mode in an HA configuration you must ensure
that the healthy controller remains down. You can safely respond yto these prompts.
Recable the system and reassign disks - AFF A400
Continue the replacement procedure by recabling the storage and confirming disk
reassignment.
Step 1: Recable the system
After running diagnostics, you must recable the controller module’s storage and network connections.
Steps
1. Recable the system.
2. Verify that the cabling is correct by using Active IQ Config Advisor.
a. Download and install Config Advisor.
b. Enter the information for the target system, and then click Collect Data.
c. Click the Cabling tab, and then examine the output. Make sure that all disk shelves are displayed and
all disks appear in the output, correcting any cabling issues you find.
d. Check other cabling by clicking the appropriate tab, and then examining the output from Config Advisor.
Step 2: Reassign disks
If the storage system is in an HA pair, the system ID of the new controller module is automatically assigned to
the disks when the giveback occurs at the end of the procedure. You must confirm the system ID change when
you boot the replacement controller and then verify that the change was implemented.
This procedure applies only to systems running ONTAP in an HA pair.
16

1. If the replacement controller is in Maintenance mode (showing the *> prompt, exit Maintenance mode and
go to the LOADER prompt: halt
2. From the LOADER prompt on the replacement controller, boot the controller, entering yif you are prompted
to override the system ID due to a system ID mismatch:boot_ontap
3. Wait until the Waiting for giveback… message is displayed on the replacement controller console and
then, from the healthy controller, verify that the new partner system ID has been automatically assigned:
storage failover show
In the command output, you should see a message that the system ID has changed on the impaired
controller, showing the correct old and new IDs. In the following example, node2 has undergone
replacement and has a new system ID of 151759706.
node1> `storage failover show`
Ê Takeover
Node Partner Possible State Description
------------ ------------ --------
-------------------------------------
node1 node2 false System ID changed on
partner (Old:
Ê 151759755, New:
151759706), In takeover
node2 node1 - Waiting for giveback
(HA mailboxes)
4. From the healthy controller, verify that any coredumps are saved:
a. Change to the advanced privilege level: set -privilege advanced
You can respond Ywhen prompted to continue into advanced mode. The advanced mode prompt
appears (*>).
b. Save any coredumps: system node run -node local-node-name partner savecore
c. Wait for the `savecore`command to complete before issuing the giveback.
You can enter the following command to monitor the progress of the savecore command: system
node run -node local-node-name partner savecore -s
d. Return to the admin privilege level: set -privilege admin
5. If your storage system has Storage or Volume Encryption configured, you must restore Storage or Volume
Encryption functionality by using one of the following procedures, depending on whether you are using
onboard or external key management:
◦Restore onboard key management encryption keys
◦Restore external key management encryption keys
6. Give back the controller:
a. From the healthy controller, give back the replaced controller’s storage: storage failover
giveback -ofnode replacement_node_name
17

The replacement controller takes back its storage and completes booting.
If you are prompted to override the system ID due to a system ID mismatch, you should enter y.
If the giveback is vetoed, you can consider overriding the vetoes.
Find the High-Availability Configuration content for your version of ONTAP 9
b. After the giveback has been completed, confirm that the HA pair is healthy and that takeover is
possible: storage failover show
The output from the storage failover show command should not include the System ID changed
on partner message.
7. Verify that the disks were assigned correctly: storage disk show -ownership
The disks belonging to the replacement controller should show the new system ID. In the following
example, the disks owned by node1 now show the new system ID, 1873775277:
node1> `storage disk show -ownership`
Disk Aggregate Home Owner DR Home Home ID Owner ID DR Home ID
Reserver Pool
----- ------ ----- ------ -------- ------- ------- -------
--------- ---
1.0.0 aggr0_1 node1 node1 - 1873775277 1873775277 -
1873775277 Pool0
1.0.1 aggr0_1 node1 node1 1873775277 1873775277 -
1873775277 Pool0
.
.
.
8. If the system is in a MetroCluster configuration, monitor the status of the controller: metrocluster node
show
The MetroCluster configuration takes a few minutes after the replacement to return to a normal state, at
which time each controller will show a configured state, with DR Mirroring enabled and a mode of normal.
The metrocluster node show -fields node-systemid command output displays the old system
ID until the MetroCluster configuration returns to a normal state.
9. If the controller is in a MetroCluster configuration, depending on the MetroCluster state, verify that the DR
home ID field shows the original owner of the disk if the original owner is a controller on the disaster site.
This is required if both of the following are true:
◦The MetroCluster configuration is in a switchover state.
◦The replacement controller is the current owner of the disks on the disaster site.
Disk ownership changes during HA takeover and MetroCluster switchover in a four-node MetroCluster
18