NetApp FAS9500 Instruction sheet

Boot media
Install and maintain
NetApp
July 14, 2023
This PDF was generated from https://docs.netapp.com/us-en/ontap-
systems/fas9500/bootmedia_replace_overview.html on July 14, 2023. Always check docs.netapp.com for
the latest.

Table of Contents
Boot media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê1
Replace the boot media - FAS9500 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê1
Pre-shutdown checks for onboard encryption keys - FAS9500 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê1
Shut down the impaired controller - FAS9500 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê5
Remove the controller, replace the boot media, and transfer the boot image - FAS9500 . . . . . . . . . . . . . . . . Ê6
Boot the recovery image - FAS9500 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê11
Post boot media replacement steps for OKM, NSE, and NVE - FAS9500 . . . . . . . . . . . . . . . . . . . . . . . . . . Ê13
Return the failed part to NetApp - FAS9500. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ê17

Boot media
Replace the boot media - FAS9500
The boot media stores a primary and secondary set of system (boot image) files that the
system uses when it boots. Depending on your network configuration, you can perform
either a nondisruptive or disruptive replacement.
You must have a USB flash drive, formatted to FAT32, with the appropriate amount of storage to hold the
image_xxx.tgz.
You also must copy the image_xxx.tgz file to the USB flash drive for later use in this procedure.
•The nondisruptive and disruptive methods for replacing a boot media both require you to restore the var
file system:
◦For nondisruptive replacement, the HA pair does not require connection to a network to restore the var
file system. The HA pair in a single chassis has an internal e0S connection, which is used to transfer
var config between them.
◦For disruptive replacement, you do not need a network connection to restore the var file system, but
the process requires two reboots.
•You must replace the failed component with a replacement FRU component you received from your
provider.
•It is important that you apply the commands in these steps on the correct node:
◦The impaired node is the node on which you are performing maintenance.
◦The healthy node is the HA partner of the impaired node.
Pre-shutdown checks for onboard encryption keys -
FAS9500
Prior to shutting down the impaired controller and checking the status of the onboard
encryption keys, you must check the status of the impaired controller, disable automatic
giveback, and check the version of ONTAP that is running.
If you have a cluster with more than two nodes, it must be in quorum. If the cluster is not in quorum or a healthy
controller shows false for eligibility and health, you must correct the issue before shutting down the impaired
controller; see the Synchronize a node with the cluster.
Steps
1. Check the status of the impaired controller:
◦If the impaired controller is at the login prompt, log in as admin.
◦If the impaired controller is at the LOADER prompt and is part of HA configuration, log in as admin on
the healthy controller.
◦If the impaired controller is in a standalone configuration and at LOADER prompt, contact
mysupport.netapp.com.
2. If AutoSupport is enabled, suppress automatic case creation by invoking an AutoSupport message:
1

system node autosupport invoke -node * -type all -message
MAINT=number_of_hours_downh
The following AutoSupport message suppresses automatic case creation for two hours: cluster1:*>
system node autosupport invoke -node * -type all -message MAINT=2h
3. Check the version of ONTAP the system is running on the impaired controller if up, or on the partner
controller if the impaired controller is down, using the version -v command:
◦If <lno-DARE> or <1Ono-DARE> is displayed in the command output, the system does not support
NVE, proceed to shut down the controller.
ONTAP 9.6 and later
Before shutting down the impaired controller, you need to verify whether the system has either NetApp Volume
Encryption (NVE) or NetApp Storage Encryption (NSE) enabled. If so, you need to verify the configuration.
1. Verify whether NVE is in use for any volumes in the cluster: volume show -is-encrypted true
If any volumes are listed in the output, NVE is configured and you need to verify the NVE configuration. If
no volumes are listed, check whether NSE is configured and in use.
2. Verify whether NSE is configured and in use: storage encryption disk show
◦If the command output lists the drive details with Mode & Key ID information, NSE is configured and
you need to verify the NSE configuration and in use.
◦If no disks are shown, NSE is not configured.
◦If NVE and NSE are not configured, no drives are protected with NSE keys, it’s safe to shut down the
impaired controller.
Verify NVE configuration
1. Display the key IDs of the authentication keys that are stored on the key management servers: security
key-manager key query
After the ONTAP 9.6 release, you may have additional key manager types. The types are
KMIP, AKV, and GCP. The process for confirming these types is the same as confirming
external or onboard key manager types.
◦If the Key Manager type displays external and the Restored column displays yes, it’s safe to shut
down the impaired controller.
◦If the Key Manager type displays onboard and the Restored column displays yes, you need to
complete some additional steps.
◦If the Key Manager type displays external and the Restored column displays anything other than
yes, you need to complete some additional steps.
◦If the Key Manager type displays onboard and the Restored column displays anything other than
yes, you need to complete some additional steps.
2. If the Key Manager type displays onboard and the Restored column displays yes, manually back up
the OKM information:
a. Go to advanced privilege mode and enter ywhen prompted to continue: set -priv advanced
2

b. Enter the command to display the key management information: security key-manager onboard
show-backup
c. Copy the contents of the backup information to a separate file or your log file. You’ll need it in disaster
scenarios where you might need to manually recover OKM.
d. Return to admin mode: set -priv admin
e. Shut down the impaired controller.
3. If the Key Manager type displays external and the Restored column displays anything other than
yes:
a. Restore the external key management authentication keys to all nodes in the cluster: security key-
manager external restore
If the command fails, contact NetApp Support.
mysupport.netapp.com
b. Verify that the Restored column equals yes for all authentication keys: security key-manager
key query
c. Shut down the impaired controller.
4. If the Key Manager type displays onboard and the Restored column displays anything other than yes:
a. Enter the onboard security key-manager sync command: security key-manager onboard sync
Enter the customer’s onboard key management passphrase at the prompt. If the
passphrase cannot be provided, contact NetApp Support. mysupport.netapp.com
b. Verify the Restored column shows yes for all authentication keys: security key-manager key
query
c. Verify that the Key Manager type shows onboard, and then manually back up the OKM information.
d. Go to advanced privilege mode and enter ywhen prompted to continue: set -priv advanced
e. Enter the command to display the key management backup information: security key-manager
onboard show-backup
f. Copy the contents of the backup information to a separate file or your log file. You’ll need it in disaster
scenarios where you might need to manually recover OKM.
g. Return to admin mode: set -priv admin
h. You can safely shut down the controller.
Verify NSE configuration
1. Display the key IDs of the authentication keys that are stored on the key management servers: security
key-manager key query -key-type NSE-AK
After the ONTAP 9.6 release, you may have additional key manager types. The types are
KMIP, AKV, and GCP. The process for confirming these types is the same as confirming
external or onboard key manager types.
3

◦If the Key Manager type displays external and the Restored column displays yes, it’s safe to shut
down the impaired controller.
◦If the Key Manager type displays onboard and the Restored column displays yes, you need to
complete some additional steps.
◦If the Key Manager type displays external and the Restored column displays anything other than
yes, you need to complete some additional steps.
◦If the Key Manager type displays external and the Restored column displays anything other than
yes, you need to complete some additional steps.
2. If the Key Manager type displays onboard and the Restored column displays yes, manually back up
the OKM information:
a. Go to advanced privilege mode and enter ywhen prompted to continue: set -priv advanced
b. Enter the command to display the key management information: security key-manager onboard
show-backup
c. Copy the contents of the backup information to a separate file or your log file. You’ll need it in disaster
scenarios where you might need to manually recover OKM.
d. Return to admin mode: set -priv admin
e. You can safely shut down the controller.
3. If the Key Manager type displays external and the Restored column displays anything other than
yes:
a. Restore the external key management authentication keys to all nodes in the cluster: security key-
manager external restore
If the command fails, contact NetApp Support.
mysupport.netapp.com
b. Verify that the Restored column equals yes for all authentication keys: security key-manager
key query
c. You can safely shut down the controller.
4. If the Key Manager type displays onboard and the Restored column displays anything other than yes:
a. Enter the onboard security key-manager sync command: security key-manager onboard sync
Enter the customer’s onboard key management passphrase at the prompt. If the passphrase cannot be
provided, contact NetApp Support.
mysupport.netapp.com
b. Verify the Restored column shows yes for all authentication keys: security key-manager key
query
c. Verify that the Key Manager type shows onboard, and then manually back up the OKM information.
d. Go to advanced privilege mode and enter ywhen prompted to continue: set -priv advanced
e. Enter the command to display the key management backup information: security key-manager
onboard show-backup
4

f. Copy the contents of the backup information to a separate file or your log file. You’ll need it in disaster
scenarios where you might need to manually recover OKM.
g. Return to admin mode: set -priv admin
h. You can safely shut down the controller.
Shut down the impaired controller - FAS9500
Shut down or take over the impaired controller using one of the following options.
After completing the NVE or NSE tasks, you need to complete the shutdown of the impaired node.
To shut down the impaired controller, you must determine the status of the controller and, if necessary, take
over the controller so that the healthy controller continues to serve data from the impaired controller storage.
About this task
•If you are using NetApp Storage Encryption, you must have reset the MSID using the instructions in the
Returning SEDs to unprotected mode.
•If you have a SAN system, you must have checked event messages (cluster kernel-service show)
for impaired controller SCSI blade. The cluster kernel-service show command displays the node
name, quorum status of that node, availability status of that node, and operational status of that node.
Each SCSI-blade process should be in quorum with the other nodes in the cluster. Any issues must be
resolved before you proceed with the replacement.
•If you have a cluster with more than two nodes, it must be in quorum. If the cluster is not in quorum or a
healthy controller shows false for eligibility and health, you must correct the issue before shutting down the
impaired controller; see Synchronize a node with the cluster.
Steps
1. If AutoSupport is enabled, suppress automatic case creation by invoking an AutoSupport message:
system node autosupport invoke -node * -type all -message
MAINT=number_of_hours_downh
The following AutoSupport message suppresses automatic case creation for two hours: cluster1:>
system node autosupport invoke -node * -type all -message MAINT=2h
2. Disable automatic giveback from the console of the healthy controller: storage failover modify
–node local -auto-giveback false
When you see Do you want to disable auto-giveback?, enter y.
3. Take the impaired controller to the LOADER prompt:
If the impaired controller is
displaying…
Then…
The LOADER prompt Go to Remove controller module.
Waiting for giveback… Press Ctrl-C, and then respond ywhen prompted.
5

If the impaired controller is
displaying…
Then…
System prompt or password
prompt
Take over or halt the impaired controller from the healthy controller:
storage failover takeover -ofnode
impaired_node_name
When the impaired controller shows Waiting for giveback…, press
Ctrl-C, and then respond y.
Remove the controller, replace the boot media, and transfer
the boot image - FAS9500
You must remove and open the controller module, locate and replace the boot media in
the controller, and then transfer the image to the replacement boot media.
Step 1: Remove the controller module
To access components inside the controller, you must first remove the controller module from the system and
then remove the cover on the controller module.
Steps
1. If you are not already grounded, properly ground yourself.
2. Unplug the cables from the impaired controller module, and keep track of where the cables were
connected.
3. Slide the terra cotta button on the cam handle downward until it unlocks.
Animation - Remove controller module
6

A1
B1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
AB
2
1
Cam handle release button
Cam handle
4. Rotate the cam handle so that it completely disengages the controller module from the chassis, and then
slide the controller module out of the chassis.
Make sure that you support the bottom of the controller module as you slide it out of the chassis.
5. Place the controller module lid-side up on a stable, flat surface, press the blue button on the cover, slide
the cover to the back of the controller module, and then swing the cover up and lift it off of the controller
module.
1
7

Controller module cover locking button
Step 2: Replace the boot media
You must locate the boot media in the controller and follow the directions to replace it.
Steps
1. Lift the black air duct at the back of the controller module and then locate the boot media using the
following illustration or the FRU map on the controller module:
Animation - Replace boot media
1
2
Press release tab
Boot media
2. Press the blue button on the boot media housing to release the boot media from its housing, and then
gently pull it straight out of the boot media socket.
Do not twist or pull the boot media straight up, because this could damage the socket or the
boot media.
3. Align the edges of the replacement boot media with the boot media socket, and then gently push it into the
socket.
4. Check the boot media to make sure that it is seated squarely and completely in the socket.
If necessary, remove the boot media and reseat it into the socket.
5. Push the boot media down to engage the locking button on the boot media housing.
8

6. Reinstall the controller module lid by aligning the pins on the lid with the slots on the motherboard carrier,
and then slide the lid into place.
Step 3: Transfer the boot image to the boot media
You can install the system image to the replacement boot media using a USB flash drive with the image
installed on it. However, you must restore the var file system during this procedure.
Before you begin
•You must have a USB flash drive, formatted to FAT32, with at least 4GB capacity.
•A copy of the same image version of ONTAP as what the impaired controller was running. You can
download the appropriate image from the Downloads section on the NetApp Support Site
◦If NVE is enabled, download the image with NetApp Volume Encryption, as indicated in the download
button.
◦If NVE is not enabled, download the image without NetApp Volume Encryption, as indicated in the
download button.
•If your system is a stand-alone system you do not need a network connection, but you must perform an
additional reboot when restoring the var file system.
Steps
1. Align the end of the controller module with the opening in the chassis, and then gently push the controller
module halfway into the system.
2. Recable the controller module, as needed.
3. Insert the USB flash drive into the USB slot on the controller module.
Make sure that you install the USB flash drive in the slot labeled for USB devices, and not in the USB
console port.
4. Push the controller module all the way into the system, making sure that the cam handle clears the USB
flash drive, firmly push the cam handle to finish seating the controller module, and then push the cam
handle to the closed position.
The node begins to boot as soon as it is completely installed into the chassis.
5. Interrupt the boot process to stop at the LOADER prompt by pressing Ctrl-C when you see Starting
AUTOBOOT press Ctrl-C to abort….
If you miss this message, press Ctrl-C, select the option to boot to Maintenance mode, and then halt the
node to boot to LOADER.
6. Although the environment variables and bootargs are retained, you should check that all required boot
environment variables and bootargs are properly set for your system type and configuration using the
printenv bootarg name command and correct any errors using the setenv variable-name
<value> command.
a. Check the boot environment variables:
▪bootarg.init.boot_clustered
▪partner-sysid
▪bootarg.init.flash_optimized for AFF
▪bootarg.init.san_optimized for AFF
9

▪bootarg.init.switchless_cluster.enable
b. If External Key Manager is enabled, check the bootarg values, listed in the kenv ASUP output:
▪bootarg.storageencryption.support <value>
▪bootarg.keymanager.support <value>
▪kmip.init.interface <value>
▪kmip.init.ipaddr <value>
▪kmip.init.netmask <value>
▪kmip.init.gateway <value>
c. If Onboard Key Manager is enabled, check the bootarg values, listed in the kenv ASUP output:
▪bootarg.storageencryption.support <value>
▪bootarg.keymanager.support <value>
▪bootarg.onboard_keymanager <value>
d. Save the environment variables you changed with the savenv command
e. Confirm your changes using the printenv variable-name command.
7. Set your network connection type at the LOADER prompt:
◦If you are configuring DHCP: ifconfig e0a -auto
The target port you configure is the target port you use to communicate with the
impaired node from the healthy node during var file system restore with a network
connection. You can also use the e0M port in this command.
◦If you are configuring manual connections: ifconfig e0a -addr=filer_addr -mask=netmask
-gw=gateway-dns=dns_addr-domain=dns_domain
▪filer_addr is the IP address of the storage system.
▪netmask is the network mask of the management network that is connected to the HA partner.
▪gateway is the gateway for the network.
▪dns_addr is the IP address of a name server on your network.
▪dns_domain is the Domain Name System (DNS) domain name.
If you use this optional parameter, you do not need a fully qualified domain name in the netboot
server URL. You need only the server’s host name.
Other parameters might be necessary for your interface. You can enter help ifconfig at the
firmware prompt for details.
8. If the controller is in a stretch or fabric-attached MetroCluster, you must restore the FC adapter
configuration:
a. Boot to Maintenance mode: boot_ontap maint
b. Set the MetroCluster ports as initiators: ucadmin modify -m fc -t iniitator adapter_name
c. Halt to return to Maintenance mode: halt
10

The changes will be implemented when the system is booted.
Boot the recovery image - FAS9500
You must boot the ONTAP image from the USB drive, restore the file system, and verify
the environmental variables.
1. From the LOADER prompt, boot the recovery image from the USB flash drive: boot_recovery
The image is downloaded from the USB flash drive.
2. When prompted, either enter the name of the image or accept the default image displayed inside the
brackets on your screen.
3. Restore the var file system:
If your system has… Then…
A network connection a. Press ywhen prompted to restore the backup configuration.
b. Press ywhen prompted to overwrite
/etc/ssh/ssh_host_ecdsa_key.
c. Press ywhen prompted to confirm if the restore backup was
successful.
d. Press Ywhen prompted to the restored configuration copy.
e. Set the healthy node to advanced privilege level: set
-privilege advanced
f. Run the restore backup command: system node restore-
backup -node local -target-address
impaired_node_IP_address
g. Return the node to admin level: set -privilege admin
h. Press ywhen prompted to use the restored configuration.
i. Press ywhen prompted to reboot the node.
No network connection a. Press nwhen prompted to restore the backup configuration.
b. Reboot the system when prompted by the system.
c. Select the Update flash from backup config (sync flash) option
from the displayed menu.
If you are prompted to continue with the update, press y.
11

If your system has… Then…
No network connection and is in a
MetroCluster IP configuration
a. Press nwhen prompted to restore the backup configuration.
b. Reboot the system when prompted by the system.
c. Wait for the iSCSI storage connections to connect.
You can proceed after you see the following messages:
date-and-time [node-
name:iscsi.session.stateChanged:notice]:
iSCSI session state is changed to Connected
for the target iSCSI-target (type:
dr_auxiliary, address: ip-address).
date-and-time [node-
name:iscsi.session.stateChanged:notice]:
iSCSI session state is changed to Connected
for the target iSCSI-target (type:
dr_partner, address: ip-address).
date-and-time [node-
name:iscsi.session.stateChanged:notice]:
iSCSI session state is changed to Connected
for the target iSCSI-target (type:
dr_auxiliary, address: ip-address).
date-and-time [node-
name:iscsi.session.stateChanged:notice]:
iSCSI session state is changed to Connected
for the target iSCSI-target (type:
dr_partner, address: ip-address).
d. Select the Update flash from backup config (sync flash) option
from the displayed menu.
If you are prompted to continue with the update, press y.
4. Ensure that the environmental variables are set as expected:
a. Take the node to the LOADER prompt.
b. Check the environment variable settings with the printenv command.
c. If an environment variable is not set as expected, modify it with the
setenvenvironment_variable_name changed_value command.
d. Save your changes using the saveenv command.
5. The next depends on your system configuration:
◦If your system has onboard keymanager, NSE or NVE configured, go to Post boot media replacement
steps for OKM, NSE, and NVE
12

◦If your system does not have onboard keymanager, NSE or NVE configured, complete the steps in this
section.
6. From the LOADER prompt, enter the boot_ontap command.
If you see… Then…
The login prompt Go to the next step.
Waiting for giveback… a. Log into the partner node.
b. Confirm the target node is ready for giveback with the storage
failover show command.
7. Connect the console cable to the partner node.
8. Give back the node using the storage failover giveback -fromnode local command.
9. At the cluster prompt, check the logical interfaces with the net int -is-home false command.
If any interfaces are listed as "false", revert those interfaces back to their home port using the net int
revert command.
10. Move the console cable to the repaired node and run the version -v command to check the ONTAP
versions.
11. Restore automatic giveback if you disabled it by using the storage failover modify -node local
-auto-giveback true command.
Post boot media replacement steps for OKM, NSE, and NVE
- FAS9500
Once environment variables are checked, you must complete steps specific to restore
Onboard Key Manager (OKM), NetApp Storage Encryption (NSE) and NetApp Volume
Encryption (NVE).
1. Determine which section you should use to restore your OKM, NSE, or NVE configurations: If NSE or NVE
are enabled along with Onboard Key Manager you must restore settings you captured at the beginning of
this procedure.
◦If NSE or NVE are enabled and Onboard Key Manager is enabled, go to Restore NVE or NSE when
Onboard Key Manager is enabled.
◦If NSE or NVE are enabled for ONTAP 9.6, go to Restore NSE/NVE on systems running ONTAP 9.6
and later.
Restore NVE or NSE when Onboard Key Manager is enabled
1. Connect the console cable to the target node.
2. Use the boot_ontap command at the LOADER prompt to boot the node.
3. Check the console output:
13

If the console displays… Then…
The LOADER prompt Boot the node to the boot menu: boot_ontap menu
Waiting for giveback…. a. Enter Ctrl-C at the prompt
b. At the message: Do you wish to halt this node rather than wait
[y/n]? , enter: y
c. At the LOADER prompt, enter the boot_ontap menu command.
4. At the Boot Menu, enter the hidden command, recover_onboard_keymanager, and reply yat the
prompt.
5. Enter the passphrase for the onboard key manager you obtained from the customer at the beginning of this
procedure.
6. When prompted to enter the backup data, paste the backup data you captured at the beginning of this
section, when asked. Paste the output of security key-manager backup show OR security key-
manager onboard show-backup command.
The data is output from either security key-manager backup show or security key-
manager onboard show-backup` command.
Example of backup data:
Enter the backup data:
--------------------------BEGIN BACKUP--------------------------
TmV0QXBwIEtleSBCbG9iAAEAAAAEAAAAcAEAAAAAAADuD+byAAAAACEAAAAAAAAA
QAAAAAAAAABvOlH0AAAAAMh7qDLRyH1DBz12piVdy9ATSFMT0C0TlYFss4PDjTaV
dzRYkLd1PhQLxAWJwOIyqSr8qY1SEBgm1IWgE5DLRqkiAAAAAAAAACgAAAAAAAAA
3WTh7gAAAAAAAAAAAAAAAAIAAAAAAAgAZJEIWvdeHr5RCAvHGclo+wAAAAAAAAAA
IgAAAAAAAAAoAAAAAAAAAEOTcR0AAAAAAAAAAAAAAAACAAAAAAAJAGr3tJA/
LRzUQRHwv+1aWvAAAAAAAAAAACQAAAAAAAAAgAAAAAAAAACdhTcvAAAAAJ1PXeBf
ml4NBsSyV1B4jc4A7cvWEFY6lLG6hc6tbKLAHZuvfQ4rIbYAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA . . . .
H4nPQM0nrDRYRa9SCv8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAA
---------------------------END BACKUP---------------------------
7. At the Boot Menu select the option for Normal Boot.
The system boots to Waiting for giveback… prompt.
8. Move the console cable to the partner node and log in as admin.
9. Confirm the target node is ready for giveback with the storage failover show command.
10. Give back only the CFO aggregates with the storage failover giveback -fromnode local
14

-only-cfo-aggregates true command.
◦If the command fails because of a failed disk, physically disengage the failed disk, but leave the disk in
the slot until a replacement is received.
◦If the command fails because of an open CIFS session, check with the customer how to close out CIFS
sessions.
Terminating CIFS can cause loss of data.
◦If the command fails because the partner "not ready", wait 5 minutes for the NVRAMs to synchronize.
◦If the command fails because of an NDMP, SnapMirror, or SnapVault process, disable the process. See
the appropriate content for more information.
11. Once the giveback completes, check the failover and giveback status with the storage failover show
and storage failover show-giveback commands.
Only the CFO aggregates (root aggregate and CFO style data aggregates) will be shown.
12. If you are running ONTAP 9.6 or later, run the security key-manager onboard sync:
a. Run the security key-manager onboard sync command and then enter the passphrase when
prompted.
b. Enter the security key-manager key-query command to see a detailed view of all keys stored
in the onboard key manager and verify that the Restored column = yes/true for all authentication
keys.
If the Restored column = anything other than yes/true, contact Customer Support.
c. Wait 10 minutes for the key to synchronize across the cluster.
13. Move the console cable to the partner node.
14. Give back the target node using the storage failover giveback -fromnode local command.
15. Check the giveback status, three minutes after it reports complete, using the storage failover show
command.
If giveback is not complete after 20 minutes, contact Customer Support.
16. At the clustershell prompt, enter the net int show -is-home false command to list the logical
interfaces that are not on their home node and port.
If any interfaces are listed as false, revert those interfaces back to their home port using the net int
revert command.
17. Move the console cable to the target node and run the version -v command to check the ONTAP
versions.
18. Restore automatic giveback if you disabled it by using the storage failover modify -node local
-auto-giveback true command.
Restore NSE/NVE on systems running ONTAP 9.6 and later
1. Connect the console cable to the target node.
15

2. Use the boot_ontap command at the LOADER prompt to boot the node.
3. Check the console output:
If the console displays… Then…
The login prompt Go to step 7.
Waiting for giveback… a. Log into the partner node.
b. Confirm the target node is ready for giveback with the storage
failover show command.
4. Move the console cable to the partner node and give back the target node storage using the storage
failover giveback -fromnode local -only-cfo-aggregates true local command.
◦If the command fails because of a failed disk, physically disengage the failed disk, but leave the disk in
the slot until a replacement is received.
◦If the command fails because of an open CIFS sessions, check with customer how to close out CIFS
sessions.
Terminating CIFS can cause loss of data.
◦If the command fails because the partner is "not ready", wait 5 minutes for the NVMEMs to
synchronize.
◦If the command fails because of an NDMP, SnapMirror, or SnapVault process, disable the process. See
the appropriate content for more information.
5. Wait 3 minutes and check the failover status with the storage failover show command.
6. At the clustershell prompt, enter the net int show -is-home false command to list the logical
interfaces that are not on their home node and port.
If any interfaces are listed as false, revert those interfaces back to their home port using the net int
revert command.
7. Move the console cable to the target node and run the version -v command to check the ONTAP
versions.
8. Restore automatic giveback if you disabled it by using the storage failover modify -node local
-auto-giveback true command.
9. Use the storage encryption disk show at the clustershell prompt, to review the output.
10. Use the security key-manager key-query command to display the encryption and authentication
keys that are stored on the key management servers.
◦If the Restored column = yes/true, you are done and can proceed to complete the replacement
process.
◦If the Key Manager type = external and the Restored column = anything other than yes/true,
use the security key-manager external restore command to restore the key IDs of the
authentication keys.
If the command fails, contact Customer Support.
16

◦If the Key Manager type = onboard and the Restored column = anything other than yes/true,
use the security key-manager onboard sync command to re-sync the Key Manager type.
Use the security key-manager key-query command to verify that the Restored column =
yes/true for all authentication keys.
11. Connect the console cable to the partner node.
12. Give back the node using the storage failover giveback -fromnode local command.
13. Restore automatic giveback if you disabled it by using the storage failover modify -node local
-auto-giveback true command.
Return the failed part to NetApp - FAS9500
Return the failed part to NetApp, as described in the RMA instructions shipped with the
kit. See the Part Return & Replacements page for further information.
17

Copyright information
Copyright © 2023 NetApp, Inc. All Rights Reserved. Printed in the U.S. No part of this document covered by
copyright may be reproduced in any form or by any means—graphic, electronic, or mechanical, including
photocopying, recording, taping, or storage in an electronic retrieval system—without prior written permission
of the copyright owner.
Software derived from copyrighted NetApp material is subject to the following license and disclaimer:
THIS SOFTWARE IS PROVIDED BY NETAPP “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL
NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
NetApp reserves the right to change any products described herein at any time, and without notice. NetApp
assumes no responsibility or liability arising from the use of products described herein, except as expressly
agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any
patent rights, trademark rights, or any other intellectual property rights of NetApp.
The product described in this manual may be protected by one or more U.S. patents, foreign patents, or
pending applications.
LIMITED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set
forth in subparagraph (b)(3) of the Rights in Technical Data -Noncommercial Items at DFARS 252.227-7013
(FEB 2014) and FAR 52.227-19 (DEC 2007).
Data contained herein pertains to a commercial product and/or commercial service (as defined in FAR 2.101)
and is proprietary to NetApp, Inc. All NetApp technical data and computer software provided under this
Agreement is commercial in nature and developed solely at private expense. The U.S. Government has a non-
exclusive, non-transferrable, nonsublicensable, worldwide, limited irrevocable license to use the Data only in
connection with and in support of the U.S. Government contract under which the Data was delivered. Except
as provided herein, the Data may not be used, disclosed, reproduced, modified, performed, or displayed
without the prior written approval of NetApp, Inc. United States Government license rights for the Department
of Defense are limited to those rights identified in DFARS clause 252.227-7015(b) (FEB 2014).
Trademark information
NETAPP, the NETAPP logo, and the marks listed at http://www.netapp.com/TM are trademarks of NetApp, Inc.
Other company and product names may be trademarks of their respective owners.
18