NetApp FAS2500 Use and care manual

Doc Rev -003
README FIRST

AWebserver is needed to netboot the ONTAP kernel.

is HA. Netboot_using_Partner doc is >> here.

Request customer to download the exact version of Data ONTAP from support site to their Webserver’s
root directory
http://mysupport.netapp.com/NOW/cgi-bin/software/




This AP includes "Cluster-Mode" (C-Mode) and 7-Mode ONTAP commands. 8.3 and higher is C-Mode only.
●
The login name for C-Mode systems is "admin", not "root".
●The ONTAP version and mode is listed in your dispatch!
●
●
●

Link to Statement of Volatility by Platform is:
http://support.netapp.com/info/web/ECMP1132988.html

README FIRST
C-Mode:Has two console command shells, clustershell and nodeshell. The default shell is clustershell.
IF clustershell, the console prompt includes a double colon ( ). Ex(1): cluster > Ex(2):cluster storage>
To switch from clustershell to nodeshell, enter
'run local'
at the ::> prompt, then the double colons (::)
are removed. To exit nodeshell, enter 'exit' or Ctrl-D.
From clustershell, nodeshell commands can be entered by prefacing the 7-Mode command with “run local".
Ex: cluster::>
run local sysconfig -v
Note, all 7-Mode commands are not supported in C-Mode.
Call and inform customer that the process requires a Webserver or another NetApp controller (if 7-mode)
to serve the ONTAP files that can communicate with the service processor e0M port on the TARGET.
(Have them test using ping bi-directionally from the webserver and e0M port address.) Fix it if it fails.
AP doc rev is at top of page - If using hard-copy for secure site, be sure to print all the linked documents in this AP.
If node is running
7-mode
and the system is
HA
, you can use the
PARTNER
to serve the files. Even
another NetApp controller on same subnet can be the Webserver. The dispatch informs you if the system
OR Use your own Webserver on your laptop if you have one.
For "non-disruptive" process to work, the PARTNER node has to be able to communicate to TARGET e0M port. If
not, the "disruptive" method must be used.
A
"disruptive"
procedure will speed the process up a bit but should only be used within a maintenance window as
there is an outage of about 3 minutes. The customer can inform you if they are in a maintenance window and can
sustain an outage on the TARGET node.
For NSE enabled systems, confirm with customer that they have not power cycled the controller chassis or any disk
shelves with NSE drives. If power cycled, do NOT proceed - Contact

NGS.
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers
Select “Data ONTAP” in the <Select Software field> field and enter the ONTAP version in the version
field.
If controller is running “P” or “D” versions of ONTAP, Ex: 8.2.3P6, enter the complete ONTAP version
in the version field. i.e. “8.2.3P6” in this example.
::
::
::

This procedure will take approximately 2 hours.
I. Appliance / USB Boot Device Visual Checks VIII. Boot PROM Variable Checks
II. Node Pre-Checks IX. Prepare and Start the Netboot Process
III. Node State Check and Shutdown Procedure X.
Transfer the System Files using Non-Disruptive Procedure
IV. XI. Transfer the System Files using Disruptive Procedure
V. Remove Cable Tray and Cables, extract the MB Tray XII. Boot the Operating System - 'giveback' if applicable
VI. Replace the Boot Device XIII.
Send Autosupport, Enable Options, Submit logs, Part Return
VII. Partially Reinsert the Controller and reconnect the cables
I.
Step
1The FAS2552 is a 2u model and FAS2554 is a 4u model. Review Figures below.
2Continue with Section I on next page.
Action Description
Capture the Current System Configuration
Page 1 of 25
SECTION OUTLINE
FAS2500 Model : Appliance / USB Boot Device Visual Checks
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers
2u
FAS2520, 2552
FAS2554
4u
PSU-1
IOIOI
(Console)
Controller Module System Serial Number: Each
Controller Module Card (A or B) has
it's own. Verify the system serial
SAS Ports:
0b, 0a
Wrench port
Management (e0M), SP
10/100/1000 Mb
"Locked" wrench
Port ACP (e0P)
10/100/1000 Mb
1 Gigabit
Ethernet:
e0a, e0b
Fig 1 Fig 3
Controller Fault LED (Amber)
PSU-2
B
A
Fig 2: FAS2520,2552 Rear
A
Controller Fault LED (Amber)
PSU-1
PSU-2
PSU-4
PSU-3
B
Fig 4: FAS2554 Rear
Fig 5
AC Power
System Shelf ID
" ! " LED is ON when hardware failures are
detected or if controller failover is disabled.
10Gb Ethernet:
e0c, e0d, e0e, e0f
FAS2520
FAS2552, 2554 (shown)
HA Configuration has 2 Controllers cards,
(A & B) as shown. A single controller
system has only one card that is in slot A.

I.
Step
II.
Step
1
2Adhere to anti-static precautions. (A paper ESD strap is included inside the RMA box if you don't have your own)
3Do not remove the Boot Device from its bag until you are ready to use it.
4Go to Section III "Node State Check and Shutdown" on next page.
FAS2500 Model : Node Pre-Checks
Action Description
Verify the "Order Reference 8xxxxxxxxx number on the RMA packing slip is the same as the Part Request (PREQ) number
listed in your dispatch notes.
Page 2 of 25
FAS2500 Model : Appliance / USB Boot Device Visual Checks (cont.)
Action Description
Fig 6: USB Boot Device
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers
Base P/N is"1425A"

III.
Always capture the node’s console output to a text file,even if using the end-user's computer. To review the Job Aid on how
to connect to the console (IOIOI) port and serial emulator options, click here >
Console Attach Aid
Table-1
“A” controller is OK or powered off.
“B” controller is failed, starting up, initializing, or is in Maintenance mode.
The system is ready for service, go to step 5.
The system is ready for service, go to step 5.
Continue with Section III on next page.
Chassis has dual controllers installed and controller(s) “A”, “B” or both is failed, starting up,
initializing, or is in Maintenance mode. (NOTE: If HA is disabled, both LEDs will be ON)
“A” controller is failed, starting up, initializing, or is in Maintenance mode.
“B” controller is not installed.
Locate the Faulted Controller: Look at controller bezels that have the ( ! ) Status LED ON. LEDs: Pg-1, Fig1, 3. Will need to
examine the rear of unit to determine the chassis configuration and if it appears to be set up as a HA system. A controller with
( ! ) means the controller has some sort of FAULT.
2
Fault LEDs on Rear
What does this mean?
Both A & B controllers are functioning properly OR both powered off.
OR one controller is functioning properly and other one is powered off.
“A” controller is failed, starting up, initializing, or is in Maintenance mode.
“B” controller is OK or is powered off.
4
3
Perform a console check on each controller to determine the TARGET controller.
WARNING
HA (Active-Active) configurations: If the failure has caused the PARTNER head to takeover, you may have
been dispatched on the PARATNER's controller's serial number, not the failed one. Perform console checks.
IF Console Response is: Action:
LOADER-A|B prompt
Waiting for giveback....
Login or Password Prompt or
system-hostname>
IF HA system: Check PARATNER's console response per
WARNING above. IF both systems are UP, engage customer to
verify the s/n in your dispatch is for the controller to be serviced -
then continue.
IF non-HA: This system is still up - go to step 5.
Page 3 of 25
FAS2500 Model : Node State Check and Shutdown Procedure
Action Description
1
Contr: A
A
(LEDs in Fig 2, 4)
Contr: A
B
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers

III.
Step
5
6
Note: The approximate hours required for this maintenance is given in the 'Section Outline' section on page-1.
7
8
9Continue with Section III on next page.
Send NetApp an ASUP Message from the Target/Partner node to prevent unnecessary cases from being created by ASUP
messages during this scheduled maintenance window. Use the appropriate command shown for the specified ONTAP Mode.
If 7-Mode, skip to step 17 in this section. If C-Mode, move console cable to the PARTNER node and login as "admin". Engage
end-user for password.
Enter: storage failover show and identify the different HA-Pairs in the cluster.
Page 4 of 25
FAS2500 Model : Node State Check and Shutdown Procedure (cont.)
Action Description
Connect console cable to the TARGET node and login (7-Mode=root, C-Mode=admin). Engage end-user for password. If the
Target node is not UP, connect to the PARTNER node and login.
For 7-Mode : options autosupport.doit MAINT=<n>h <n> is the number of hours required for maintenance.
Node> options autosupport.doit MAINT=2h
Example Only
For Cluster-Mode : system node autosupport invoke -node * -type all -message MAINT=<n>h
<n> is the number of hours required for maintenance.
Cluster::> system node autosupport invoke -node * -type all -message MAINT=2h
Example Only
Clustered ONTAP
cluster::> storage failover show
Takeover
Node Partner Possible State Description
----------- ------------ -------- ----------------------
Node-01 Node-02 false In takeover
Node-02 Node-01 - Unknown
HA Pair (Node-01, Node-02)
Example: 4-Node cluster
Node-03 Node-04 true Connected to Node-04
Node-04 Node-03 true Connected to Node-03
HA Pair (Node-03, Node-04)
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers

III.
Step
10
a)
Enter: y to the question "Do you want to continue?".
11
Enter: cluster show to display information about the nodes in the cluster.
12
Check the "Health" and "Eligibility" values for all nodes other than target node in all HA groups. Does any one show "false"?
If no, go to step 13.
If yes, contact NGS to correct the problem in the node showing false before moving into next step.
13
Check the Epsilon value for the TARGET node to be serviced.
If "true" , go to step 14.
If "false", skip to step 15.
14
Follow steps (a-c) to reassign epsilon to a healthy node. (Also, See the example given below.)
a)
Enter: cluster modify -node <Target_node> -epsilon false to remove Epsilon from the target node.
b)
c)
Enter: cluster show to confirm that Epsilon has been re-assigned properly.
15
Enter: set -priv admin to return the privilege level to "admin".
16 Continue with Section III on next page.
NOTE: A typical 'cluster with only 1 HA-Pair (2-nodes)' uses 'cluster-
HA’ configuration and in that case Epsilon will show as "false" for
both nodes.
Enter: cluster modify -node <Different_node_in_different_HA_Pair> -epsilon true
to assign Epsilon to a node in a different HA-Pair.
Page 5 of 25
FAS2500 Model : Node State Check and Shutdown Procedure (cont.)
Action Description
Enter: set -priv advanced to set the privilege level to "advanced".
(This is a required step to see the 'Epsilon' column in the next command output.)
Cluster::*> cluster modify -node Node_02 -epsilon false
Cluster::*> cluster modify -node Node_04 -epsilon true
Cluster::*> cluster show
Node Health Eligibility Epsilon
------------ ------- ------------ ------------
Node_01 true true false
Node_02 true true false
Node_03 true true false
Node_04 true true true
In this example, Epsilon from Node_02 has been reassigned to Node_04, a
node member of another HA-Pair. Step to remove Epsilon
from TARGET node.
Step to add Epsilon to
different node.
Enter this command and verify the
Epsilon has been re-added properly.
Cluster::> set -priv advanced
Warning: These advanced commands are potentially dangerous; use them only when
directed to do so by NetApp personnel.
Do you want to continue? {y|n}: y
Cluster::*> cluster show
Node Health Eligibility Epsilon
------------ ------- ------------ ------------
Node_01 true true false
Node_02 true true true
Node_03 true true false
Node_04 true true false
Example: 4-Node cluster
Epsilon is seen on 4 or more node clusters. (A typical 2-node cluster doesn't use Epsilon.)
NOTE: A typical 'cluster with
only 1 HA-Pair (2-nodes)'uses
'cluster-HA’ configuration and in
that case Epsilon will show as
"false" for both nodes.
Enter this command to view node health and which node is holding epsilon (true).
Required step to see the 'Epsilon' column in the next command.
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers

III.
Step
17
18
a)
(i)
If any option(s) shows "on", turn it "off" using the below commands. Copy-n-paste the command syntax.
b)
(i)
If any option(s) shows "true", set it to "false" using the below commands. Copy-n-paste the command syntax.
19 Continue with Section III on next page.
Check if any "auto-giveback" options are enabled. If enabled, disable the option(s) - For 7-Mode, go to step a) and for Cluster-
Mode, go to step b) below.
For 7-Mode systems: Enter the below commands to check the value of "cf.giveback.auto.enable" and
"cf.giveback.auto.after.panic.takeover" options. Copy-n-paste the command syntax.
For C-Mode systems:
Enter the below commands to check the value of "-auto-giveback" and "-auto-giveback-after-panic"
parameters. Copy-n-paste the command syntax.
Page 6 of 25
FAS2500 Model : Node State Check and Shutdown Procedure (cont.)
Action Description
IF HA config, move console cable to the PARTNER node if not already connected and continue with step 18 below.
IF non-HA, skip to step 20 in this section.
node> options cf.giveback.auto.enable
cf.giveback.auto.enable on
node> options cf.giveback.auto.after.panic.takeover
cf.giveback.auto.after.panic.takeover on
If "on", enter the below
commands to turn it "off"
node> options cf.giveback.auto.enable off
node> options cf.giveback.auto.enable
cf.giveback.auto.enable off
node> options cf.giveback.auto.after.panic.takeover off
node> options cf.giveback.auto.after.panic.takeover
cf.giveback.auto.after.panic.takeover off
off
off
Verify these
options are
set to off
before
proceeding!
cluster::> sto fa show -node local -fields auto-giveback
node auto-giveback
------------- -------------
Node-B true
cluster::> sto fa show -node local -fields auto-giveback-after-panic
node auto-giveback-after-panic
------------- -------------------------
Node-B true
If "true", enter the below
commands to turn it "false"
cluster::> sto fa modify -node local -auto-giveback false
Warning: Disabling auto-giveback under cluster HA configuration
will prevent the management cluster services from automatically
going online under alternating-failure scenarios.
Do you want to disable auto-giveback? {y|n}: y
cluster::> sto fa show -node local -fields auto-giveback
node auto-giveback
------------- -------------
Node-B false
cluster::> sto fa modify -node local -auto-giveback-after-panic false
Warning: Disabling auto giveback after takeover on panic can result
in degraded performance and availability.
Do you want to disable auto-giveback-after-panic ? {y|n}: y
cluster::> sto fa show -node local -fields auto-giveback-after-panic
node auto-giveback-after-panic
------------- -------------------------
Node-B false
false
false
Verify these
options are
set to false
before
proceeding!
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers

III.
Step
20
Move the console cable to the TARGET system if not already connected and verify the console response.
a)
b)
To review the steps for performing a Manual Takeover, click >>
here.
21
a) At the "Waiting for giveback ……" prompt, Enter: Ctrl-C
b) At the message: "Do you wish to halt this node rather than wait [y/n]? " Enter: y
c) After the system drops to the LOADER-A|B> prompt, skip to step 22.
22 Go to Section IV "Capture the Current System Configuration" on next page.
IF the console response is the LOADER prompt or "Waiting for giveback", go to step 21 below.
IF HA-Pair and both nodes are up, engage end-user to “takeover” the TARGET node from the PARTNER.
IF non-HA and node is up, engage end-user to "halt" the TARGET node.
Once the "takeover" or "halt" is complete, go to step 21 below.
Move console cable to the TARGET node if not already connected. If the console response is: "Waiting for giveback…..."
follow steps (a-c) below. If console response is LOADER-A|B> skip to step 22 below.
Page 7 of 25
FAS2500 Model : Node State Check and Shutdown Procedure (cont.)
Action Description
Waiting for giveback...(Press Ctrl-C to abort wait)
^C
This node was previously declared dead.
Pausing to check HA partner status....
...............
...............
The HA partner is currently operational and in takeover mode.
...............
waiting for giveback...
Do you wish to halt this node rather than wait [y/n]? y
Halting...
.......
LOADER-A>
Enter: y
Enter: CTRL-C
Information on
Partner Status
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers

IV.
Step
1
a) At the "Waiting for giveback ……" prompt, Enter: Ctrl-C
b) At the message: "Do you wish to halt this node rather than wait [y/n]? " Enter: y
c) After the system drops to the LOADER-A|B> prompt, continue with step 2.
2
From the LOADER, enter: printenv This command displays (and captures) all boot environmental variables.
3
Identify if this is a NSE disk based system:
4Continue with Section IV on next page.
Move console cable to the TARGET node if not already connected. If the console response is: "Waiting for giveback…..."
follow steps 1a-1c. If console response is LOADER-A|B> skip to step 2.
Page 8 of 25
FAS2500 Model : Capture the Current System Configuration
Action Description
Open the "printenv" output listing captured in step 2 above. Using the "Find" function search for the variables named: bootarg
The actual variable to look for is: 'bootarg.storageencryption.support'.If this variable exists and is set to 'true' then this is
an NSE system.
STOP
The "printenv" output is corrupt if there is any question marks ( ?? ) and periods in the variable names or
values. If garbaged output the printenv is considered corrupt.
If "printenv" output is corrupt, more checks will need to be performed from the PARTNER node when
executing step 6 on next page; if non-HA config, you may need to engage NGS later in this Action Plan to
verify if this is an NSE system or not.
LOADER-A> printenv
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers
Waiting for giveback...(Press Ctrl-C to abort wait)
^C
This node was previously declared dead.
Pausing to check HA partner status....
...............
...............
The HA partner is currently operational and in takeover mode.
...............
waiting for giveback...
Do you wish to halt this node rather than wait [y/n]? y
Halting...
.......
LOADER-A>
Step 1b): Enter y
Step 1a): Enter: CTRL-C
Information on
Partner Status

IV.
Step
5
6If the "printenv" output listed in step 2 is corrupt, follow step (a) below. Otherwise, skip to step 7 below.
a)
On the PARTNER node, execute the following commands listed:
S
T
O
P
Example A:
Example B:
STOP
7
8Go to section V, "Remove Cable Tray and Cables, extract the MB Tray" on next page.
For NSE system, enter: key_manager status to check the status of all key management servers linked to the TARGET
system. If non-NSE system, skip to step 8.
FAS2500 Model : Capture the Current System Configuration (cont.)
Action Description
Page 9 of 25
If HA-config, move console cable to the PARTNER node and login (7-Mode=root, C-Mode=admin) and continue with step 6.
Engage end-user for password. Note: If non-HA, go to section V, "Remove Cable Tray and Cables, extract the MB Tray".
::>run local ----> Run this command from ClusterShell only if Clustered ONTAP.
> priv set advanced
>disk encrypt show
If the disk encrypt show command output is "ERROR: The system is not configured to run this
command",this is not an NSE disk based system (Ref Example A below).
If the command output list the disk details with Key ID info, this is an NSE system (Ref Example B).
Confirm with customer that they have not power cycled the controller chassis or any disk shelves with NSE drives. If power
cycled, do NOT proceed - Contact NGS.
cm2554-ks::> run local ----> Run this command only for Clustered ONTAP.
Type 'exit' or 'Ctrl-D' to return to the CLI
cm2554-ks-01> priv set advanced
Warning: These advanced commands are potentially dangerous; use
them only when directed to do so by NetApp
personnel.
cm2554-ks-01*> disk encrypt show
ERROR: The system is not configured to run this command.
cm2554-ks::> run local
Type 'exit' or 'Ctrl-D' to return to the CLI
cm2554-ks-01> priv set advanced
Warning: These advanced commands are potentially dangerous; use
them only when directed to do so by NetApp
personnel.
cm2554-ks-01*> disk encrypt show
Disk Key ID Locked?
---------- ---------------------------------------------------------------- -------
0d.01.8 080DAD83080D7DD7010000000000000002BACBCF1729496812F6A4D2E00BE925 Yes
0d.01.16 080DAD83080D7DD7010000000000000002BACBCF1729496812F6A4D2E00BE925 Yes
0d.01.21 080DAD83080D7DD7010000000000000002BACBCF1729496812F6A4D2E00BE925 Yes
0d.01.12 080DAD83080D7DD7010000000000000002BACBCF1729496812F6A4D2E00BE925 Yes
0d.01.17 080DAD83080D7DD7010000000000000002BACBCF1729496812F6A4D2E00BE925 Yes
0d.01.10 080DAD83080D7DD7010000000000000002BACBCF1729496812F6A4D2E00BE925 Yes
.......
.......
ERROR: The system is not configured to run this command.
This message indicates
that disk encryption is
not enabled and this is
not an NSE system.
cm2554-ks-01*> key_manager status
Key server 10.63.122.58 is responding
Key server 10.63.122.113 is responding
cm2554-ks-01*>
Take note of the Key server IPs that are responding.
(Maximum: 4 Key servers)
This disk listing indicates that this is an NSE system.
Boot Device Replacement for the FAS2500
For NetApp Authorized Service Engineers