VERITAS Sun Enterprise 10000 User manual

March 2005
N16815F
VERITAS Cluster Server
Application Note: Sun Enterprise 10000
Dynamic Reconfiguration
Solaris

ii VCS Application Note: Sun E10000 Dynamic Reconfiguraion
Disclaimer
The information contained in this publication is subject to change without notice. VERITAS Software
Corporation makes no warranty of any kind with regard to this manual, including, but not limited to,
the implied warranties of merchantability and fitness for a particular purpose. VERITAS Software
Corporation shall not be liable for errors contained herein or for incidental or consequential damages
in connection with the furnishing, performance, or use of this manual.
VERITAS Legal Notice
Copyright © 2003-2005 VERITAS Software Corporation. All rights reserved. VERITAS, the VERITAS
Logo, and all other VERITAS product names and slogans are trademarks or registered trademarks of
VERITAS Software Corporation. VERITAS, the VERITAS logo, Reg. U.S. Pat. & Tm. Off. Other product
names and/or slogans mentioned herein may be trademarks or registered trademarks of their
respective companies.
VERITAS Software Corporation
350 Ellis Street
Mountain View, CA 94043
USA
Phone [phone] Fax [phone]
http://www.veritas.com

1
VCS Application Note: Sun Enterprise 10000
Dynamic Reconfiguration
Introduction
This application note describes how to perform Dynamic Reconfiguration (DR) operations
on VCS clustered system domains of the SunTM EnterpriseTM 10000 server.
The Solaris DR utility enables you to reconfigure the resources of system boards so that
the boards can be replaced without system downtime. Before you can physically remove a
board, you must “detach” it, or reconfigure it such that its resources can be disabled and
removed from the domain configuration. Likewise, after you have physically replaced a
board in a domain, you must “attach” it, or reconfigure it into the domain.
Note Currently, VCS does not support using DR in clusters where I/O controllers and
storage use Sun’s Alternate Pathing (AP).
In a VCS cluster of domains, it’s possible that the DR operations in a domain might cause
VCS to detect that resources are not available and to initiate failover to the other domain.
Therefore, it is advisable to freeze the service groups running in the domain before
running DR.
Do not use the following procedures to DR a system board containing a VCS private
heartbeat link. If you need to do so, you must stop VCS before proceeding.
Supported Hardware
◆Sun Enterprise 10000 (E10K)
Supported Software
◆Solaris 8, 9, and 10
◆VERITAS Cluster Server, version 2.0, 3.5, 4.0, and 4.1
◆VERITAS VolumeManager (VxVM), as supported by the VCS version
◆VERITAS File System, as supported by the VCS version

Dynamic Reconfiguration in VCS Environment
2 VCS Application Note: Sun E10000 Dynamic Reconfiguration
Dynamic Reconfiguration in VCS Environment
The system board in a domain may contain I/O controllers, CPUs, or memory. Typically,
within a domain, system boards have their functions duplicated on other system boards.
For example, a board with CPU or memory can be removed dynamically because another
board has the equivalent functions.
Boards with I/O controllers can be dynamically reconfigured as long as you use VxVM
with the Dynamic Multipathing (DMP) feature to manage the shared storage.
Setting Environment Variable to Enable DR
Before performing DR operations on a domain, you must first set the appropriate
environment variable.
Enable the Kernel Cage Variable for DR
Using the Solaris 8 operating environment, you must set the system(4) variable,
kernel_cage_enable, to 1 (enabled). By default, this variable is set to zero (kernel cage
disabled), preventing DR Detach operations.
Edit the file /etc/system so that kernel_cage_enable equals 1.
.
set kernel_cage_enable=1
.
Reboot the domain. To verify the kernel cage is enabled, check the file
/var/adm/messages. Look for the message:
NOTICE: DR Kernel Cage is ENABLED

Chapter , 3
Dynamic Reconfiguration in VCS Environment
Detaching and Attaching I/O System Boards
In the configuration shown below, VCS runs on Domains A and B with service groups
online on Domain A. Shared storage consists of a VxVM disk group with Dynamic
Multipathing (DMP) enabled. Dynamic Reconfiguration of I/O boards depends on DMP
being configured for the storage.
In the example, the system board SB3, which has a disk controller, is to be removed,
repaired, and replaced. The administrator disables the controller, and the disk controller
on SB1 automatically takes over because of the DMP functionality. Using DR commands,
the administrator can detach, or remove the board from the Domain A’s configuration.
When this is complete, the board can be physically removed.
Replacing the board—a controller board in this case—involves physically installing it and
reconnecting it to the shared storage. Reconfiguring the board requires using DR
commands to “attach” it to the domain, after which the controller can be re-enabled.
c1 c3 c4
DOMAIN BDOMAIN A
I/O STORAGE
SB1 SB2 SB3 SB4 SB9 SB10 SB11 SB12
c2 c3
CPU/
MEM
CPU/
MEM
CPU/
MEM
CPU/
MEM
I/OI/O I/O I/O

Dynamic Reconfiguration in VCS Environment
4 VCS Application Note: Sun E10000 Dynamic Reconfiguration
Detaching I/O System Boards with DMP Enabled
Make sure the kernel_cage_enable variable is set.
1. Freeze the VCS service groups running on the domain where you intend to perform
DR operations. Freezing the service groups prevents them from being taken offline or
failed over. Repeat the following command for each service group:
# hagrp -freeze ser_grp_name
2. Connect to the SSP server and log in to the domain whose system board requires
Dynamic Reconfiguration (DR).
ssp:D1% echo $SUNW_HOSTNAME
3. Enter the dr(1M) shell:
ssp:D1% dr
4. To verify the board is an I/O board, enter:
dr> drshow sb# IO
If the display lists the disks connected to the controller, the system board is an I/O
board.
5. If the system board is an I/O board, open another window and log in as root to the
domain you are currently reconfiguring.
6. Disable the controller on the I/O system board:
# vxdmpadm disable ctlr=ctlr#
7. In the window where you are running DR, start detaching the I/O board by entering:
dr> drain sb#
8. Monitor the progress of the drain operation by entering:
dr> drshow sb# drain
9. When you see the message:
Percent Complete= 100% (0 KBytes remaining)
complete the detach operation:
dr> complete_detach sb#

Chapter , 5
Dynamic Reconfiguration in VCS Environment
10. To verify that the board is no longer configured, type the following command:
dr> drshow sb#
The detached board should not appear in the detailed listing.
11. Exit the dr shell:
dr> exit
12. If the board is not to be immediately replaced, unfreeze any frozen service groups:
# hagrp -unfreeze ser_grp_name
Repeat for each service group.
Attaching I/O System Boards with DMP Enabled
1. Freeze the VCS service groups running on the domain where you intend to attach a
system board. Repeat the following command for each service group:
# hagrp -freeze ser_grp_name
2. After physically replacing a previously removed I/O board, make sure it is connected
to the shared storage.
3. From the SSP server, enter the dr(1M) shell:
ssp:D1% dr
4. Follow the Sun procedure to attach the system board, described here briefly:
dr> init_attach sb#
Complete the attach operation:
dr> complete_attach sb#
5. Verify that the DR attach operation has succeeded. Type:
dr> drshow #sb
The new system board should show in the list of configured boards.
6. Exit the dr shell.
dr> exit

Dynamic Reconfiguration in VCS Environment
6 VCS Application Note: Sun E10000 Dynamic Reconfiguration
7. Log in as root to the domain where you are adding the system board. Enable the
controller by entering:
# vxdmpadm enable ctlr=ctlr#
8. When you have successfully attached and enabled the system I/O board, unfreeze
any frozen service groups:
# hagrp -unfreeze ser_grp_name
Repeat for each service group.
9. Verify that VCS is still up and running.
Detaching CPU/Memory Boards
Note Use the following procedure if no I/O devices on the system board are used.
Make sure the kernel_cage_enable variable is set.
1. Freeze the VCS service groups running on the domain where you intend to detach a
CPU/Memory board. Freezing the service groups prevents them from being taken
offline or failed over. Repeat the following command for each service group:
# hagrp -freeze ser_grp_name
2. Connect to the SSP server and log in to the domain whose system board requires
Dynamic Reconfiguration (DR).
ssp:D1% echo $SUNW_HOSTNAME
3. Enter the dr(1M) shell:
ssp:D1% dr
4. In the window where you are running DR, start detaching the I/O board by entering:
dr> drain sb#
5. Monitor the progress of the drain operation by entering:
dr> drshow sb# drain
6. When you see the message
Percent Complete= 100% (0 KBytes remaining)
complete the detach operation:
dr> complete_detach sb#

Chapter , 7
Dynamic Reconfiguration in VCS Environment
7. To verify that the board is no longer configured, type the following command:
dr> drshow sb#
The detached board should not appear in the detailed listing.
8. Exit the dr shell:
dr > exit
9. If the board is not to be immediately replaced, unfreeze any frozen service groups:
# hagrp -unfreeze ser_grp_name
10. Repeat for each service group.
Attaching CPU/Memory Boards
Note The following procedure can be used if none of the I/O devices on the system board
are used.
1. Freeze the VCS service groups running on the domain where you intend to attach a
system board. Repeat the following command for each service group:
# hagrp -freeze ser_grp_name
2. Physically replace the CPU/Memory board.
3. From the SSP server, enter the dr(1M) shell:
ssp:D1% dr
4. Follow the Sun procedure to attach the system board, described here briefly:
dr> init_attach sb#
Complete the attach operation:
dr> complete_attach sb#
5. Verify that the DR attach operation has succeeded. Type:
dr> drshow #sb
The new system board should show in the list of configured boards.
6. Exit the dr shell.
dr> exit

Using VM without DMP Enabled
8 VCS Application Note: Sun E10000 Dynamic Reconfiguration
7. When you have successfully attached the CPU/Memory board, unfreeze any frozen
service groups:
# hagrp -unfreeze ser_grp_name
Repeat for each service group.
8. Verify that VCS is still up and running.
Using VM without DMP Enabled
If you have the Volume Manager DMP feature disabled for some or all of the disks in the
shared storage, and you must perform DR operations within the cluster, we recommend
using the VCS DiskReservation agent to guard against data corruption. In the event of a
“split-brain” condition, that is, when two processors in a cluster can simultaneously write
to the shared storage, the DiskReservation agent ensures that only one processor has
access to the storage at one time. See the VCS Bundled Agents Reference Guide for
information on configuring the DiskReservation agent.