Symantec Veritas Solaris User manual

Veritas™ Cluster Server
Application Note: Sun Enterprise
10000 Dynamic Reconfiguration
Solaris
5.0
N18540F

Veritas Cluster Server
Application Note: Sun Enterprise 10000 Dynamic
Reconfiguration
Copyright © 2006 Symantec Corporation. All rights reserved.
Symantec, the Symantec logo, andVeritas are trademarks or registered trademarks of
Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be
trademarks of their respective owners.
The product described in this document is distributed under licenses restricting its use,
copying, distribution, and decompilation/reverse engineering. No part of this document
may be reproduced in any form by any means without prior written authorization of
Symantec Corporation and its licensors,if any.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED
CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED
WARRANTY OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE OR
NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH
DISCLAIMERS ARE HELD TO BE LEGALLY INVALID, SYMANTEC CORPORATION SHALL
NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN CONNECTION
WITHTHE FURNISHING PERFORMANCE, OR USE OF THIS DOCUMENTATION. THE
INFORMATION CONTAINED IN THIS DOCUMENTATION IS SUBJECT TO CHANGE
WITHOUT NOTICE.
The Licensed Software and Documentation are deemed to be “commercial computer
software” and “commercial computer software documentation” as defined in FAR
Sections12.212 and DFARS Section 227.7202.
Symantec Corporation
20330 Stevens Creek Blvd.
Cupertino, CA 95014
www.symantec.com

Third-party legal notices
Third-party software may berecommended, distributed, embedded,or bundled
with this Symantec product. Such third-party software is licensed separately by
its copyright holder. All third-party copyrights associated with this product are
listed in the accompanyingrelease notes.
Solaris is a trademark of Sun Microsystems, Inc.
Technical support
For technical assistance, visit http://support.veritas.com and select phone or
email support. Use the Knowledge Base search feature to access resources such
as TechNotes, product alerts, software downloads, hardware compatibility lists,
and our customer email notification service.

VCS Application Note: Sun
Enterprise 10000 Dynamic
Reconfiguration
■Introduction
■Supported hardware
■Supported software
■Dynamic reconfiguration in VCS environment
■Using VM without DMP enabled

6 VCS Application Note: Sun Enterprise 10000 Dynamic Reconfiguration
Introduction
Introduction
This application note describes how to perform Dynamic Reconfiguration (DR)
operations on VCS clustered system domains of the SunTM EnterpriseTM 10000
server.
The Solaris DR utility enables you to reconfigure the resources of system boards
so that the boards can be replaced without system downtime. Before you can
physically remove a board, you must “detach” it, or reconfigure it suchthat its
resources can be disabled and removed from the domain configuration.
Likewise, after you have physically replaced a board in a domain, you must
“attach” it, orreconfigure it into the domain.
Note: Currently, VCS does not support using DR in clusters where I/O
controllers and storage use Sun’s Alternate Pathing (AP).
In a VCS cluster of domains, it’s possible that the DR operations in a domain
might cause VCS to detect that resources are not available and to initiate
failover to the other domain. Therefore, it is advisable to freeze the service
groups running in the domain before running DR.
Do not use the following procedures to DR a system board containing a VCS
private heartbeat link. If you need to do so, you must stop VCS before
proceeding.
Supported hardware
■Sun Enterprise 10000 (E10K)
Supported software
■Solaris 8, 9, and 10
■VERITAS Cluster Server, version 2.0, 3.5, 4.0, and 4.1
■VERITAS Volume Manager (VxVM), as supported bythe VCS version
■VERITAS File System, as supported bythe VCS version

VCS Application Note: Sun Enterprise 10000 Dynamic Reconfiguration 7
Dynamic reconfiguration in VCS environment
Dynamic reconfiguration in VCS environment
The system board in a domain may contain I/O controllers, CPUs, or memory.
Typically, within a domain, system boards have their functions duplicated on
other system boards. For example, a board with CPU or memory can be removed
dynamically because another board has the equivalent functions.
Boards with I/O controllers can be dynamically reconfigured as longas you use
VxVM with the Dynamic Multipathing (DMP) feature to manage the shared
storage.
Setting environment variable to enable DR
Before performing DR operations on a domain, you must first set the
appropriate environment variable.
Enable the kernel cage variable for DR
Using the Solaris 8 operating environment, you must set the system(4)
variable, kernel_cage_enable, to 1 (enabled). By default, this variable is set
to zero (kernel cage disabled), preventing DR Detach operations.
Edit the file /etc/system so that kernel_cage_enable equals 1.
.
set kernel_cage_enable=1
.
Reboot the domain. To verify the kernel cage is enabled, check the file
/var/adm/messages. Look for the message:
NOTICE: DR Kernel Cage is ENABLED
Detaching and attaching I/O system boards
In the configuration shown below, VCS runs on Domains A and B with service
groups online onDomain A. Shared storage consists of a VxVM disk group with

8 VCS Application Note: Sun Enterprise 10000 Dynamic Reconfiguration
Dynamic reconfiguration in VCS environment
Dynamic Multipathing (DMP) enabled. Dynamic Reconfiguration of I/O boards
depends on DMP being configured for the storage.
In the example, the system board SB3, which has a disk controller, is to be
removed, repaired, and replaced. The administrator disables the controller, and
the disk controller on SB1 automatically takes over because of the DMP
functionality. Using DR commands, the administrator can detach, or remove the
board from the Domain A’s configuration. When this is complete, the board can
be physically removed.
Replacing the board—a controller board in this case—involves physically
installing it and reconnecting it to the shared storage. Reconfiguring the board
requires using DR commands to “attach” it to the domain, after which the
controller can be re-enabled.
Detaching I/O system boards with DMP enabled
Make sure the kernel_cage_enable variable is set.
1 Freeze the VCS service groups running on the domain where you intend to
perform DR operations. Freezing the service groups prevents them from
being taken offline or failed over. Repeat the following command for each
service group:
# hagrp -freeze ser_grp_name
2 Connect tothe SSP server and log in tothe domain whose system board
requires Dynamic Reconfiguration (DR).
ssp:D1% echo $SUNW_HOSTNAME
3 Enter the dr(1M) shell:
ssp:D1% dr

VCS Application Note: Sun Enterprise 10000 Dynamic Reconfiguration 9
Dynamic reconfiguration in VCS environment
4 To verify the boardis an I/O board, enter:
dr> drshow sb# IO
If the display lists the disks connected to the controller, the system board is
an I/O board.
5 If the system board is an I/O board, open another windowand log in as root
to the domain you are currently reconfiguring.
6 Disable the controller on the I/O system board:
# vxdmpadm disable ctlr=ctlr#
7 In the window where you are running DR, start detaching the I/O board by
entering:
dr> drain sb#
8 Monitor the progress of the drain operation byentering:
dr> drshow sb# drain
9 When you see the message:
Percent Complete= 100% (0 KBytes remaining)
complete the detach operation:
dr> complete_detach sb#
10 To verify that the board is no longer configured, type the following
command:
dr> drshow sb#
The detached board should not appear in the detailed listing.
11 Exit the dr shell:
dr> exit
12 If the board is not to be immediately replaced, unfreeze any frozen service
groups:
# hagrp -unfreeze ser_grp_name
Repeat for each service group.
Attaching I/O systemboards withDMP enabled
1 Freeze the VCS service groups running on the domain where you intend to
attach a system board. Repeat the following command for each service
group:
# hagrp -freeze ser_grp_name
2 After physically replacing a previously removed I/O board, makesureit is
connected to the shared storage.
3 From the SSP server, enter the dr(1M) shell:
ssp:D1% dr

10 VCS Application Note: Sun Enterprise 10000 Dynamic Reconfiguration
Dynamic reconfiguration in VCS environment
4 Follow the Sun procedure to attach the system board, described here briefly:
dr> init_attach sb#
Complete the attachoperation:
dr> complete_attach sb#
5 Verify that the DR attach operation has succeeded. Type:
dr> drshow #sb
The new system board should show in the listof configured boards.
6 Exit the dr shell.
dr> exit
7 Log in as root tothe domain where you are adding the system board. Enable
the controller byentering:
# vxdmpadm enable ctlr=ctlr#
8 When you have successfully attached and enabled the system I/O board,
unfreeze any frozen service groups:
# hagrp -unfreeze ser_grp_name
Repeat for each service group.
9 Verify that VCS is still up and running.
Detaching CPU/memoryboards
Note: Use the following procedure if no I/O devices on the system board are
used.
Make sure the kernel_cage_enable variable is set.
1 Freeze the VCS service groups running on the domain where you intend to
detach a CPU/Memory board. Freezing the service groups prevents them
from being taken offline or failed over. Repeat the following command for
eachservice group:
# hagrp -freeze ser_grp_name
2 Connect tothe SSP server and log in tothe domain whose system board
requires Dynamic Reconfiguration (DR).
ssp:D1% echo $SUNW_HOSTNAME
3 Enter the dr(1M) shell:
ssp:D1% dr
4 In the window where you are running DR, start detaching the I/O board by
entering:
dr> drain sb#
5 Monitor the progress of the drain operation byentering:
dr> drshow sb# drain

VCS Application Note: Sun Enterprise 10000 Dynamic Reconfiguration 11
Dynamic reconfiguration in VCS environment
6 When you see the message
Percent Complete= 100% (0 KBytes remaining)
complete the detach operation:
dr> complete_detach sb#
7 To verify that the board is no longer configured, type the following
command:
dr> drshow sb#
The detached board should not appear in the detailed listing.
8 Exit the dr shell:
dr > exit
9 If the board is not to be immediately replaced, unfreeze any frozen service
groups:
# hagrp -unfreeze ser_grp_name
10 Repeat for each service group.
Attaching CPU/Memoryboards
Note: Use the following procedure ifnone of the I/O devices on the system board
are used.
1 Freeze the VCS service groups running on the domain where you intend to
attach a system board. Repeat the following command for each service
group:
# hagrp -freeze ser_grp_name
2 Physically replace the CPU/Memory board.
3 From the SSP server, enter the dr(1M) shell:
ssp:D1% dr
4 Follow the Sun procedure to attach the system board, described here briefly:
dr> init_attach sb#
Complete the attachoperation:
dr> complete_attach sb#
5 Verify that the DR attach operation has succeeded. Type:
dr> drshow #sb
The new system board should show in the list of configured boards.
6 Exit the dr shell.
dr> exit
7 When you have successfullyattached the CPU/Memory board, unfreeze any
frozen service groups:

12 VCS Application Note: Sun Enterprise 10000 Dynamic Reconfiguration
Using VM without DMP enabled
# hagrp -unfreeze ser_grp_name
Repeat for each service group.
8 Verify that VCS is still up and running.
Using VM without DMP enabled
If you have the Volume Manager DMP feature disabled for some or all of the
disks inthe shared storage, and you must perform DR operations within the
cluster, we recommend using the VCS DiskReservation agent to guard against
data corruption. In the event of a “split-brain” condition, that is, when two
processors in a cluster can simultaneously write to the shared storage, the
DiskReservation agent ensures that only one processor has access to the storage
at one time. See the VCS Bundled Agents Reference Guide for information on
configuring the DiskReservation agent.