Red Hat ENTERPRISE LINUX 5 - VIRTUAL SERVER... User manual

Red Hat Enterprise Linux 5
Configuration Example
- Fence Devices
Configuring Fence Devices in a Red Hat Cluster

Configuration Example - Fence Devices
Red Hat Enterprise Linux 5 Configuration Example - Fence
Devices
Configuring Fence Devices in a Red Hat Cluster
Edition 1
Copyright © 2009 Red Hat Inc.. This material may only be distributed subject to the terms and
conditions set forth in the Open Publication License, V1.0 or later (the latest version of the OPL is
presently available at http://www.opencontent.org/openpub/).
Red Hat and the Red Hat "Shadow Man" logo are registered trademarks of Red Hat, Inc. in the United
States and other countries.
All other trademarks referenced herein are the property of their respective owners.
1801 Varsity Drive
Raleigh, NC 27606-2072 USA
Phone: [phone]
Phone: [phone]
Fax: [phone]
PO Box 13588 Research Triangle Park, NC 27709 USA
This book describes a procedure for configuring fence devices in a Red Hat Cluster using Conga.

iii
Introduction v
1. About This Guide ............................................................................................................ v
2. Audience ........................................................................................................................ v
3. Software Versions ........................................................................................................... v
4. Related Documentation ................................................................................................... v
5. Feedback ....................................................................................................................... vi
6. Document Conventions ................................................................................................... vi
6.1. Typographic Conventions ..................................................................................... vi
6.2. Pull-quote Conventions ....................................................................................... viii
6.3. Notes and Warnings ............................................................................................ ix
1. Configuring Fence Devices in a Red Hat Cluster 1
2. Configuring an APC Switch as a Fence Device 3
2.1. APC Fence Device Prerequisite Configuration ............................................................... 3
2.2. APC Fence Device Components to Configure ................................................................ 4
2.3. APC Fence Device Configuration Procedure .................................................................. 5
2.4. Cluster Configuration File with APC Fence Device ......................................................... 8
2.5. Testing the APC Fence Device Configuration ............................................................... 10
3. Configuring IPMI Management Boards as Fencing Devices 11
3.1. IPMI Fence Device Prerequisite Configuration .............................................................. 12
3.2. IPMI Fence Device Components to Configure .............................................................. 12
3.3. IPMI Fence Device Configuration Procedure ................................................................ 14
3.4. Cluster Configuration File with IPMI Fence Device ........................................................ 16
3.5. Testing the IPMI Fence Device Configuration ............................................................... 18
4. Troubleshooting 19
5. The GFS Withdraw Function 23
A. Revision History 25
Index 27

iv

v
Introduction
1. About This Guide
This book describes procedures for configuring fence devices in a Red Hat Cluster using Conga.
2. Audience
This book is intended to be used by system administrators managing systems running the Linux
operating system. It requires familiarity with Red Hat Enterprise Linux 5 and Red Hat Cluster Suite.
3. Software Versions
Software Description
RHEL5 refers to RHEL5 and higher
GFS refers to GFS for RHEL5 and higher
Table 1. Software Versions
4. Related Documentation
For more information about using Red Hat Enterprise Linux, refer to the following resources:
•Red Hat Enterprise Linux Installation Guide — Provides information regarding installation of Red
Hat Enterprise Linux 5.
•Red Hat Enterprise Linux Deployment Guide — Provides information regarding the deployment,
configuration and administration of Red Hat Enterprise Linux 5.
For more information about Red Hat Cluster Suite for Red Hat Enterprise Linux 5, refer to the following
resources:
•Red Hat Cluster Suite Overview — Provides a high level overview of the Red Hat Cluster Suite.
•Configuring and Managing a Red Hat Cluster — Provides information about installing, configuring
and managing Red Hat Cluster components.
•LVM Administrator's Guide: Configuration and Administration — Provides a description of the
Logical Volume Manager (LVM), including information on running LVM in a clustered environment.
•Global File System: Configuration and Administration — Provides information about installing,
configuring, and maintaining Red Hat GFS (Red Hat Global File System).
•Global File System 2: Configuration and Administration — Provides information about installing,
configuring, and maintaining Red Hat GFS2 (Red Hat Global File System 2).
•Using Device-Mapper Multipath — Provides information about using the Device-Mapper Multipath
feature of Red Hat Enterprise Linux 5.
•Using GNBD with Global File System — Provides an overview on using Global Network Block
Device (GNBD) with Red Hat GFS.

Introduction
vi
•Linux Virtual Server Administration — Provides information on configuring high-performance
systems and services with the Linux Virtual Server (LVS).
•Red Hat Cluster Suite Release Notes — Provides information about the current release of Red Hat
Cluster Suite.
Red Hat Cluster Suite documentation and other Red Hat documents are available in HTML,
PDF, and RPM versions on the Red Hat Enterprise Linux Documentation CD and online at http://
www.redhat.com/docs/.
5. Feedback
If you spot a typo, or if you have thought of a way to make this manual better, we would love to
hear from you. Please submit a report in Bugzilla (http://bugzilla.redhat.com/bugzilla/) against the
component rh-cs.
Be sure to mention the manual's identifier:
Bugzilla component: Documentation-cluster
Book identifier: Cluster_Config_Example_Fence(EN)-5 (2009-06-18T15:20)
By mentioning this manual's identifier, we know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If you have
found an error, please include the section number and some of the surrounding text so we can find it
easily.
6. Document Conventions
This manual uses several conventions to highlight certain words and phrases and draw attention to
specific pieces of information.
In PDF and paper editions, this manual uses typefaces drawn from the Liberation Fonts1 set. The
Liberation Fonts set is also used in HTML editions if the set is installed on your system. If not,
alternative but equivalent typefaces are displayed. Note: Red Hat Enterprise Linux 5 and later includes
the Liberation Fonts set by default.
6.1. Typographic Conventions
Four typographic conventions are used to call attention to specific words and phrases. These
conventions, and the circumstances they apply to, are as follows.
Mono-spaced Bold
Used to highlight system input, including shell commands, file names and paths. Also used to highlight
key caps and key-combinations. For example:
To see the contents of the file my_next_bestselling_novel in your current
working directory, enter the cat my_next_bestselling_novel command at the
shell prompt and press Enter to execute the command.
1 https://fedorahosted.org/liberation-fonts/

Typographic Conventions
vii
The above includes a file name, a shell command and a key cap, all presented in Mono-spaced Bold
and all distinguishable thanks to context.
Key-combinations can be distinguished from key caps by the hyphen connecting each part of a key-
combination. For example:
Press Enter to execute the command.
Press Ctrl+Alt+F1 to switch to the first virtual terminal. Press Ctrl+Alt+F7 to
return to your X-Windows session.
The first sentence highlights the particular key cap to press. The second highlights two sets of three
key caps, each set pressed simultaneously.
If source code is discussed, class names, methods, functions, variable names and returned values
mentioned within a paragraph will be presented as above, in Mono-spaced Bold. For example:
File-related classes include filesystem for file systems, file for files, and dir for
directories. Each class has its own associated set of permissions.
Proportional Bold
This denotes words or phrases encountered on a system, including application names; dialogue
box text; labelled buttons; check-box and radio button labels; menu titles and sub-menu titles. For
example:
Choose System > Preferences > Mouse from the main menu bar to launch Mouse
Preferences. In the Buttons tab, click the Left-handed mouse check box and click
Close to switch the primary mouse button from the left to the right (making the mouse
suitable for use in the left hand).
To insert a special character into a gedit file, choose Applications > Accessories
> Character Map from the main menu bar. Next, choose Search > Find… from the
Character Map menu bar, type the name of the character in the Search field and
click Next. The character you sought will be highlighted in the Character Table.
Double-click this highlighted character to place it in the Text to copy field and then
click the Copy button. Now switch back to your document and choose Edit > Paste
from the gedit menu bar.
The above text includes application names; system-wide menu names and items; application-specific
menu names; and buttons and text found within a GUI interface, all presented in Proportional Bold and
all distinguishable by context.
Note the > shorthand used to indicate traversal through a menu and its sub-menus. This is to avoid
the difficult-to-follow 'Select Mouse from the Preferences sub-menu in the System menu of the main
menu bar' approach.
Mono-spaced Bold Italic or Proportional Bold Italic
Whether Mono-spaced Bold or Proportional Bold, the addition of Italics indicates replaceable or
variable text. Italics denotes text you do not input literally or displayed text that changes depending on
circumstance. For example:
To connect to a remote machine using ssh, type ssh [e-mail] at
a shell prompt. If the remote machine is example.com and your username on that
machine is john, type ssh [email protected].

Introduction
viii
The mount -o remount file-system command remounts the named file
system. For example, to remount the /home file system, the command is mount -o
remount /home.
To see the version of a currently installed package, use the rpm -q package
command. It will return a result as follows: package-version-release.
Note the words in bold italics above — username, domain.name, file-system, package, version and
release. Each word is a placeholder, either for text you enter when issuing a command or for text
displayed by the system.
Aside from standard usage for presenting the title of a work, italics denotes the first use of a new and
important term. For example:
When the Apache HTTP Server accepts requests, it dispatches child processes
or threads to handle them. This group of child processes or threads is known as
a server-pool. Under Apache HTTP Server 2.0, the responsibility for creating and
maintaining these server-pools has been abstracted to a group of modules called
Multi-Processing Modules (MPMs). Unlike other modules, only one module from the
MPM group can be loaded by the Apache HTTP Server.
6.2. Pull-quote Conventions
Two, commonly multi-line, data types are set off visually from the surrounding text.
Output sent to a terminal is set in Mono-spaced Roman and presented thus:
books Desktop documentation drafts mss photos stuff svn
books_tests Desktop1 downloads images notes scripts svgs
Source-code listings are also set in Mono-spaced Roman but are presented and highlighted as
follows:
package org.jboss.book.jca.ex1;
import javax.naming.InitialContext;
public class ExClient
{
public static void main(String args[])
throws Exception
{
InitialContext iniCtx = new InitialContext();
Object ref = iniCtx.lookup("EchoBean");
EchoHome home = (EchoHome) ref;
Echo echo = home.create();
System.out.println("Created Echo");
System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));

Notes and Warnings
ix
}
}
6.3. Notes and Warnings
Finally, we use three visual styles to draw attention to information that might otherwise be overlooked.
Note
A note is a tip or shortcut or alternative approach to the task at hand. Ignoring a note
should have no negative consequences, but you might miss out on a trick that makes your
life easier.
Important
Important boxes detail things that are easily missed: configuration changes that only
apply to the current session, or services that need restarting before an update will apply.
Ignoring Important boxes won't cause data loss but may cause irritation and frustration.
Warning
A Warning should not be ignored. Ignoring warnings will most likely cause data loss.

x

Chapter 1.
1
Configuring Fence Devices in a Red
Hat Cluster
This document provides two configuration examples, showing the steps needed to configure fence
devices in a Red Hat cluster using the Conga configuration tool. For general information about fencing
and fence device configuration, see Configuring and Managing a Red Hat Cluster.
This document includes the procedures for configuring the following two fence devices:
• APC switch: single APC device to fence all cluster nodes
• IPMI management board: separate IPMI management boards for each cluster node
This remainder of this document is organized as follows:
•Chapter 2, Configuring an APC Switch as a Fence Device describes the procedure for configuring
an APC switch as a fence device in a Red Hat cluster.
•Chapter 3, Configuring IPMI Management Boards as Fencing Devices describes the procedure for
configuring an IPMI management board as a fence device in a Red Hat cluster.
•Chapter 4, Troubleshooting provides some guidelines to follow when your configuration does not
behave as expected.
•Chapter 5, The GFS Withdraw Function summarizes some general concerns to consider when
configuring fence devices in a Red Hat cluster.

2

Chapter 2.
3
Configuring an APC Switch as a Fence
Device
This chapter provides the procedures for configuring an APC switch as a fence device in a Red Hat
cluster using the Conga configuration tool.
Figure 2.1, “Using an APC Switch as a Fence Device” shows the configuration this procedure yields.
In this configuration a three node cluster uses an APC switch as the fencing device. Each node in the
cluster is connected to a port in the APC switch.
Figure 2.1. Using an APC Switch as a Fence Device
2.1. APC Fence Device Prerequisite Configuration
Table 2.1, “Configuration Prerequisities” summarizes the prerequisite components that have been set
up before this procedure begins.
Component Name Comment
cluster apcclust three-node cluster
cluster node clusternode1.example.com node in cluster apcclust configured with APC switch to
administer power supply

Chapter 2. Configuring an APC Switch as a Fence Device
4
Component Name Comment
cluster node clusternode2.example.com node in cluster apcclust configured with APC switch to
administer power supply
cluster node clusternode3.example.com node in cluster apcclust configured with APC switch to
administer power supply
IP address 10.15.86.96 IP address for the APC switch that controls
the power for for clusternode1.example.com,
clusternode2.example.com, and
clusternode3.example.com
login apclogin login value for the APC switch that controls
the power for for clusternode1.example.com,
clusternode2.example.com, and
clusternode3.example.com
password apcpword password for the APC switch that controls
the power for for clusternode1.example.com,
clusternode2.example.com, and
clusternode3.example.com
port 1 port number on APC switch that
clusternode1.example.com connects to
port 2 port number on APC switch that
clusternode2.example.com connects to
port 3 port number on APC switch that
clusternode3.example.com connects to
Table 2.1. Configuration Prerequisities
2.2. APC Fence Device Components to Configure
This procedure configures an APC switch as a fence device that will be used for each
node in cluster apcclust. Then the procedure configures that switch as the fencing
device for clusternode1.example.com, clusternode2.example.com, and
clusternode1.example.com.
Table 2.2, “Fence Device Components to Configure for APC Fence Device” summarizes
the components of the APC fence device that this procedure configures for cluster node
clusternode1.example.com.
Fence
Device
Component
Value Description
Fencing
Type
APC Power Switch type of fencing device to configure
Name apcfence name of the APC fencing device
IP address 10.15.86.96 IP address of the APC switch to configure as a fence
device for node1.example.com, node2.example.com,
node3.example.com
login apclogin login value for the APC switch that controls
the power for for clusternode1.example.com,

APC Fence Device Configuration Procedure
5
Fence
Device
Component
Value Description
clusternode2.example.com, and
clusternode3.example.com
password apcpword password for the APC switch that controls
the power for for clusternode1.example.com,
clusternode2.example.com, and
clusternode3.example.com
Table 2.2. Fence Device Components to Configure for APC Fence Device
Table 2.3, “Fence Agent Components to Specify for Each Node in apcclust ” summarizes the
components of the APC fence device that you must specify for the cluster nodes in apcclust.
Fence Agent
Component
Value Description
fence device apcfence name of the APC fence device you defined as a shared device
port 1 port number on the APC switch for node1.example.com
port 2 port number on the APC switch for node2.example.com
port 3 port number on the APC switch for node3.example.com
Table 2.3. Fence Agent Components to Specify for Each Node in apcclust
The remainder of the fence device components that you configure for each node appear automatically
when you specify that you will be configured the apcfence fence device that you previously defined
as a shared fence device.
2.3. APC Fence Device Configuration Procedure
This section provides the procedure for adding an APC fence device to each node of cluster
apcclust. This example uses the same APC switch for each cluster node, which will first be
configured as a shared fence device. After configuring the APC switch as a shared fence device, the
device will be added as a fence device for each node in the cluster.
To configure an APC switch as a shared fence device using Conga, perform the following procedure:
1. As an administrator of luci Select the cluster tab. This displays the Choose a cluster to
administer screen.
2. From the Choose a cluster to administer screen, you should see the previously configured
cluster apcclust displayed, along with the nodes that make up the cluster. Click on apcclust to
select the cluster.
3. At the detailed menu for the cluster apcclust (below the clusters menu on the left side of the
screen), click Shared Fence Devices. Clicking Shared Fence Devices causes the display of any
shared fence devices previously configured for a cluster and causes the display of menu items for
fence device configuration: Add a Fence Device and Configure a Fence Device.
4. Click Add a Fence Device. Clicking Add a Fence Device causes the Add a Sharable Fence
Device page to be displayed.

Chapter 2. Configuring an APC Switch as a Fence Device
6
5. At the Add a Sharable Fence Device page, click the drop-down box under Fencing Type and
select APC Power Switch. This causes Conga to display the components of an APC Power
Switch fencing type, as shown in Figure 2.2, “Adding a Sharable Fence Device”.
Figure 2.2. Adding a Sharable Fence Device
6. For Name, enter apcfence.
7. For IP Address, enter 10.15.86.96.
8. For Login, enter apclogin.
9. For Password, enter apcpword.
10. For Password Script, leave blank.
11. Click Add this shared fence device.
Clicking Add this shared fence device causes a progress page to be displayed temporarily. After
the fence device has been added, the detailed cluster properties menu is updated with the fence
device under Configure a Fence Device.

APC Fence Device Configuration Procedure
7
After configuring the APC switch as a shared fence device, use the following procedure to configure
the APC switch as the fence device for node clusternode1.example.com
1. At the detailed menu for the cluster apcclust (below the clusters menu), click Nodes. Clicking
Nodes causes the display of the status of each node in apcclust.
2. At the bottom of the display for node clusternode1.example.com, click Manage Fencing for
this Node. This displays the configuration screen for node clusternode1.example.com.
3. At the Main Fencing Method display, click Add a fence device to this level. This causes a
dropdown menu to display.
4. From the dropdown menu, the apcfence fence device you have already created should display
as one of the menu options under Use an Existing Fence Device. Select apcfence (APC
Power Device). This causes a fence device configuration menu to display with the Name, IP
Address, Login, Password, and Password Script values already configured, as defined when
you configured apcfence as a shared fence device. This is shown in Figure 2.3, “Adding an
Existing Fence Device to a Node”.
Figure 2.3. Adding an Existing Fence Device to a Node
5. For Port, enter 1. Do not enter any value for Switch.

Chapter 2. Configuring an APC Switch as a Fence Device
8
6. Click Update main fence properties. This causes a confirmation screen to be displayed.
7. On the confirmation screen, Click OK. A progress page is displayed after which the display returns
to the status page for clusternode1.example.com in cluster apcclust.
After configuring apcfence as the fencing device for clusternode1.example.com, use the
same procedure to configure apcfence as the fencing device for clusternode2.example.com,
specifying Port 2 for clusternode2.example.com, as in the following procedure:
1. On the status page for clusternode1.example.com in cluster apcclust, the other nodes
in apcclust are displayed below the Configure menu item below the Nodes menu item on
the left side of the screen. Click clusternode2.example.com to display the status screen for
clusternode2.example.com.
2. At the Main Fencing Method display, click Add a fence device to this level. This causes a
dropdown manu to display.
3. As for clusternode1.example.com, the apcfence fence device should display as one of the
menu options on the dropdown menu, under Use an Existing Fence Device. Select apcfence
(APC Power Device). This causes a fence device configuration menu to display with the Name,
IP Address, Login, Password, Password Script values already configured, as defined when you
configured apcfence as a shared fence device.
4. For Port, enter 2. Do not enter any value for Switch.
5. Click Update main fence properties.
Similarly, configure apcfence as the main fencing method for clusternode3.example.com,
specifying 3 as the Port number.
2.4. Cluster Configuration File with APC Fence Device
Configuring a cluster with Conga modifies the cluster configuration file. This section shows the cluster
configuration file before and after the procedures documented in Section 2.3, “APC Fence Device
Configuration Procedure” were performed.
Before the cluster resources and service were configured, the cluster.conf file appeared as
follows.
<?xml version="1.0"?>
<cluster alias="apcclust" config_version="12" name="apcclust">
<fence_daemon clean_start="0" post_fail_delay="0"
post_join_delay="3"/>
<clusternodes>
<clusternode name="clusternode1.example.com" nodeid="1"
votes="1">
<fence/>
</clusternode>
<clusternode name="clusternode2.example.com" nodeid="2"
votes="1">
<fence/>

Cluster Configuration File with APC Fence Device
9
</clusternode>
<clusternode name="clusternode3.example.com" nodeid="3"
votes="1">
<fence>
<method name="1"/>
</fence>
</clusternode>
</clusternodes>
<cman/>
<fencedevices/>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
After the cluster resources and service were configured, the cluster.conf file appears as follows.
<?xml version="1.0"?>
<cluster alias="apcclust" config_version="19" name="apcclust">
<fence_daemon clean_start="0" post_fail_delay="0"
post_join_delay="3"/>
<clusternodes>
<clusternode name="clusternode1.example.com" nodeid="1"
votes="1">
<fence>
<method name="1">
<device name="apcfence" port="1"/>
</method>
</fence>
</clusternode>
<clusternode name="clusternode2.example.com" nodeid="2"
votes="1">
<fence>
<method name="1">
<device name="apcfence" port="2"/>
</method>
</fence>
</clusternode>
<clusternode name="clusternode3.example.com" nodeid="3"
votes="1">
<fence>
<method name="1">
<device name="apcfence" port="3"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman/>

Chapter 2. Configuring an APC Switch as a Fence Device
10
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="10.15.86.96"
login="apclogin" name="apcfence" passwd="apcpword"/>
</fencedevices>
<rm>
<failoverdomains/>
<resources/>
</rm>
</cluster>
2.5. Testing the APC Fence Device Configuration
To check whether the configuration you have defined works as expected, you can use the
fence_node to fence a node manually. The fence_node program reads the fencing settings from
the cluster.conf file for the given node and then runs the configured fencing agent against the
node.
To test whether the APC switch has been successfully configured as a fence device for the three
nodes in cluster apcclust, execute the following commands and check whether the nodes have
been fenced.
# /sbin/fence_node clusternode1.example.com
# /sbin/fence_node clusternode2.example.com
# /sbin/fence_node clusternode3.example.com