Red Hat ENTERPRISE LINUX 5 - VIRTUAL SERVER... Quick guide

Configuring and Managing a Red
Hat Cluster
Red Hat Cluster for Red Hat Enterprise
Linux 5

Configuring and Managing a Red Hat Cluster: Red Hat Cluster
for Red Hat Enterprise Linux 5
Copyright ©2007 Red Hat, Inc.
Configuring and Managing a Red Hat Cluster describes the configuration and management of
Red Hat cluster systems for Red Hat Enterprise Linux 5. It does not include information about
Red Hat Linux Virtual Servers (LVS). Information about installing and configuring LVS is in a
separate document.
1801 Varsity Drive
Raleigh, NC 27606-2072
USA
Phone: [phone]
Phone: [phone]
Fax: [phone]
PO Box 13588
Research Triangle Park, NC 27709
USA
Documentation-Deployment
Copyright ©2007 by Red Hat, Inc. This material may be distributed only subject to the terms and conditions set forth in
the Open Publication License, V1.0 or later (the latest version is presently available at ht-
tp://www.opencontent.org/openpub/).
Distribution of substantively modified versions of this document is prohibited without the explicit permission of the copy-
right holder.
Distribution of the work or derivative of the work in any standard (paper) book form for commercial purposes is prohib-
ited unless prior permission is obtained from the copyright holder.
Red Hat and the Red Hat "Shadow Man" logo are registered trademarks of Red Hat, Inc. in the United States and other
countries.
All other trademarks referenced herein are the property of their respective owners.
The GPG fingerprint of the [email protected] key is:
CA 20 86 86 2B D6 9D FC 65 F6 EC C4 21 91 80 CD DB 42 A6 0E

Table of Contents
Introduction ...............................................................................................................vi
1. Document Conventions .................................................................................vii
2. Feedback .................................................................................................... viii
1. Red Hat Cluster Configuration and Management Overview ....................................... 1
1. Configuration Basics ...................................................................................... 1
1.1. Setting Up Hardware ........................................................................... 1
1.2. Installing Red Hat Cluster software ....................................................... 2
1.3. Configuring Red Hat Cluster Software .................................................. 2
2. Conga ........................................................................................................... 4
3. system-config-cluster Cluster Administration GUI ............................................. 7
3.1. Cluster Configuration Tool ................................................................... 7
3.2. Cluster Status Tool .............................................................................. 9
4. Command Line Administration Tools ..............................................................10
5. Configuration Considerations .........................................................................11
2. Configuring Red Hat Cluster With Conga ................................................................13
1. Configuration Tasks ......................................................................................13
2. Starting luci and ricci .....................................................................................13
3. Creating A Cluster .........................................................................................15
4. Global Cluster Properties ...............................................................................15
5. Configuring Fence Devices ............................................................................17
5.1. Creating a Shared Fence Device .........................................................18
5.2. Modifying or Deleting a Fence Device ..................................................19
6. Configuring Cluster Members .........................................................................20
6.1. Initially Configuring Members ..............................................................20
6.2. Adding a Member to a Running Cluster ...............................................21
6.3. Deleting a Member from a Cluster .......................................................22
7. Configuring a Failover Domain .......................................................................23
7.1. Adding a Failover Domain ...................................................................24
7.2. Modifying a Failover Domain ...............................................................25
8. Adding Cluster Resources .............................................................................26
9. Adding a Cluster Service to the Cluster ..........................................................29
10. Configuring Cluster Storage .........................................................................30
3. Managing Red Hat Cluster With Conga ..................................................................32
1. Starting, Stopping, and Deleting Clusters ........................................................32
2. Managing Cluster Nodes ...............................................................................33
3. Managing High-Availability Services ...............................................................34
4. Diagnosing and Correcting Problems in a Cluster ............................................35
4. Configuring Red Hat Cluster With system-config-cluster ..........................................36
1. Configuration Tasks ......................................................................................36
2. Starting the Cluster Configuration Tool ...........................................................37
3. Naming The Cluster ......................................................................................39
4. Configuring Fence Devices ............................................................................40
5. Adding and Deleting Members .......................................................................41
5.1. Adding a Member to a Cluster .............................................................41
5.2. Adding a Member to a Running Cluster ...............................................43
iv

5.3. Deleting a Member from a Cluster .......................................................45
6. Configuring a Failover Domain .......................................................................46
6.1. Adding a Failover Domain ...................................................................47
6.2. Removing a Failover Domain ..............................................................50
6.3. Removing a Member from a Failover Domain .......................................50
7. Adding Cluster Resources .............................................................................51
8. Adding a Cluster Service to the Cluster ..........................................................53
9. Propagating The Configuration File: New Cluster ............................................56
10. Starting the Cluster Software .......................................................................57
5. Managing Red Hat Cluster With system-config-cluster .............................................58
1. Starting and Stopping the Cluster Software .....................................................58
2. Managing High-Availability Services ...............................................................58
3. Modifying the Cluster Configuration ................................................................60
4. Backing Up and Restoring the Cluster Database .............................................61
5. Disabling the Cluster Software .......................................................................63
6. Diagnosing and Correcting Problems in a Cluster ............................................63
A. Example of Setting Up Apache HTTP Server ..........................................................64
1. Apache HTTP Server Setup Overview ............................................................64
2. Configuring Shared Storage ...........................................................................65
3. Installing and Configuring the Apache HTTP Server ........................................65
B. Fence Device Parameters .....................................................................................68
C. Upgrading A Red Hat Cluster from RHEL 4 to RHEL 5 ............................................74
Index .......................................................................................................................77
Configuring and Managing a Red Hat Cluster
v

Introduction
This document provides information about installing, configuring and managing Red Hat Cluster
components. Red Hat Cluster components are part of Red Hat Cluster Suite and allow you to
connect a group of computers (called nodes or members) to work together as a cluster. This
document does not include information about installing, configuring, and managing Linux Virtual
Server (LVS) software. Information about that is in a separate document.
The audience of this document should have advanced working knowledge of Red Hat Enter-
prise Linux and understand the concepts of clusters, storage, and server computing.
This document is organized as follows:
• Chapter 1, Red Hat Cluster Configuration and Management Overview
• Chapter 2, Configuring Red Hat Cluster With Conga
• Chapter 3, Managing Red Hat Cluster With Conga
• Chapter 4, Configuring Red Hat Cluster With system-config-cluster
• Chapter 5, Managing Red Hat Cluster With system-config-cluster
• Appendix A, Example of Setting Up Apache HTTP Server
• Appendix B, Fence Device Parameters
• Appendix C, Upgrading A Red Hat Cluster from RHEL 4 to RHEL 5
For more information about Red Hat Enterprise Linux 5, refer to the following resources:
•Red Hat Enterprise Linux Installation Guide — Provides information regarding installation of
Red Hat Enterprise Linux 5.
•Red Hat Enterprise Linux Deployment Guide — Provides information regarding the deploy-
ment, configuration and administration of Red Hat Enterprise Linux 5.
For more information about Red Hat Cluster Suite for Red Hat Enterprise Linux 5, refer to the
following resources:
•Red Hat Cluster Suite Overview — Provides a high level overview of the Red Hat Cluster
Suite.
•LVM Administrator's Guide: Configuration and Administration — Provides a description of
the Logical Volume Manager (LVM), including information on running LVM in a clustered en-
vironment.
•Global File System: Configuration and Administration — Provides information about in-
stalling, configuring, and maintaining Red Hat GFS (Red Hat Global File System).
•Using GNBD with Global File System — Provides an overview on using Global Network
Block Device (GNBD) with Red Hat GFS.
vi

•Linux Virtual Server Administration — Provides information on configuring high-performance
systems and services with the Linux Virtual Server (LVS).
•Red Hat Cluster Suite Release Notes — Provides information about the current release of
Red Hat Cluster Suite.
Red Hat Cluster Suite documentation and other Red Hat documents are available in HTML,
PDF, and RPM versions on the Red Hat Enterprise Linux Documentation CD and online at ht-
tp://www.redhat.com/docs/.
1. Document Conventions
Certain words in this manual are represented in different fonts, styles, and weights. This high-
lighting indicates that the word is part of a specific category. The categories include the follow-
ing:
Courier font
Courier font represents commands,file names and paths, and prompts .
When shown as below, it indicates computer output:
Desktop about.html logs paulwesterberg.png
Mail backupfiles mail reports
bold Courier font
Bold Courier font represents text that you are to type, such as: service jonas start
If you have to run a command as root, the root prompt (#) precedes the command:
#gconftool-2
italic Courier font
Italic Courier font represents a variable, such as an installation directory: install_dir/bin/
bold font
Bold font represents application programs and text found on a graphical interface.
When shown like this: OK , it indicates a button on a graphical application interface.
Additionally, the manual uses different strategies to draw your attention to pieces of information.
In order of how critical the information is to you, these items are marked as follows:
Note
A note is typically information that you need to understand the behavior of the sys-
tem.
1. Document Conventions
vii

Tip
A tip is typically an alternative way of performing a task.
Important
Important information is necessary, but possibly unexpected, such as a configura-
tion change that will not persist after a reboot.
Caution
A caution indicates an act that would violate your support agreement, such as re-
compiling the kernel.
Warning
A warning indicates potential data loss, as may happen when tuning hardware for
maximum performance.
2. Feedback
If you spot a typo, or if you have thought of a way to make this manual better, we would love to
hear from you. Please submit a report in Bugzilla (http://bugzilla.redhat.com/bugzilla/) against
the component rh-cs.
Be sure to mention the manual's identifier:
rh-cs(EN)-5 (2007-01-23T09:05)
By mentioning this manual's identifier, we know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If
you have found an error, please include the section number and some of the surrounding text
so we can find it easily.
2. Feedback
viii

Chapter 1. Red Hat Cluster
Configuration and Management
Overview
Red Hat Cluster allows you to connect a group of computers (called nodes or members) to work
together as a cluster. You can use Red Hat Cluster to suit your clustering needs (for example,
setting up a cluster for sharing files on a GFS file system or setting up service failover).
1. Configuration Basics
To set up a cluster, you must connect the nodes to certain cluster hardware and configure the
nodes into the cluster environment. This chapter provides an overview of cluster configuration
and management, and tools available for configuring and managing a Red Hat Cluster.
Configuring and managing a Red Hat Cluster consists of the following basic steps:
1. Setting up hardware. Refer to Section 1.1, “Setting Up Hardware”.
2. Installing Red Hat Cluster software. Refer to Section 1.2, “Installing Red Hat Cluster soft-
ware”.
3. Configuring Red Hat Cluster Software. Refer to Section 1.3, “Configuring Red Hat Cluster
Software”.
1.1. Setting Up Hardware
Setting up hardware consists of connecting cluster nodes to other hardware required to run a
Red Hat Cluster. The amount and type of hardware varies according to the purpose and avail-
ability requirements of the cluster. Typically, an enterprise-level cluster requires the following
type of hardware (refer to Figure 1.1, “Red Hat Cluster Hardware Overview”). For considerations
about hardware and other cluster configuration concerns, refer to Section 5, “Configuration Con-
siderations” or check with an authorized Red Hat representative.
• Cluster nodes — Computers that are capable of running Red Hat Enterprise Linux 5 soft-
ware, with at least 1GB of RAM.
• Ethernet switch or hub for public network — This is required for client access to the cluster.
• Ethernet switch or hub for private network — This is required for communication among the
cluster nodes and other cluster hardware such as network power switches and Fibre Chan-
nel switches.
• Network power switch — A network power switch is recommended to perform fencing in an
enterprise-level cluster.
• Fibre Channel switch — A Fibre Channel switch provides access to Fibre Channel storage.
1

Other options are available for storage according to the type of storage interface; for ex-
ample, iSCSI or GNBD. A Fibre Channel switch can be configured to perform fencing.
• Storage — Some type of storage is required for a cluster. The type required depends on the
purpose of the cluster.
Figure 1.1. Red Hat Cluster Hardware Overview
1.2. Installing Red Hat Cluster software
To install Red Hat Cluster software, you must have entitlements for the software. If you are us-
ing the Conga configuration GUI, you can let it install the cluster software. If you are using other
tools to configure the cluster, secure and install the software as you would with Red Hat Enter-
prise Linux software.
1.3. Configuring Red Hat Cluster Software
Configuring Red Hat Cluster software consists of using configuration tools to specify the rela-
tionship among the cluster components. Figure 1.2, “Cluster Configuration Structure” shows an
example of the hierarchical relationship among cluster nodes, high-availability services, and re-
sources. The cluster nodes are connected to one or more fencing devices. Nodes can be
1.2. Installing Red Hat Cluster software
2

grouped into a failover domain for a cluster service. The services comprise resources such as
NFS exports, IP addresses, and shared GFS partitions.
Figure 1.2. Cluster Configuration Structure
The following cluster configuration tools are available with Red Hat Cluster:
•Conga — This is a comprehensive user interface for installing, configuring, and managing
Red Hat clusters, computers, and storage attached to clusters and computers.
•system-config-cluster — This is a user interface for configuring and managing a Red Hat
cluster.
• Command line tools — This is a set of command line tools for configuring and managing a
Red Hat cluster.
A brief overview of each configuration tool is provided in the following sections:
• Section 2, “Conga”
• Section 3, “system-config-cluster Cluster Administration GUI”
• Section 4, “Command Line Administration Tools”
1.3. Configuring Red Hat Cluster Software
3

In addition, information about using Conga and system-config-cluster is provided in sub-
sequent chapters of this document. Information about the command line tools is available in the
man pages for the tools.
2. Conga
Conga is an integrated set of software components that provides centralized configuration and
management of Red Hat clusters and storage. Conga provides the following major features:
• One Web interface for managing cluster and storage
• Automated Deployment of Cluster Data and Supporting Packages
• Easy Integration with Existing Clusters
• No Need to Re-Authenticate
• Integration of Cluster Status and Logs
• Fine-Grained Control over User Permissions
The primary components in Conga are luci and ricci, which are separately installable. luci is a
server that runs on one computer and communicates with multiple clusters and computers via
ricci.ricci is an agent that runs on each computer (either a cluster member or a standalone
computer) managed by Conga.
luci is accessible through a Web browser and provides three major functions that are access-
ible through the following tabs:
•homebase — Provides tools for adding and deleting computers, adding and deleting users,
and configuring user privileges. Only a system administrator is allowed to access this tab.
•cluster — Provides tools for creating and configuring clusters. Each instance of luci lists
clusters that have been set up with that luci. A system administrator can administer all
clusters listed on this tab. Other users can administer only clusters that the user has permis-
sion to manage (granted by an administrator).
•storage — Provides tools for remote administration of storage. With the tools on this tab,
you can manage storage on computers whether they belong to a cluster or not.
To administer a cluster or storage, an administrator adds (or registers) a cluster or a computer
toaluci server. When a cluster or a computer is registered with luci, the FQDN hostname or IP
address of each computer is stored in a luci database.
You can populate the database of one luci instance from another luciinstance. That capability
provides a means of replicating a luci server instance and provides an efficient upgrade and
testing path. When you install an instance of luci, its database is empty. However, you can im-
port part or all of a luci database from an existing luci server when deploying a new luci server.
Each luci instance has one user at initial installation — admin. Only the admin user may add
systems to a luci server. Also, the admin user can create additional user accounts and determ-
2. Conga
4

ine which users are allowed to access clusters and computers registered in the luci database. It
is possible to import users as a batch operation in a new luci server, just as it is possible to im-
port clusters and computers.
When a computer is added to a luci server to be administered, authentication is done once. No
authentication is necessary from then on (unless the certificate used is revoked by a CA). After
that, you can remotely configure and manage clusters and storage through the luci user inter-
face. luci and ricci communicate with each other via XML.
The following figures show sample displays of the three major luci tabs: homebase,cluster,
and storage.
For more information about Conga, refer to Chapter 2, Configuring Red Hat Cluster With
Conga, Chapter 3, Managing Red Hat Cluster With Conga, and the online help available with
the luci server.
Figure 1.3. luci homebase Tab
2. Conga
5

Figure 1.4. luci cluster Tab
2. Conga
6

Figure 1.5. luci storage Tab
3. system-config-cluster Cluster Administration GUI
This section provides an overview of the cluster administration graphical user interface (GUI)
available with Red Hat Cluster Suite — system-config-cluster. The GUI is for use with the
cluster infrastructure and the high-availability service management components. The GUI con-
sists of two major functions: the Cluster Configuration Tool and the Cluster Status Tool. The
Cluster Configuration Tool provides the capability to create, edit, and propagate the cluster
configuration file (/etc/cluster/cluster.conf). The Cluster Status Tool provides the capability
to manage high-availability services. The following sections summarize those functions.
3.1. Cluster Configuration Tool
You can access the Cluster Configuration Tool (Figure 1.6, “Cluster Configuration Tool”)
through the Cluster Configuration tab in the Cluster Administration GUI.
3. system-config-cluster Cluster Administration GUI
7

Figure 1.6. Cluster Configuration Tool
The Cluster Configuration Tool represents cluster configuration components in the configura-
tion file (/etc/cluster/cluster.conf) with a hierarchical graphical display in the left panel. A tri-
angle icon to the left of a component name indicates that the component has one or more sub-
ordinate components assigned to it. Clicking the triangle icon expands and collapses the portion
of the tree below a component. The components displayed in the GUI are summarized as fol-
lows:
•Cluster Nodes — Displays cluster nodes. Nodes are represented by name as subordinate
elements under Cluster Nodes. Using configuration buttons at the bottom of the right frame
(below Properties), you can add nodes, delete nodes, edit node properties, and configure
fencing methods for each node.
•Fence Devices — Displays fence devices. Fence devices are represented as subordinate
3.1. Cluster Configuration Tool
8

elements under Fence Devices. Using configuration buttons at the bottom of the right frame
(below Properties), you can add fence devices, delete fence devices, and edit fence-device
properties. Fence devices must be defined before you can configure fencing (with the Man-
age Fencing For This Node button) for each node.
•Managed Resources — Displays failover domains, resources, and services.
•Failover Domains — For configuring one or more subsets of cluster nodes used to run a
high-availability service in the event of a node failure. Failover domains are represented
as subordinate elements under Failover Domains. Using configuration buttons at the
bottom of the right frame (below Properties), you can create failover domains (when
Failover Domains is selected) or edit failover domain properties (when a failover domain
is selected).
•Resources — For configuring shared resources to be used by high-availability services.
Shared resources consist of file systems, IP addresses, NFS mounts and exports, and
user-created scripts that are available to any high-availability service in the cluster. Re-
sources are represented as subordinate elements under Resources. Using configuration
buttons at the bottom of the right frame (below Properties), you can create resources
(when Resources is selected) or edit resource properties (when a resource is selected).
Note
The Cluster Configuration Tool provides the capability to configure private
resources, also. A private resource is a resource that is configured for use
with only one service. You can configure a private resource within a Service
component in the GUI.
•Services — For creating and configuring high-availability services. A service is con-
figured by assigning resources (shared or private), assigning a failover domain, and de-
fining a recovery policy for the service. Services are represented as subordinate ele-
ments under Services. Using configuration buttons at the bottom of the right frame
(below Properties), you can create services (when Services is selected) or edit service
properties (when a service is selected).
3.2. Cluster Status Tool
You can access the Cluster Status Tool (Figure 1.7, “Cluster Status Tool”) through the Cluster
Management tab in Cluster Administration GUI.
3.2. Cluster Status Tool
9

Figure 1.7. Cluster Status Tool
The nodes and services displayed in the Cluster Status Tool are determined by the cluster
configuration file (/etc/cluster/cluster.conf). You can use the Cluster Status Tool to enable,
disable, restart, or relocate a high-availability service.
4. Command Line Administration Tools
In addition to Conga and the system-config-cluster Cluster Administration GUI, command line
tools are available for administering the cluster infrastructure and the high-availability service
management components. The command line tools are used by the Cluster Administration GUI
and init scripts supplied by Red Hat. Table 1.1, “Command Line Tools” summarizes the com-
mand line tools.
4. Command Line Administration Tools
10

Command Line
Tool Used With Purpose
ccs_tool —
Cluster Configur-
ation System
Tool
Cluster Infra-
structure ccs_tool is a program for making online updates to the cluster configuration file. It provides the capability to
create and modify cluster infrastructure components
(for example, creating a cluster, adding and removing a
node). For more information about this tool, refer to the
ccs_tool(8) man page.
cman_tool —
Cluster Manage-
ment Tool
Cluster Infra-
structure cman_tool is a program that manages the CMAN cluster manager. It provides the capability to join a cluster,
leave a cluster, kill a node, or change the expected
quorum votes of a node in a cluster. For more informa-
tion about this tool, refer to the cman_tool(8) man page.
fence_tool —
Fence Tool Cluster Infra- structure fence_tool is a program used to join or leave the de- fault fence domain. Specifically, it starts the fence dae-
mon (fenced) to join the domain and kills fenced to
leave the domain. For more information about this tool,
refer to the fence_tool(8) man page.
clustat —
Cluster Status
Utility
High-availability
Service Manage-
ment Compon-
ents
The clustat command displays the status of the
cluster. It shows membership information, quorum view,
and the state of all configured user services. For more
information about this tool, refer to the clustat(8) man
page.
clusvcadm —
Cluster User
Service Adminis-
tration Utility
High-availability
Service Manage-
ment Compon-
ents
The clusvcadm command allows you to enable, disable,
relocate, and restart high-availability services in a
cluster. For more information about this tool, refer to the
clusvcadm(8) man page.
Table 1.1. Command Line Tools
5. Configuration Considerations
You can configure a Red Hat Cluster in a variety of ways to suit your needs. Take into account
the following considerations when you plan, configure, and implement your Red Hat Cluster.
No-single-point-of-failure hardware configuration
Clusters can include a dual-controller RAID array, multiple bonded network channels, mul-
tiple paths between cluster members and storage, and redundant un-interruptible power
supply (UPS) systems to ensure that no single failure results in application down time or
loss of data.
Alternatively, a low-cost cluster can be set up to provide less availability than a no-
single-point-of-failure cluster. For example, you can set up a cluster with a single-controller
RAID array and only a single Ethernet channel.
5. Configuration Considerations
11

Certain low-cost alternatives, such as host RAID controllers, software RAID without cluster
support, and multi-initiator parallel SCSI configurations are not compatible or appropriate for
use as shared cluster storage.
Data integrity assurance
To ensure data integrity, only one node can run a cluster service and access cluster-service
data at a time. The use of power switches in the cluster hardware configuration enables a
node to power-cycle another node before restarting that node's cluster services during a
failover process. This prevents two nodes from simultaneously accessing the same data
and corrupting it. It is strongly recommended that fence devices (hardware or software solu-
tions that remotely power, shutdown, and reboot cluster nodes) are used to guarantee data
integrity under all failure conditions. Watchdog timers provide an alternative way to to en-
sure correct operation of cluster service failover.
Ethernet channel bonding
Cluster quorum and node health is determined by communication of messages among
cluster nodes via Ethernet. In addition, cluster nodes use Ethernet for a variety of other crit-
ical cluster functions (for example, fencing). With Ethernet channel bonding, multiple Ether-
net interfaces are configured to behave as one, reducing the risk of a single-point-of-failure
in the typical switched Ethernet connection among cluster nodes and other cluster hard-
ware.
5. Configuration Considerations
12