Red Hat CLUSTER SUITE - FOR RHEL 4 Installation guide

Red Hat Cluster Suite
for RHEL 4
Overview

Red Hat Cluster Suitefor RHEL 4: Overview
Copyright © 2000-2006 Red Hat, Inc.
Red Hat, Inc.
1801 Varsity Drive
Raleigh NC 27606-2072 USA
Phone: [phone]
Phone: [phone]
Fax: [phone]
PO Box 13588
Research Triangle Park NC 27709 USA
rh-cs [RHCS overview](EN)-4-Print-RHI (2006-1129T16:36)
This material may be distributed only subject to the terms and conditions set forth in the Open Publication
License, V1.0 or later (the latest version is presently available at http://www.opencontent.org/openpub/).
Distribution of substantively modified versions of this material is prohibited without the explicit permission of
the copyright holder. Distribution of the work or derivative of the work in any standard (paper) book form for
commercial purposes is prohibited unless prior permission is obtained from the copyright holder. The content
described in this paragraph is copyrighted by © Red Hat, Inc. (2000-2006).
Red Hat and the Red Hat "Shadow Man" logo are registered trademarks of Red Hat, Inc. in the United States
and other countries.
All other trademarks referenced herein are the property of their respective owners.
The GPG fingerprint of the [email protected] key is:
CA 20 86 86 2B D6 9D FC 65 F6 EC C4 21 91 80 CD DB 42 A6 0E

Table of Contents
About This Document........................................................................................................ i
1. Document Conventions ........................................................................................ i
2. Feedback ..............................................................................................................v
1. Red Hat Cluster Suite Overview ..................................................................................1
1.1. Cluster Basics....................................................................................................1
1.2. Red Hat Cluster Suite Introduction ...................................................................2
1.3. Cluster Infrastructure ........................................................................................4
1.3.1. Cluster Management ..........................................................................4
1.3.2. Lock Management .............................................................................6
1.3.3. Fencing...............................................................................................6
1.3.4. Cluster Configuration System ..........................................................10
1.4. High-availability Service Management ..........................................................12
1.5. Red Hat GFS ...................................................................................................14
1.5.1. Superior Performance and Scalability .............................................16
1.5.2. Performance, Scalability, Moderate Price........................................17
1.5.3. Economy and Performance ..............................................................18
1.6. Cluster Logical Volume Manager ...................................................................19
1.7. Global Network Block Device ........................................................................21
1.8. Linux Virtual Server .......................................................................................22
1.8.1. Two-Tier LVS Topology ..................................................................24
1.8.2. Three-Tier LVS Topology ................................................................27
1.8.3. Routing Methods..............................................................................28
1.8.4. Persistence and Firewall Marks .......................................................32
1.9. Cluster Administration GUI............................................................................33
1.9.1. Cluster Configuration Tool ...........................................................34
1.9.2. Cluster Status Tool.........................................................................38
1.10. Linux Virtual Server Administration GUI ....................................................40
1.10.1. CONTROL/MONITORING.......................................................40
1.10.2. GLOBAL SETTINGS .................................................................41
1.10.3. REDUNDANCY ...........................................................................43
1.10.4. VIRTUAL SERVERS ..................................................................44
2. Red Hat Cluster Suite Component Summary ...........................................................53
2.1. Cluster Components........................................................................................53
2.2. Man Pages ....................................................................................................... 59
Index..................................................................................................................................63

About This Document
This document provides a high-level overview of Red Hat Cluster Suite for Red Hat Enter-
prise Linux 4. Although the information in this document is an overview, you should have
advanced working knowledge of Red Hat Enterprise Linux and understand the concepts of
server computing to gain a good comprehension of the information. For more information
about using Red Hat Enterprise Linux, refer to the following resources:
•Red Hat Enterprise Linux Installation Guide — Provides information regarding instal-
lation.
•Red Hat Enterprise Linux Introduction to System Administration — Provides introduc-
tory information for new Red Hat Enterprise Linux system administrators.
•Red Hat Enterprise Linux System Administration Guide — Provides more detailed in-
formation about configuring Red Hat Enterprise Linux to suit your particular needs as a
user.
•Red Hat Enterprise Linux Reference Guide — Provides detailed information suited for
more experienced users to reference when needed, as opposed to step-by-step instruc-
tions.
•Red Hat Enterprise Linux Security Guide — Details the planning and the tools involved
in creating a secured computing environment for the data center, workplace, and home.
This document contains overview information about Red Hat Cluster Suite for Red Hat
Enterprise Linux 4 and is part of a documentation set that provides conceptual, procedural,
and reference information about Red Hat Cluster Suite for Red Hat Enterprise Linux 4.
Red Hat Cluster Suite documentation and other Red Hat documents are available in HTML,
PDF, and RPM versions on the Red Hat Enterprise Linux Documentation CD and online
at the following location:
http://www.redhat.com/docs/
1. Document Conventions
In this manual, certain words are represented in different fonts, typefaces, sizes, and
weights. This highlighting is systematic; different words are represented in the same style
to indicate their inclusion in a specific category. The types of words that are represented
this way include the following:
command
Linux commands (and other operating system commands, when used) are represented
this way. This style should indicate to you that you can type the word or phrase on
the command line and press [Enter] to invoke a command. Sometimes a command

ii About This Document
contains words that would be displayed in a different style on their own (such as file
names). In these cases, they are considered to be part of the command, so the entire
phrase is displayed as a command. For example:
Use the cat testfile command to view the contents of a file, named testfile,
in the current working directory.
file name
File names, directory names, paths, and RPM package names are represented this
way. This style indicates that a particular file or directory exists with that name on
your system. Examples:
The .bashrc file in your home directory contains bash shell definitions and aliases
for your own use.
The /etc/fstab file contains information about different system devices and file
systems.
Install the webalizer RPM if you want to use a Web server log file analysis program.
application
This style indicates that the program is an end-user application (as opposed to system
software). For example:
Use Mozilla to browse the Web.
[key]
A key on the keyboard is shown in this style. For example:
To use [Tab] completion, type in a character and then press the [Tab] key. Your termi-
nal displays the list of files in the directory that start with that letter.
[key]-[combination]
A combination of keystrokes is represented in this way. For example:
The [Ctrl]-[Alt]-[Backspace] key combination exits your graphical session and returns
you to the graphical login screen or the console.
text found on a GUI interface
A title, word, or phrase found on a GUI interface screen or window is shown in this
style. Text shown in this style indicates that a particular GUI screen or an element on
a GUI screen (such as text associated with a checkbox or field). Example:
Select the Require Password checkbox if you would like your screensaver to require
a password before stopping.

About This Document iii
top level of a menu on a GUI screen or window
A word in this style indicates that the word is the top level of a pulldown menu. If you
click on the word on the GUI screen, the rest of the menu should appear. For example:
Under File on a GNOME terminal, the New Tab option allows you to open multiple
shell prompts in the same window.
Instructions to type in a sequence of commands from a GUI menu look like the fol-
lowing example:
Go to Applications (the main menu on the panel) => Programming => Emacs Text
Editor to start the Emacs text editor.
button on a GUI screen or window
This style indicates that the text can be found on a clickable button on a GUI screen.
For example:
Click on the Back button to return to the webpage you last viewed.
computer output
Text in this style indicates text displayed to a shell prompt such as error messages and
responses to commands. For example:
The ls command displays the contents of a directory. For example:
Desktop about.html logs paulwesterberg.png
Mail backupfiles mail reports
The output returned in response to the command (in this case, the contents of the
directory) is shown in this style.
prompt
A prompt, which is a computer’s way of signifying that it is ready for you to input
something, is shown in this style. Examples:
$
#
[stephen@maturin stephen]$
leopard login:
user input
Text that the user types, either on the command line or into a text box on a GUI screen,
is displayed in this style. In the following example, text is displayed in this style:
To boot your system into the text based installation program, you must type in the
text command at the boot: prompt.

iv About This Document
<replaceable>
Text used in examples that is meant to be replaced with data provided by the user
is displayed in this style. In the following example, <version-number>is dis-
played in this style:
The directory for the kernel source is /usr/src/kernels/<version-number>/,
where <version-number>is the version and type of kernel installed on this
system.
Additionally, we use several different strategies to draw your attention to certain pieces of
information. In order of urgency, these items are marked as a note, tip, important, caution,
or warning. For example:
Note
Remember that Linux is case sensitive. In other words, a rose is not a ROSE is not a
rOsE.
Tip
The directory /usr/share/doc/ contains additional documentation for packages installed
on your system.
Important
If you modify the DHCP configuration file, the changes do not take effect until you restart
the DHCP daemon.
Caution
Do not perform routine tasks as root — use a regular user account unless you need to
use the root account for system administration tasks.

About This Document v
Warning
Be careful to remove only the necessary partitions. Removing other partitions could result
in data loss or a corrupted system environment.
2. Feedback
If you spot a typo, or if you have thought of a way to make this document
better, we would love to hear from you. Please submit a report in Bugzilla
(http://bugzilla.redhat.com/bugzilla/) against the component rh-cs.
Be sure to mention the document’s identifier:
rh-cs [RHCS overview](EN)-4-Print-RHI
By mentioning this document’s identifier, we know exactly which version of the guide you
have.
If you have a suggestion for improving the documentation, try to be as specific as possible.
If you have found an error, please include the section number and some of the surrounding
text so we can find it easily.
(2007-1108T16:26)

vi About This Document

Chapter 1.
Red Hat Cluster Suite Overview
Clustered systems provide reliability, scalability, and availability to critical production ser-
vices. Using Red Hat Cluster Suite, you can create a cluster to suit your needs for perfor-
mance, high availability, load balancing, scalability, file sharing, and economy. This chap-
ter provides an overview of Red Hat Cluster Suite components and functions, and consists
of the following sections:
•Section 1.1 Cluster Basics
•Section 1.2 Red Hat Cluster Suite Introduction
•Section 1.3 Cluster Infrastructure
•Section 1.4 High-availability Service Management
•Section 1.5 Red Hat GFS
•Section 1.6 Cluster Logical Volume Manager
•Section 1.7 Global Network Block Device
•Section 1.8 Linux Virtual Server
•Section 1.9 Cluster Administration GUI
•Section 1.10 Linux Virtual Server Administration GUI
1.1. Cluster Basics
A cluster is two or more computers (called nodes or members) that work together to per-
form a task. There are four major types of clusters:
•Storage
•High availability
•Load balancing
•High performance
Storage clusters provide a consistent file system image across servers in a cluster, allowing
the servers to simultaneously read and write to a single shared file system. A storage cluster
simplifies storage administration by limiting the installation and patching of applications
to one file system. Also, with a cluster-wide file system, a storage cluster eliminates the
need for redundant copies of application data and simplifies backup and disaster recovery.
Red Hat Cluster Suite provides storage clustering through Red Hat GFS.

2 Chapter 1. Red Hat Cluster Suite Overview
High-availability clusters provide continuous availability of services by eliminating single
points of failure and by failing over services from one cluster node to another in case a node
becomes inoperative. Typically, services in a high-availability cluster read and write data
(via read-write mounted file systems). Therefore, a high-availability cluster must maintain
data integrity as one cluster node takes over control of a service from another cluster node.
Node failures in a high-availability cluster are not visible from clients outside the cluster.
(High-availability clusters are sometimes referred to as failover clusters.) Red Hat Cluster
Suite provides high-availability clustering through its High-availability Service Manage-
ment component.
Load-balancing clusters dispatch network service requests to multiple cluster nodes to bal-
ance the request load among the cluster nodes. Load balancing provides cost-effective scal-
ability because you can match the number of nodes according to load requirements. If a
node in a load-balancing cluster becomes inoperative, the load-balancing software detects
the failure and redirects requests to other cluster nodes. Node failures in a load-balancing
cluster are not visible from clients outside the cluster. Red Hat Cluster Suite provides load-
balancing through LVS (Linux Virtual Server).
High-performance clusters use cluster nodes to perform concurrent calculations. A high-
performance cluster allows applications to work in parallel, therefore enhancing the perfor-
mance of the applications. (High performance clusters are also referred to as computational
clusters or grid computing.)
Note
The cluster types summarized in the preceding text reflect basic configurations; your
needs might require a combination of the clusters described.
1.2. Red Hat Cluster Suite Introduction
Red Hat Cluster Suite (RHCS) is an integrated set of software components that can be de-
ployed in a variety of configurations to suit your needs for performance, high-availability,
load balancing, scalability, file sharing, and economy.
RHCS consists of the following major components (refer to Figure 1-1):
•Cluster infrastructure — Provides fundamental functions for nodes to work together as
a cluster: configuration-file management, membership management, lock management,
and fencing.
•High-availability Service Management — Provides failover of services from one cluster
node to another in case a node becomes inoperative.

Chapter 1. Red Hat Cluster Suite Overview 3
•Cluster administration tools — Configuration and management tools for setting up, configuring,
and managing a Red Hat cluster. The tools are for use with the Cluster Infrastructure
components and with the High-availability and ervice Management components.
•Linux Virtual erver (LV ) — Routing software that provides IP-Load-balancing. LV runs in
a pair of redundant servers that distributes client requests evenly to real servers that are behind
the LV servers.
You can supplement Red Hat Cluster uite with the following components, which are part of an
optional package (and not part of Red Hat Cluster uite)
•Red Hat GF (Global File ystem) — Provides a cluster file system for use with Red Hat
Cluster uite. GF allows multiple nodes to share storage at a block level as if the storage were
connected locally to each cluster node.
•Cluster Logical Volume Manager (CLVM) — Provides volume management of cluster storage.
•Global Network Block Device (GNBD) — An ancillary component of GF that exports block-
level storage to Ethernet. This is an economical way to make block-level storage available to
Red Hat GF .
For a lower level summary of Red Hat Cluster uite components, refer to Chapter 2 Red Hat Cluster
Suite Component Summary.
Figure 1-1. Red Hat Cluster Suite Introduction
NOT : Figure 1-1. Red Hat Cluster uite Introduction includes GF , CLVM, and GNBD, which are
components that are part of an optional package and not part of Red Hat Cluster uite.
:

4 Chapter 1. Red Hat Cluster Suite Overview
1.3. Cluster Infrastructure
The Red Hat Cluster Suite cluster infrastructure provides the basic functions for a group of
computers (called nodes or members) to work together as a cluster. Once a cluster is formed
using the cluster infrastructure, you can use other Red Hat Cluster Suite components to
suit your clustering needs (for example, setting up a cluster for sharing files on a GFS file
system or setting up service failover). The cluster infrastructure performs the following
functions:
•Cluster management
•Lock management
•Fencing
•Cluster configuration management
1.3.1. Cluster Management
Cluster management manages cluster quorum and cluster membership. One of the follow-
ing Red Hat Cluster Suite components performs cluster management: CMAN (an abbre-
viation for cluster manager) or GULM (Grand Unified Lock Manager). CMAN operates
as the cluster manager if a cluster is configured to use DLM (Distributed Lock Manager)
as the lock manager. GULM operates as the cluster manager if a cluster is configured to
use GULM as the lock manager. The major difference between the two cluster managers is
that CMAN is a distributed cluster manager and GULM is a client-server cluster manager.
CMAN runs in each cluster node; cluster management is distributed across all nodes in
the cluster (refer to Figure 1-2). GULM runs in nodes designated as GULM server nodes;
cluster management is centralized in the nodes designated as GULM server nodes (refer to
Figure 1-3). GULM server nodes manage the cluster through GULM clients in the cluster
nodes. With GULM, cluster management operates in a limited number of nodes: either
one, three, or five nodes configured as GULM servers.
The cluster manager keeps track of cluster quorum by monitoring the count of cluster nodes
that run cluster manager. (In a CMAN cluster, all cluster nodes run cluster manager; in a
GULM cluster only the GULM servers run cluster manager.) If more than half the nodes
that run cluster manager are active, the cluster has quorum. If half the nodes that run cluster
manager (or fewer) are active, the cluster does not have quorum, and all cluster activity is
stopped. Cluster quorum prevents the occurrence of a "split-brain" condition — a condition
where two instances of the same cluster are running. A split-brain condition would allow
each cluster instance to access cluster resources without knowledge of the other cluster
instance, resulting in corrupted cluster integrity.
In a CMAN cluster, quorum is determined by communication of heartbeats among cluster
nodes via Ethernet. Optionally, quorum can be determined by a combination of communi-
cating heartbeats via Ethernet and through a quorum disk. For quorum via Ethernet, quo-
rum consists of 50 percent of the node votes plus 1. For quorum via quorum disk, quorum
consists of user-specified conditions.

Chapter 1. Red Hat Cluster Suite Overview 5
Note
In a CMAN cluster, by default each node has one quorum vote for establishing quorum.
Optionally, you can configure each node to have more than one vote.
In a GULM cluster, the quorum consists of a majority of nodes designated as GULM
servers according to the number of GULM servers configured:
•Configured with one GULM server — Quorum equals one GULM server.
•Configured with three GULM servers — Quorum equals two GULM servers.
•Configured with five GULM servers — Quorum equals three GULM servers.
The cluster manager keeps track of membership by monitoring heartbeat messages from
other cluster nodes. When cluster membership changes, the cluster manager notifies the
other infrastructure components, which then take appropriate action. For example, if node
A joins a cluster and mounts a GFS file system that nodes B and C have already mounted,
then an additional journal and lock management is required for node A to use that GFS
file system. If a cluster node does not transmit a heartbeat message within a prescribed
amount of time, the cluster manager removes the node from the cluster and communicates
to other cluster infrastructure components that the node is not a member. Again, other
cluster infrastructure components determine what actions to take upon notification that
node is no longer a cluster member. For example, Fencing would fence the node that is no
longer a member.
Figure 1-2. CMAN/DLM Overview

6 Chapter 1. Red Hat Cluster Suite Overview
Figure 1-3. GULM Overview
1.3.2. Lock Management
Lock management is a common cluster-infrastructure service that provides a mechanism
for other cluster infrastructure components to synchronize their access to shared resources.
In a Red Hat cluster, one of the following Red Hat Cluster Suite components operates as
the lock manager: DLM (Distributed Lock Manager) or GULM (Grand Unified Lock Man-
ager). The major difference between the two lock managers is that DLM is a distributed
lock manager and GULM is a client-server lock manager. DLM runs in each cluster node;
lock management is distributed across all nodes in the cluster (refer to Figure 1-2). DLM
can be the lock manager only in a cluster configured with CMAN as its cluster manager.
GULM runs in nodes designated as GULM server nodes; lock management is centralized
in the nodes designated as GULM server nodes. GULM server nodes manage locks through
GULM clients in the cluster nodes (refer to Figure 1-3). With GULM, lock management
operates in a limited number of nodes: either one, three, or five nodes configured as GULM
servers. GFS and CLVM use locks from the lock manager. GFS uses locks from the lock
manager to synchronize access to file system metadata (on shared storage). CLVM uses
locks from the lock manager to synchronize updates to LVM volumes and volume groups
(also on shared storage).
1.3.3. Fencing
Fencing is the disconnection of a node from the cluster’s shared storage. Fencing cuts off
I/O from shared storage, thus ensuring data integrity.
The cluster infrastructure performs fencing through one of the following programs accord-
ing to the type of cluster manager and lock manager that is configured:

Chapter 1. Red Hat Cluster Suite Overview 7
•Configured with CMAN/DLM — fenced, the fence daemon, performs fencing.
•Configured with GULM servers — GULM performs fencing.
When the cluster manager determines that a node has failed, it communicates to other
cluster-infrastructure components that the node has failed. The fencing program (either
fenced or GULM), when notified of the failure, fences the failed node. Other cluster-
infrastructure components determine what actions to take — that is, they perform any
recovery that needs to done. For example, DLM and GFS (in a cluster configured with
CMAN/DLM), when notified of a node failure, suspend activity until they detect that the
fencing program has completed fencing the failed node. Upon confirmation that the failed
node is fenced, DLM and GFS perform recovery. DLM releases locks of the failed node;
GFS recovers the journal of the failed node.
The fencing program determines from the cluster configuration file which fencing method
to use. Two key elements in the cluster configuration file define a fencing method: fencing
agent and fencing device. The fencing program makes a call to a fencing agent specified
in the cluster configuration file. The fencing agent, in turn, fences the node via a fencing
device. When fencing is complete, the fencing program notifies the cluster manager.
Red Hat Cluster Suite provides a variety of fencing methods:
•Power fencing — A fencing method that uses a power controller to power off an inop-
erable node
•Fibre Channel switch fencing — A fencing method that disables the Fibre Channel port
that connects storage to an inoperable node
•GNBD fencing — A fencing method that disables an inoperable node’s access to a
GNBD server
•Other fencing — Several other fencing methods that disable I/O or power of an inoper-
able node, including IBM Bladecenters, PAP, DRAC/MC, HP ILO, IPMI, IBM RSA II,
and others
Figure 1-4 shows an example of power fencing. In the example, the fencing program in
node A causes the power controller to power off node D. Figure 1-5 shows an example of
Fibre Channel switch fencing. In the example, the fencing program in node A causes the
Fibre Channel switch to disable the port for node D, disconnecting node D from storage.

8 Chapter 1. Red Hat Cluster Suite Overview
Figure 1-4. Power Fencing Example
Figure 1-5. Fibre Channel Switch Fencing Example

Chapter 1. Red Hat Cluster Suite Overview 9
Specifying a fencing method consists of editing a cluster configuration file to assign a
fencing-method name, the fencing agent, and the fencing device for each node in the clus-
ter.
Note
Other fencing parameters may be necessary depending on the type of cluster manager
(either CMAN or GULM) selected in a cluster.
The way in which a fencing method is specified depends on if a node has either dual power
supplies or multiple paths to storage. If a node has dual power supplies, then the fencing
method for the node must specify at least two fencing devices — one fencing device for
each power supply (refer to Figure 1-6). Similarly, if a node has multiple paths to Fibre
Channel storage, then the fencing method for the node must specify one fencing device for
each path to Fibre Channel storage. For example, if a node has two paths to Fibre Channel
storage, the fencing method should specify two fencing devices — one for each path to
Fibre Channel storage (refer to Figure 1-7).
Figure 1-6. Fencing a Node with Dual Power Supplies

10 Chapter 1. Red Hat Cluster Suite Overview
Figure 1-7. Fencing a Node with Dual Fibre Channel Connections
You can configure a node with one fencing method or multiple fencing methods. When
you configure a node for one fencing method, that is the only fencing method available for
fencing that node. When you configure a node for multiple fencing methods, the fencing
methods are cascaded from one fencing method to another according to the order of the
fencing methods specified in the cluster configuration file. If a node fails, it is fenced using
the first fencing method specified in the cluster configuration file for that node. If the first
fencing method is not successful, the next fencing method specified for that node is used.
If none of the fencing methods is successful, then fencing starts again with the first fencing
method specified, and continues looping through the fencing methods in the order specified
in the cluster configuration file until the node has been fenced.
1.3.4. Cluster Configuration System
The Cluster Configuration System (CCS) manages the cluster configuration and provides
configuration information to other cluster components in a Red Hat cluster. CCS runs in
each cluster node and makes sure that the cluster configuration file in each cluster node is
up to date. For example, if a cluster system administrator updates the configuration file in
Node A, CCS propagates the update from Node A to the other nodes in the cluster (refer
to Figure 1-8).