Novell LINUX ENTERPRISE DESKTOP 10 SP2 User manual

SUSE Linux Enterprise
Server
www.novell.com10 SP2
May08,2008 Heartbeat

Heartbeat
All content is copyright © Novell, Inc.
Legal Notice
This manual is protected under Novell intellectual property rights. By reproducing, duplicating or
distributing this manual you explicitly agree to conform to the terms and conditions of this license
agreement.
This manual may be freely reproduced, duplicated and distributed either as such or as part of a bundled
package in electronic and/or printed format, provided however that the following conditions are ful-
lled:
That this copyright notice and the names of authors and contributors appear clearly and distinctively
on all reproduced, duplicated and distributed copies. That this manual, specically for the printed
format, is reproduced and/or distributed for noncommercial use only. The express authorization of
Novell, Inc must be obtained prior to any other use of any manual or part thereof.
For Novell trademarks, see the Novell Trademark and Service Mark list http://www.novell
.com/company/legal/trademarks/tmlist.html. * Linux is a registered trademark of
Linus Torvalds. All other third party trademarks are the property of their respective owners. A trademark
symbol (®, ™ etc.) denotes a Novell trademark; an asterisk (*) denotes a third party trademark.
All information found in this book has been compiled with utmost attention to detail. However, this
does not guarantee complete accuracy. Neither Novell, Inc., SUSE LINUX Products GmbH, the authors,
nor the translators shall be held liable for possible errors or the consequences thereof.

Contents
About This Guide v
1Overview 1
1.1 ProductFeatures.......................... 1
1.2 ProductBenets.......................... 2
1.3 Cluster Congurations . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Heartbeat Cluster Components . . . . . . . . . . . . . . . . . . . . 8
1.5 Architecture............................ 8
2Installation and Setup 13
2.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Software Requirements . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Shared Disk System Requirements . . . . . . . . . . . . . . . . . . 14
2.4 Preparations........................... 15
2.5 Installing Heartbeat . . . . . . . . . . . . . . . . . . . . . . . . 17
2.6 Conguring STONITH . . . . . . . . . . . . . . . . . . . . . . . 21
3Setting Up a Simple Resource 25
3.1 Conguring a Resource with the Heartbeat GUI . . . . . . . . . . . . 25
3.2 Manual Conguration of a Resource . . . . . . . . . . . . . . . . . 27
4Conguring and Managing Cluster Resources 29
4.1 Graphical HA Management Client . . . . . . . . . . . . . . . . . . 29
4.2 Creating Cluster Resources . . . . . . . . . . . . . . . . . . . . . 31
4.3 Conguring Resource Constraints . . . . . . . . . . . . . . . . . . 33
4.4 Specifying Resource Failover Nodes . . . . . . . . . . . . . . . . . 33

4.5 Specifying Resource Failback Nodes (Resource Stickiness) . . . . . . . . 34
4.6 Conguring Resource Monitoring . . . . . . . . . . . . . . . . . . 35
4.7 Starting a New Cluster Resource . . . . . . . . . . . . . . . . . . . 36
4.8 Removing a Cluster Resource . . . . . . . . . . . . . . . . . . . . 36
4.9 Conguring a Heartbeat Cluster Resource Group . . . . . . . . . . . . 37
4.10 Conguring a Heartbeat Clone Resource . . . . . . . . . . . . . . . 40
4.11 Migrating a Cluster Resource . . . . . . . . . . . . . . . . . . . . 41
5Manual Conguration of a Cluster 43
5.1 Conguration Basics . . . . . . . . . . . . . . . . . . . . . . . . 44
5.2 Conguring Resources . . . . . . . . . . . . . . . . . . . . . . . 47
5.3 Conguring Constraints . . . . . . . . . . . . . . . . . . . . . . 52
5.4 Conguring CRM Options . . . . . . . . . . . . . . . . . . . . . 55
5.5 For More Information . . . . . . . . . . . . . . . . . . . . . . . 59
6Managing a Cluster 61
7Creating Resources 103
7.1 STONITHAgents......................... 103
7.2 ResourceAgents......................... 104
7.3 Writing OCF Resource Agents . . . . . . . . . . . . . . . . . . . 104
8Troubleshooting 107
AHB OCF Agents 109
Terminology 173

About This Guide
Heartbeat is an open source clustering software for Linux. Heartbeat ensures high
availability and manageability of critical network resources including data, applications,
and services. It is a multinode clustering product that supports failover, failback, and
migration (load balancing) of individually managed cluster resources.
This guide is intended for administrators given the task of building Linux clusters. Basic
background information is provided to build up an understanding of the Heartbeat ar-
chitecture. Setup, conguration, and maintenance of a Heartbeat cluster are covered in
detail. Heartbeat offers a graphical user interface as well as many command line tools.
Both approaches are covered in detail to help the administrators choose the appropriate
tool matching their particular needs.
NOTE
This manual covers Heartbeat version 2 and higher. Whenever the authors use
the term “Heartbeat”, they refer to Heartbeat 2, even when no explicit version
number is given.
This guide contains the following:
Overview
Before starting to install and congure your cluster, learn about the features
Heartbeat offers. Familiarize yourself with the Heartbeat terminology and its basic
concepts.
Installation and Setup
Learn about hardware and software requirements that must be met before you can
consider installing and running your own cluster. Perform a basic installation and
conguration using the Heartbeat graphical user interface.
Setting Up a Simple Resource
After you have completed the basic cluster conguration, check out a quick step-
by-step instruction of how to congure an example resource. Choose from a GUI-
based approach or a command line-driven one.

Conguring and Managing Cluster Resources
Managing resources encompasses much more than just the initial conguration.
Learn how to use the Heartbeat graphical user interface to congure and manage
resources.
Manual Conguration of a Cluster
Managing resources encompasses much more than just the initial conguration.
Learn how to use the Heartbeat command line tools to congure and manage re-
sources.
Managing a Cluster
Heartbeat provides a comprehensive set of command line tools to assist you in
managing your own cluster. Get to know the most important ones for your daily
cluster management tasks.
Creating Resources
In case you consider writing your own resource agents for Heartbeat or modifying
existing ones, get some detailed background information about the different types
of resource agents and how to create them.
Troubleshooting
Managing your own cluster requires you to perform a certain amount of troubleshoot-
ing. Learn about the most common problems with Heartbeat and how to x them.
Terminology
Refer to this chapter for some basic Heartbeat terminology that helps you understand
the Heartbeat fundamentals.
1 Feedback
We want to hear your comments and suggestions about this manual and the other doc-
umentation included with this product. Please use the User Comments feature at the
bottom of each page of the online documentation and enter your comments there.
vi Heartbeat

2 Documentation Updates
Expect updates to this documentation in the near future while it is extended to match
the capabilities of the software itself and to address more and more use cases. By using
the User Comments feature described in Section 1, “Feedback” (page vi), let us know
on which aspects of Heartbeat this guide should elaborate.
For the latest version of this documentation, see the SLES 10 SP2 documentation Web
site at http://www.novell.com/documentation/sles10.
3 Documentation Conventions
The following typographical conventions are used in this manual:
•/etc/passwd: lenames and directory names
•placeholder: replace placeholder with the actual value
•PATH: the environment variable PATH
•ls,--help: commands, options, and parameters
•user: users or groups
•Alt,Alt +F1: a key to press or a key combination; keys are shown in uppercase as
on a keyboard
•File,File >Save As: menu items, buttons
•►amd64 ipf: This paragraph is only relevant for the specied architectures. The
arrows mark the beginning and the end of the text block.◄
►ipseries s390 zseries: This paragraph is only relevant for the specied architec-
tures. The arrows mark the beginning and the end of the text block.◄
•Dancing Penguins (Chapter Penguins, ↑Another Manual): This is a reference to a
chapter in another manual.
About This Guide vii

1
Overview
Heartbeat is an open source server clustering system that ensures high availability and
manageability of critical network resources including data, applications, and services.
It is a multinode clustering product for Linux that supports failover, failback, and mi-
gration (load balancing) of individually managed cluster resources. Heartbeat is shipped
with SUSE Linux Enterprise Server 10 and provides you with the means to make virtual
machines (containing services) highly available.
This chapter introduces the main product features and benets of Heartbeat. Find sev-
eral exemplary scenarios for conguring clusters and learn about the components
making up a Heartbeat version 2 cluster. The last section provides an overview of the
Heartbeat architecture, describing the individual architecture layers and processes
within the cluster.
1.1 Product Features
Heartbeat includes several important features to help you ensure and manage the
availability of your network resources. These include:
• Support for Fibre Channel or iSCSI storage area networks
• Multi-node active cluster, containing up to 16 Linux servers. Any server in the
cluster can restart resources (applications, services, IP addresses, and le systems)
from a failed server in the cluster.
Overview 1

• A single point of administration through either a graphical Heartbeat tool or a
command line tool. Both tools let you congure and monitor your Heartbeat cluster.
•The ability to tailor a cluster to the specic applications and hardware infrastructure
that t your organization.
• Dynamic assignment and reassignment of server storage on an as-needed basis.
• Time-dependent conguration enables resources to fail back to repaired nodes at
specied times.
• Support for shared disk systems. Although shared disk systems are supported, they
are not required.
• Support for cluster le systems like OCFS 2.
• Support for cluster-aware logical volume managers like EVMS.
1.2 Product Benets
Heartbeat allows you to congure up to 16 Linux servers into a high-availability cluster,
where resources can be dynamically switched or moved to any server in the cluster.
Resources can be congured to automatically switch or be moved in the event of a re-
source server failure, or they can be moved manually to troubleshoot hardware or balance
the workload.
Heartbeat provides high availability from commodity components. Lower costs are
obtained through the consolidation of applications and operations onto a cluster.
Heartbeat also allows you to centrally manage the complete cluster and to adjust re-
sources to meet changing workload requirements (thus, manually “load balance” the
cluster).
An equally important benet is the potential reduction of unplanned service outages as
well as planned outages for software and hardware maintenance and upgrades.
Reasons that you would want to implement a Heartbeat cluster include:
• Increased availability
• Improved performance
2Heartbeat

• Low cost of operation
• Scalability
• Disaster recovery
• Data protection
• Server consolidation
• Storage consolidation
Shared disk fault tolerance can be obtained by implementing RAID on the shared disk
subsystem.
The following scenario illustrates some of the benets Heartbeat can provide.
Suppose you have congured a three-server cluster, with a Web server installed on
each of the three servers in the cluster. Each of the servers in the cluster hosts two Web
sites. All the data, graphics, and Web page content for each Web site are stored on a
shared disk subsystem connected to each of the servers in the cluster. The following
gure depicts how this setup might look.
Figure 1.1 Three-Server Cluster
During normal cluster operation, each server is in constant communication with the
other servers in the cluster and performs periodic polling of all registered resources to
detect failure.
Overview 3

Suppose Web Server 1 experiences hardware or software problems and the users de-
pending on Web Server 1 for Internet access, e-mail, and information lose their connec-
tions. The following gure shows how resources are moved when Web Server 1 fails.
Figure 1.2 Three-Server Cluster after One Server Fails
Web Site A moves to Web Server 2 and Web Site B moves to Web Server 3. IP addresses
and certicates also move to Web Server 2 and Web Server 3.
When you congured the cluster, you decided where the Web sites hosted on each Web
server would go should a failure occur. In the previous example, you congured Web
Site A to move to Web Server 2 and Web Site B to move to Web Server 3. This way,
the workload once handled by Web Server 1 continues to be available and is evenly
distributed between any surviving cluster members.
When Web Server 1 failed, Heartbeat software
• Detected a failure
• Remounted the shared data directories that were formerly mounted on Web server
1 on Web Server 2 and Web Server 3.
• Restarted applications that were running on Web Server 1 on Web Server 2 and
Web Server 3
• Transferred IP addresses to Web Server 2 and Web Server 3
In this example, the failover process happened quickly and users regained access to
Web site information within seconds, and in most cases, without needing to log in again.
4Heartbeat

Now suppose the problems with Web Server 1 are resolved, and Web Server 1 is returned
to a normal operating state. Web Site A and Web Site B can either automatically fail
back (move back) to Web Server 1, or they can stay where they are. This is dependent
on how you congured the resources for them. There are advantages and disadvantages
to both alternatives. Migrating the services back to Web Server 1 will incur some down-
time. Heartbeat also allows you to defer the migration until a period when it will cause
little or no service interruption.
Heartbeat also provides resource migration capabilities. You can move applications,
Web sites, etc. to other servers in your cluster without waiting for a server to fail.
For example, you could have manually moved Web Site A or Web Site B from Web
Server 1 to either of the other servers in the cluster. You might want to do this to upgrade
or perform scheduled maintenance on Web Server 1, or just to increase performance
or accessibility of the Web sites.
1.3 Cluster Congurations
Heartbeat cluster congurations might or might not include a shared disk subsystem.
The shared disk subsystem can be connected via high-speed Fibre Channel cards, cables,
and switches, or it can be congured to use iSCSI. If a server fails, another designated
server in the cluster automatically mounts the shared disk directories previously
mounted on the failed server. This gives network users continuous access to the direc-
tories on the shared disk subsystem.
IMPORTANT: Shared Disk Subsystem with EVMS
When using a shared disk subsystem with EVMS, that subsystem must be con-
nected to all servers in the cluster.
Typical Heartbeat resources might include data, applications, and services. The following
gure shows how a typical Fibre Channel cluster conguration might look.
Overview 5

Figure 1.3 Typical Fibre Channel Cluster Conguration
Although Fibre Channel provides the best performance, you can also congure your
cluster to use iSCSI. iSCSI is an alternative to Fibre Channel that can be used to create
a low-cost SAN. The following gure shows how a typical iSCSI cluster conguration
might look.
6Heartbeat

Figure 1.4 Typical iSCSI Cluster Conguration
Although most clusters include a shared disk subsystem, it is also possible to create a
Heartbeat cluster without a share disk subsystem. The following gure shows how a
Heartbeat cluster without a shared disk subsystem might look.
Figure 1.5 Typical Cluster Conguration Without Shared Storage
Overview 7

1.4 Heartbeat Cluster Components
The following components make up a Heartbeat version 2 cluster:
• From 2 to 16 Linux servers, each containing at least one local disk device.
• Heartbeat software running on each Linux server in the cluster.
• Optional: A shared disk subsystem connected to all servers in the cluster.
• Optional: High-speed Fibre Channel cards, cables, and switch used to connect the
servers to the shared disk subsystem.
• At least two communications mediums over which Heartbeat servers can commu-
nicate. These currently include Ethernet (mcast, ucast, or bcast) or Serial.
• A STONITH device. A STONITH device is a power switch which the cluster uses
to reset nodes that are considered dead. Resetting non-heartbeating nodes is the
only reliable way to ensure that no data corruption is performed by nodes that hang
and only appear to be dead.
See The High-Availability Linux Project [http://www.linux-ha.org/
STONITH] for more information on STONITH.
1.5 Architecture
This section provides a brief overview of the Heartbeat architecture. It identies and
provides information on the Heartbeat architectural components, and describes how
those components interoperate.
1.5.1 Architecture Layers
Heartbeat has a layered architecture. Figure 1.6, “Heartbeat Architecture” (page 9)
illustrates the different layers and their associated components.
8Heartbeat

Figure 1.6 Heartbeat Architecture
Messaging and Infrastructure Layer
The primary or rst layer is the messaging/infrastructure layer, also known as the
Heartbeat layer. This layer contains components that send out the Heartbeat messages
containing “I'm alive” signals, as well as other information. The Heartbeat program
resides in the messaging/infrastructure layer.
Membership Layer
The second layer is the membership layer. The membership layer is responsible for
calculating the largest fully connected set of cluster nodes and synchronizing this view
Overview 9

to all of its members. It performs this task based on the information it gets from the
Heartbeat layer. The logic that takes care of this task is contained in the Cluster Con-
sensus Membership service, which provides an organized cluster topology overview
(node-wise) to cluster components that are the higher layers.
Resource Allocation Layer
The third layer is the resource allocation layer. This layer is the most complex, and
consists of the following components:
Cluster Resource Manager
Every action taken in the resource allocation layer passes through the Cluster Re-
source Manager. If any other components of the resource allocation layer, or other
components which are in a higher layer need to communicate, they do so through
the local Cluster Resource Manager.
On every node, the Cluster Resource Manager maintains the Cluster Information
Base, or CIB (see Cluster Information Base (page 10) below). One Cluster Resource
Manager in the cluster is elected as the Designated Coordinator (DC), meaning that
it has the master CIB. All other CIBs in the cluster are a replicas of the master CIB.
Normal read and write operations on the CIB are serialized through the master CIB.
The DC is the only entity in the cluster that can decide that a cluster-wide change
needs to be performed, such as fencing a node or moving resources around.
Cluster Information Base
The Cluster Information Base or CIB is an in-memory XML representation of the
entire cluster conguration and status, including node membership, resources,
constraints, etc. There is one master CIB in the cluster, maintained by the DC. All
the other nodes contain a CIB replica. If an administrator wants to manipulate the
cluster's behavior, he can use either the cibadmin command line tool or the
Heartbeat GUI tool.
NOTE: Usage of Heartbeat GUI Tool and cibadmin
The Heartbeat GUI tool can be used from any machine with a connection
to the cluster. The cibadmin command must be used on a cluster node,
and is not restricted to only the DC node.
10 Heartbeat

Policy Engine (PE) and Transition Engine (TE)
Whenever the Designated Coordinator needs to make a cluster-wide change (react
to a new CIB), the Policy Engine is used to calculate the next state of the cluster
and the list of (resource) actions required to achieve it. The commands computed
by the Policy Engine are then executed by the Transition Engine. The DC will send
out messages to the relevant Cluster Resource Managers in the cluster, who then
use their Local Resource Managers (see Local Resource Manager (LRM) (page 11)
below) to perform the necessary resource manipulations. The PE/TE pair only runs
on the DC node.
Local Resource Manager (LRM)
The Local Resource Manager calls the local Resource Agents (see Section “Resource
Layer” (page 11) below) on behalf of the CRM. It can thus perform start / stop /
monitor operations and report the result to the CRM. The LRM is the authoritative
source for all resource related information on its local node.
Resource Layer
The fourth and highest layer is the Resource Layer. The Resource Layer includes one
or more Resource Agents (RA). A Resource Agent is a program, usually a shell script,
that has been written to start, stop, and monitor a certain kind of service (a resource).
The most common Resource Agents are LSB init scripts. However, Heartbeat also
supports the more exible and powerful Open Clustering Framework Resource Agent
API. The agents supplied with Heartbeat are written to OCF specications. Resource
Agents are called only by the Local Resource Manager. Third parties can include their
own agents in a dened location in the le system and thus provide out-of-the-box
cluster integration for their own software.
1.5.2 Process Flow
Many actions performed in the cluster will cause a cluster-wide change. These actions
can include things like adding or removing a cluster resource or changing resource
constraints. It is important to understand what happens in the cluster when you perform
such an action.
For example, suppose you want to add a cluster IP address resource. To do this, you
use either the cibadmin command line tool or the Heartbeat GUI tool to modify the
master CIB. It is not required to use the cibadmin command or the GUI tool on the
Overview 11

Designated Coordinator. You can use either tool on any node in the cluster, and the
local CIB will relay the requested changes to the Designated Coordinator. The Desig-
nated Coordinator will then replicate the CIB change to all cluster nodes and will start
the transition procedure.
With help of the Policy Engine and the Transition Engine, the Designated Coordinator
obtains a series of steps that need to be performed in the cluster, possibly on several
nodes. The Designated Coordinator sends commands out via the messaging/infrastructure
layer which are received by the other Cluster Resource Managers.
If necessary, the other Cluster Resource Managers use their Local Resource Manager
to perform resource modications and report back to the Designated Coordinator about
the result. Once the Transition Engine on the Designated Coordinator concludes that
all necessary operations are successfully performed in the cluster, the cluster will go
back to the idle state and wait for further events.
If any operation was not carried out as planned, the Policy Engine is invoked again
with the new information recorded in the CIB.
When a service or a node dies, the same thing happens. The Designated Coordinator
is informed by the Cluster Consensus Membership service (in case of a node death) or
by a Local Resource Manager (in case of a failed monitor operation). The Designated
Coordinator determines that actions need to be taken in order to come to a new cluster
state. The new cluster state will be represented by a new CIB.
12 Heartbeat