Novell LINUX ENTERPRISE 11 - HIGH AVAILABILITY User manual

SUSE Linux Enterprise
High Availability
Extension
www.novell.com11
February18,2010 High Availability Guide

High Availability Guide
Copyright © 2006- 2010 Novell, Inc.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU
Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section
being this copyright notice and license. A copy of the license version 1.2 is included in the section
entitled “GNU Free Documentation License”.
SUSE®, openSUSE®, the openSUSE® logo, Novell®, the Novell® logo, the N® logo, are registered
trademarks of Novell, Inc. in the United States and other countries. Linux* is a registered trademark
of Linus Torvalds. All other third party trademarks are the property of their respective owners. A
trademark symbol (® , ™, etc.) denotes a Novell trademark; an asterisk (*) denotes a third-party
trademark.
All information found in this book has been compiled with utmost attention to detail. However, this
does not guarantee complete accuracy. Neither Novell, Inc., SUSE LINUX Products GmbH, the authors,
nor the translators shall be held liable for possible errors or the consequences thereof.

Contents
About This Guide vii
Part I Installation and Setup 1
1Conceptual Overview 3
1.1 ProductFeatures.......................... 3
1.2 ProductBenets.......................... 5
1.3 Cluster Congurations . . . . . . . . . . . . . . . . . . . . . . . 8
1.4 Architecture........................... 11
1.5 What'sNew?........................... 14
2Getting Started 19
2.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . 19
2.2 Software Requirements . . . . . . . . . . . . . . . . . . . . . . 20
2.3 Shared Disk System Requirements . . . . . . . . . . . . . . . . . . 20
2.4 Preparations........................... 21
2.5 Overview: Installing and Setting Up a Cluster . . . . . . . . . . . . . 21
3Installation and Basic Setup with YaST 23
3.1 Installing the High Availability Extension . . . . . . . . . . . . . . . 23
3.2 Initial Cluster Setup . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3 Bringing the Cluster Online . . . . . . . . . . . . . . . . . . . . . 27

Part II Conguration and Administration 29
4Conguring Cluster Resources with the GUI 31
4.1 Linux HA Management Client . . . . . . . . . . . . . . . . . . . . 32
4.2 Creating Cluster Resources . . . . . . . . . . . . . . . . . . . . . 33
4.3 Creating STONITH Resources . . . . . . . . . . . . . . . . . . . . 37
4.4 Conguring Resource Constraints . . . . . . . . . . . . . . . . . . 38
4.5 Specifying Resource Failover Nodes . . . . . . . . . . . . . . . . . 43
4.6 Specifying Resource Failback Nodes (Resource Stickiness) . . . . . . . . 45
4.7 Conguring Resource Monitoring . . . . . . . . . . . . . . . . . . 46
4.8 Starting a New Cluster Resource . . . . . . . . . . . . . . . . . . . 48
4.9 Removing a Cluster Resource . . . . . . . . . . . . . . . . . . . . 49
4.10 Conguring a Cluster Resource Group . . . . . . . . . . . . . . . . 49
4.11 Conguring a Clone Resource . . . . . . . . . . . . . . . . . . . . 54
4.12 Migrating a Cluster Resource . . . . . . . . . . . . . . . . . . . . 55
4.13 For More Information . . . . . . . . . . . . . . . . . . . . . . . 57
5Conguring Cluster Resources From Command Line 59
5.1 Command Line Tools . . . . . . . . . . . . . . . . . . . . . . . 59
5.2 Debugging Your Conguration Changes . . . . . . . . . . . . . . . 60
5.3 Creating Cluster Resources . . . . . . . . . . . . . . . . . . . . . 60
5.4 Creating a STONITH Resource . . . . . . . . . . . . . . . . . . . . 65
5.5 Conguring Resource Constraints . . . . . . . . . . . . . . . . . . 66
5.6 Specifying Resource Failover Nodes . . . . . . . . . . . . . . . . . 68
5.7 Specifying Resource Failback Nodes (Resource Stickiness) . . . . . . . . 69
5.8 Conguring Resource Monitoring . . . . . . . . . . . . . . . . . . 69
5.9 Starting a New Cluster Resource . . . . . . . . . . . . . . . . . . . 69
5.10 Removing a Cluster Resource . . . . . . . . . . . . . . . . . . . . 70
5.11 Conguring a Cluster Resource Group . . . . . . . . . . . . . . . . 70
5.12 Conguring a Clone Resource . . . . . . . . . . . . . . . . . . . . 71
5.13 Migrating a Cluster Resource . . . . . . . . . . . . . . . . . . . . 72
5.14 Testing with Shadow Conguration . . . . . . . . . . . . . . . . . 73
5.15 For More Information . . . . . . . . . . . . . . . . . . . . . . . 74
6Setting Up a Simple Testing Resource 75
6.1 Conguring a Resource with the GUI . . . . . . . . . . . . . . . . . 75
6.2 Manual Conguration of a Resource . . . . . . . . . . . . . . . . . 77
7Adding or Modifying Resource Agents 79
7.1 STONITHAgents......................... 79
7.2 Writing OCF Resource Agents . . . . . . . . . . . . . . . . . . . . 80

8Fencing and STONITH 81
8.1 ClassesofFencing......................... 81
8.2 NodeLevelFencing........................ 82
8.3 STONITH Conguration . . . . . . . . . . . . . . . . . . . . . . 84
8.4 Monitoring Fencing Devices . . . . . . . . . . . . . . . . . . . . 88
8.5 Special Fencing Devices . . . . . . . . . . . . . . . . . . . . . . 89
8.6 For More Information . . . . . . . . . . . . . . . . . . . . . . . 90
9Load Balancing with Linux Virtual Server 91
9.1 Conceptual Overview . . . . . . . . . . . . . . . . . . . . . . . 91
9.2 HighAvailability ......................... 93
9.3 For More Information . . . . . . . . . . . . . . . . . . . . . . . 94
10 Network Device Bonding 95
11 Updating Your Cluster to SUSE Linux Enterprise 11 99
11.1 Preparation and Backup . . . . . . . . . . . . . . . . . . . . . . 100
11.2 Update/Installation . . . . . . . . . . . . . . . . . . . . . . . . 101
11.3 DataConversion......................... 101
11.4 For More Information . . . . . . . . . . . . . . . . . . . . . . 103
Part III Storage and Data Replication 105
12 Oracle Cluster File System 2 107
12.1 Features and Benets . . . . . . . . . . . . . . . . . . . . . . . 107
12.2 Management Utilities and Commands . . . . . . . . . . . . . . . . 108
12.3 OCFS2Packages......................... 109
12.4 Creating an OCFS2 Volume . . . . . . . . . . . . . . . . . . . . 110
12.5 Mounting an OCFS2 Volume . . . . . . . . . . . . . . . . . . . . 113
12.6 Additional Information . . . . . . . . . . . . . . . . . . . . . . 115
13 Cluster LVM 117
13.1 Conguration of cLVM . . . . . . . . . . . . . . . . . . . . . . 117
13.2 Conguring Eligible LVM2 Devices Explicitly . . . . . . . . . . . . . 119
13.3 For More Information . . . . . . . . . . . . . . . . . . . . . . 120
14 Distributed Replicated Block Device (DRBD) 121
14.1 Installing DRBD Services . . . . . . . . . . . . . . . . . . . . . . 122

14.2 Conguring the DRBD Service . . . . . . . . . . . . . . . . . . . 122
14.3 Testing the DRBD Service . . . . . . . . . . . . . . . . . . . . . 124
14.4 Troubleshooting DRBD . . . . . . . . . . . . . . . . . . . . . . 126
14.5 Additional Information . . . . . . . . . . . . . . . . . . . . . . 128
Part IV Troubleshooting and Reference 131
15 Troubleshooting 133
15.1 Installation Problems . . . . . . . . . . . . . . . . . . . . . . . 133
15.2 “Debugging” a HA Cluster . . . . . . . . . . . . . . . . . . . . . 134
15.3 FAQs.............................. 136
15.4 Fore More Information . . . . . . . . . . . . . . . . . . . . . . 137
16 Cluster Management Tools 139
17 Cluster Resources 193
17.1 Supported Resource Agent Classes . . . . . . . . . . . . . . . . . 193
17.2 OCFReturnCodes........................ 194
17.3 Resource Options . . . . . . . . . . . . . . . . . . . . . . . . 197
17.4 Resource Operations . . . . . . . . . . . . . . . . . . . . . . . 198
17.5 Instance Attributes . . . . . . . . . . . . . . . . . . . . . . . . 199
18 HA OCF Agents 201
Part V Appendix 275
AGNU Licenses 277
A.1 GNU General Public License . . . . . . . . . . . . . . . . . . . . 277
A.2 GNU Free Documentation License . . . . . . . . . . . . . . . . . 280
Terminology 285

About This Guide
SUSE® Linux Enterprise High Availability Extension is an integrated suite of open
source clustering technologies that enables you to implement highly available physical
and virtual Linux clusters. For quick and efcient conguration and administration,
the High Availability Extension includes both a graphical user interface (GUI) and a
command line interface (CLI).
This guide is intended for administrators who need to set up, congure, and maintain
High Availability (HA) clusters. Both approaches (GUI and CLI) are covered in detail
to help the administrators choose the appropriate tool that matches their needs for per-
forming the key tasks.
The guide is divided into the following parts:
Installation and Setup
Before starting to install and congure your cluster, make yourself familiar with
cluster fundamentals and architecture, get an overview of the key features and
benets, as well as modications since the last release. Learn which hardware and
software requirements must be met and what preparations to take before executing
the next steps. Perform the installation and basic setup of your HA cluster using
YaST.
Conguration and Administration
Add, congure and manage resources, using either the GUI or the crm command
line interface. Learn how to make use of load balancing and fencing. In case you
consider writing your own resource agents or modifying existing ones, get some
background information on how to create different types of resource agents.
Storage and Data Replication
SUSE Linux Enterprise High Availability Extension ships with a cluster-aware le
system (Oracle Cluster File System, OCFS2) and volume manager (clustered
Logical Volume Manager, cLVM). For replication of your data, the High Availabil-
ity Extension also delivers DRBD (Distributed Replicated Block Device) which
you can use to mirror the data of a high availabilitly service from the active node
of a cluster to its standby node.

Troubleshooting and Reference
Managing your own cluster requires you to perform a certain amount of troubleshoot-
ing. Learn about the most common problems and how to x them. Find a compre-
hensive reference of the command line tools the High Availability Extension offers
for administering your own cluster. Also, nd a list of the most important facts and
gures about cluster resources and resource agents.
Many chapters in this manual contain links to additional documentation resources.
These include additional documentation that is available on the system as well as doc-
umentation available on the Internet.
For an overview of the documentation available for your product and the latest docu-
mentation updates, refer to http://www.novell.com/documentation.
1 Feedback
Several feedback channels are available:
• To report bugs for a product component or to submit enhancement requests, please
use https://bugzilla.novell.com/. If you are new to Bugzilla, you
might nd the Bug Writing FAQs helpful, available from the Novell Bugzilla home
page.
• We want to hear your comments and suggestions about this manual and the other
documentation included with this product. Please use the User Comments feature
at the bottom of each page of the online documentation and enter your comments
there.
2 Documentation Conventions
The following typographical conventions are used in this manual:
•/etc/passwd: directory names and lenames
•placeholder: replace placeholder with the actual value
•PATH: the environment variable PATH
viii High Availability Guide

•ls,--help: commands, options, and parameters
•user: users or groups
•Alt,Alt +F1: a key to press or a key combination; keys are shown in uppercase as
on a keyboard
•File,File >Save As: menu items, buttons
• This paragraph is only relevant for the specied architectures. The arrows mark
the beginning and the end of the text block.
This paragraph is only relevant for the specied architectures. The arrows mark
the beginning and the end of the text block.
•Dancing Penguins (Chapter Penguins, ↑Another Manual): This is a reference to a
chapter in another manual.
About This Guide ix

Part I. Installation and Setup

1
Conceptual Overview
SUSE® Linux Enterprise High Availability Extension is an integrated suite of open
source clustering technologies that enables you to implement highly available physical
and virtual Linux clusters, and to eliminate single points of failure. It ensures the high
availability and manageability of critical network resources including data, applications,
and services. Thus, it helps you maintain business continuity, protect data integrity,
and reduce unplanned downtime for your mission-critical Linux workloads.
It ships with essential monitoring, messaging, and cluster resource management func-
tionality (supporting failover, failback, and migration (load balancing) of individually
managed cluster resources). The High Availability Extension is available as add-on to
SUSE Linux Enterprise Server 11.
1.1 Product Features
SUSE® Linux Enterprise High Availability Extension helps you ensure and manage
the availability of your network resources. The following list highlights some of the
key features:
Support for a Wide Range of Clustering Scenarios
Including active/active and active/passive (N+1, N+M, N to 1, N to M) scenarios,
as well as hybrid physical and virtual clusters (allowing virtual servers to be clus-
tered with physical servers to improve services availability and resource utilization).
Conceptual Overview 3

Multi-node active cluster, containing up to 16 Linux servers. Any server in the
cluster can restart resources (applications, services, IP addresses, and le systems)
from a failed server in the cluster.
Flexible Solution
The High Availability Extension ships with OpenAIS messaging and membership
layer and Pacemaker Cluster Resource Manager. Using Pacemaker, administrators
can continually monitor the health and status of their resources, manage dependen-
cies, and automatically stop and start services based on highly congurable rules
and policies. The High Availability Extension allows you to tailor a cluster to the
specic applications and hardware infrastructure that t your organization. Time-
dependent conguration enables services to automatically migrate back to repaired
nodes at specied times.
Storage and Data Replication
With the High Availability Extension you can dynamically assign and reassign
server storage as needed. It supports Fibre Channel or iSCSI storage area networks
(SANs). Shared disk systems are also supported, but they are not a requirement.
SUSE Linux Enterprise High Availability Extension also comes with a cluster-
aware le system (Oracle Cluster File System, OCFS2) and volume manager
(clustered Logical Volume Manager, cLVM). For replication of your data, the High
Availability Extension also delivers DRBD (Distributed Replicated Block Device)
which you can use to mirror the data of a high availably service from the active
node of a cluster to its standby node.
Support for Virtualized Environments
SUSE Linux Enterprise High Availability Extension supports the mixed clustering
of both physical and virtual Linux servers. SUSE Linux Enterprise Server 11 ships
with Xen, an open source virtualization hypervisor. The cluster resource manager
in the High Availability Extension is able to recognize, monitor and manage services
running within virtual servers created with Xen, as well as services running in
physical servers. Guest systems can be managed as services by the cluster.
Resource Agents
SUSE Linux Enterprise High Availability Extension includes a huge number of
resource agents to manage resources such as Apache, IPv4, IPv6 and many more.
It also ships with resource agents for popular third party applications such as IBM
WebSphere Application Server. For a list of Open Cluster Framework (OCF) re-
source agents included with your product, refer to Chapter 18, HA OCF Agents
(page 201).
4High Availability Guide

User-friendly Administration
For easy conguration and administration, the High Availability Extension ships
with both a graphical user interface (like YaST and the Linux HA Management
Client) and a powerful unied command line interface. Both approaches provide
a single point of administration for effectively monitoring and administrating your
cluster. Learn how to do so in the following chapters.
1.2 Product Benets
The High Availability Extension allows you to congure up to 16 Linux servers into
a high-availability cluster (HA cluster), where resources can be dynamically switched
or moved to any server in the cluster. Resources can be congured to automatically
migrate in the event of a server failure, or they can be moved manually to troubleshoot
hardware or balance the workload.
The High Availability Extension provides high availability from commodity components.
Lower costs are obtained through the consolidation of applications and operations onto
a cluster. The High Availability Extension also allows you to centrally manage the
complete cluster and to adjust resources to meet changing workload requirements (thus,
manually “load balance” the cluster). Allowing clusters of more than two nodes also
provides savings by allowing several nodes to share a “hot spare”.
An equally important benet is the potential reduction of unplanned service outages as
well as planned outages for software and hardware maintenance and upgrades.
Reasons that you would want to implement a cluster include:
• Increased availability
• Improved performance
• Low cost of operation
• Scalability
• Disaster recovery
• Data protection
• Server consolidation
Conceptual Overview 5

• Storage consolidation
Shared disk fault tolerance can be obtained by implementing RAID on the shared disk
subsystem.
The following scenario illustrates some of the benets the High Availability Extension
can provide.
Example Cluster Scenario
Suppose you have congured a three-server cluster, with a Web server installed on
each of the three servers in the cluster. Each of the servers in the cluster hosts two Web
sites. All the data, graphics, and Web page content for each Web site are stored on a
shared disk subsystem connected to each of the servers in the cluster. The following
gure depicts how this setup might look.
Figure 1.1 Three-Server Cluster
During normal cluster operation, each server is in constant communication with the
other servers in the cluster and performs periodic polling of all registered resources to
detect failure.
Suppose Web Server 1 experiences hardware or software problems and the users de-
pending on Web Server 1 for Internet access, e-mail, and information lose their connec-
tions. The following gure shows how resources are moved when Web Server 1 fails.
6High Availability Guide

Figure 1.2 Three-Server Cluster after One Server Fails
Web Site A moves to Web Server 2 and Web Site B moves to Web Server 3. IP addresses
and certicates also move to Web Server 2 and Web Server 3.
When you congured the cluster, you decided where the Web sites hosted on each Web
server would go should a failure occur. In the previous example, you congured Web
Site A to move to Web Server 2 and Web Site B to move to Web Server 3. This way,
the workload once handled by Web Server 1 continues to be available and is evenly
distributed between any surviving cluster members.
When Web Server 1 failed, the High Availability Extension software
• Detected a failure and veried with STONITH that Web Server 1 was really dead
• Remounted the shared data directories that were formerly mounted on Web server
1 on Web Server 2 and Web Server 3.
• Restarted applications that were running on Web Server 1 on Web Server 2 and
Web Server 3
• Transferred IP addresses to Web Server 2 and Web Server 3
In this example, the failover process happened quickly and users regained access to
Web site information within seconds, and in most cases, without needing to log in again.
Now suppose the problems with Web Server 1 are resolved, and Web Server 1 is returned
to a normal operating state. Web Site A and Web Site B can either automatically fail
Conceptual Overview 7

back (move back) to Web Server 1, or they can stay where they are. This is dependent
on how you congured the resources for them. Migrating the services back to Web
Server 1 will incur some down-time, so the High Availability Extension also allows
you to defer the migration until a period when it will cause little or no service interrup-
tion. There are advantages and disadvantages to both alternatives.
The High Availability Extension also provides resource migration capabilities. You
can move applications, Web sites, etc. to other servers in your cluster as required for
system management.
For example, you could have manually moved Web Site A or Web Site B from Web
Server 1 to either of the other servers in the cluster. You might want to do this to upgrade
or perform scheduled maintenance on Web Server 1, or just to increase performance
or accessibility of the Web sites.
1.3 Cluster Congurations
Cluster congurations with the High Availability Extension might or might not include
a shared disk subsystem. The shared disk subsystem can be connected via high-speed
Fibre Channel cards, cables, and switches, or it can be congured to use iSCSI. If a
server fails, another designated server in the cluster automatically mounts the shared
disk directories that were previously mounted on the failed server. This gives network
users continuous access to the directories on the shared disk subsystem.
IMPORTANT: Shared Disk Subsystem with cLVM
When using a shared disk subsystem with cLVM, that subsystem must be con-
nected to all servers in the cluster from which it needs to be accessed.
Typical resources might include data, applications, and services. The following gure
shows how a typical Fibre Channel cluster conguration might look.
8High Availability Guide

Figure 1.3 Typical Fibre Channel Cluster Conguration
Although Fibre Channel provides the best performance, you can also congure your
cluster to use iSCSI. iSCSI is an alternative to Fibre Channel that can be used to create
a low-cost Storage Area Network (SAN). The following gure shows how a typical
iSCSI cluster conguration might look.
Conceptual Overview 9

Figure 1.4 Typical iSCSI Cluster Conguration
Although most clusters include a shared disk subsystem, it is also possible to create a
cluster without a share disk subsystem. The following gure shows how a cluster
without a shared disk subsystem might look.
Figure 1.5 Typical Cluster Conguration Without Shared Storage
10 High Availability Guide