Red Hat GLOBAL FILE SYSTEM 5.0 User manual

Global File System
5.0
Red Hat Global File
System
ISBN: N/A
Publication date:

This book provides information about installing, configuring, and maintaining Red Hat GFS;
(Red Hat Global File System).
Global File System

Global File System: Red Hat Global File System
Copyright ©You need to override this in your local ent file Red Hat, Inc.
Copyright ©You need to override this in your local ent file Red Hat Inc.. This material may only be distributed subject to
the terms and conditions set forth in the Open Publication License, V1.0 or later with the restrictions noted below (the
latest version of the OPL is presently available at http://www.opencontent.org/openpub/).
Distribution of substantively modified versions of this document is prohibited without the explicit permission of the
copyright holder.
Distribution of the work or derivative of the work in any standard (paper) book form for commercial purposes is
prohibited unless prior permission is obtained from the copyright holder.
Red Hat and the Red Hat "Shadow Man" logo are registered trademarks of Red Hat, Inc. in the United States and other
countries.
All other trademarks referenced herein are the property of their respective owners.
The GPG fingerprint of the [email protected] key is:
CA 20 86 86 2B D6 9D FC 65 F6 EC C4 21 91 80 CD DB 42 A6 0E
1801 Varsity Drive
Raleigh, NC 27606-2072
USA
Phone: [phone]
Phone: [phone]
Fax: [phone]
PO Box 13588
Research Triangle Park, NC 27709
USA

Global File System

Introduction ..............................................................................................................vii
1. Audience ...................................................................................................... vii
2. Related Documentation ................................................................................. vii
3. Document Conventions ................................................................................ viii
4. Send in Your Feedback ..................................................................................ix
5. Recommended References .............................................................................x
1. GFS Overview ....................................................................................................... 1
1. New and Changed Features ........................................................................... 1
2. Performance, Scalability, and Economy ........................................................... 2
2.1. Superior Performance and Scalability ................................................... 2
2.2. Performance, Scalability, Moderate Price .............................................. 3
2.3. Economy and Performance .................................................................. 4
3. GFS Functions ............................................................................................... 5
4. GFS Software Subsystems ............................................................................. 6
5. Before Setting Up GFS ................................................................................... 7
2. System Requirements ............................................................................................ 9
1. Platform Requirements ................................................................................... 9
2. Red Hat Cluster Suite ..................................................................................... 9
3. Fencing ......................................................................................................... 9
4. Fibre Channel Storage Network .....................................................................10
5. Fibre Channel Storage Devices ......................................................................10
6. Network Power Switches ...............................................................................11
7. Console Access ............................................................................................11
8. Installing GFS ...............................................................................................11
3. Getting Started .....................................................................................................13
1. Prerequisite Tasks ........................................................................................13
2. Initial Setup Tasks .........................................................................................13
4. Managing GFS .....................................................................................................15
1. Making a File System ....................................................................................15
2. Mounting a File System .................................................................................18
3. Unmounting a File System .............................................................................20
4. GFS Quota Management ...............................................................................21
4.1. Setting Quotas ...................................................................................21
4.2. Displaying Quota Limits and Usage .....................................................22
4.3. Synchronizing Quotas ........................................................................24
4.4. Disabling/Enabling Quota Enforcement ................................................25
4.5. Disabling/Enabling Quota Accounting ..................................................26
5. Growing a File System ..................................................................................27
6. Adding Journals to a File System ...................................................................29
7. Direct I/O ......................................................................................................31
7.1. O_DIRECT ...........................................................................................32
7.2. GFS File Attribute ...............................................................................32
7.3. GFS Directory Attribute .......................................................................33
8. Data Journaling .............................................................................................33
9. Configuring atime Updates ............................................................................35
9.1. Mount with noatime ...........................................................................35
v

9.2. Tune GFS atime Quantum .................................................................36
10. Suspending Activity on a File System ...........................................................37
11. Displaying Extended GFS Information and Statistics ......................................38
12. Repairing a File System ...............................................................................39
13. Context-Dependent Path Names ..................................................................40
Index .......................................................................................................................43
Global File System
vi

Introduction
Welcome to the Global File System Configuration and Administration document. This book
provides information about installing, configuring, and maintaining Red Hat GFS (Red Hat
Global File System). Red Hat GFS depends on the cluster infrastructure of Red Hat Cluster
Suite. For information about Red Hat Cluster Suite refer to Red Hat Cluster Suite Overview and
Configuring and Managing a Red Hat Cluster.
HTML and PDF versions of all the official Red Hat Enterprise Linux manuals and release notes
are available online at http://www.redhat.com/docs/.
1. Audience
This book is intended primarily for Linux system administrators who are familiar with the
following activities:
• Linux system administration procedures, including kernel configuration
• Installation and configuration of shared storage networks, such as Fibre Channel SANs
2. Related Documentation
For more information about using Red Hat Enterprise Linux, refer to the following resources:
•Red Hat Enterprise Linux Installation Guide — Provides information regarding installation of
Red Hat Enterprise Linux 5.
•Red Hat Enterprise Linux Deployment Guide — Provides information regarding the
deployment, configuration and administration of Red Hat Enterprise Linux 5.
For more information about Red Hat Cluster Suite for Red Hat Enterprise Linux 5, refer to the
following resources:
•Red Hat Cluster Suite Overview — Provides a high level overview of the Red Hat Cluster
Suite.
•Configuring and Managing a Red Hat Cluster — Provides information about installing,
configuring and managing Red Hat Cluster components.
•LVM Administrator's Guide: Configuration and Administration — Provides a description of the
Logical Volume Manager (LVM), including information on running LVM in a clustered
environment.
•Using GNBD with Global File System — Provides an overview on using Global Network Block
Device (GNBD) with Red Hat GFS.
vii

•Linux Virtual Server Administration — Provides information on configuring high-performance
systems and services with the Linux Virtual Server (LVS).
•Red Hat Cluster Suite Release Notes — Provides information about the current release of
Red Hat Cluster Suite.
3. Document Conventions
Certain words in this manual are represented in different fonts, styles, and weights. This
highlighting indicates that the word is part of a specific category. The categories include the
following:
Courier font
Courier font represents commands,file names and paths, and prompts .
When shown as below, it indicates computer output:
Desktop about.html logs paulwesterberg.png
Mail backupfiles mail reports
bold Courier font
Bold Courier font represents text that you are to type, such as: service jonas start
If you have to run a command as root, the root prompt (#) precedes the command:
#gconftool-2
italic Courier font
Italic Courier font represents a variable, such as an installation directory:
install_dir/bin/
bold font
Bold font represents application programs and text found on a graphical interface.
When shown like this: OK , it indicates a button on a graphical application interface.
Additionally, the manual uses different strategies to draw your attention to pieces of information.
In order of how critical the information is to you, these items are marked as follows:
Introduction
viii

Note
A note is typically information that you need to understand the behavior of the
system.
Tip
A tip is typically an alternative way of performing a task.
Important
Important information is necessary, but possibly unexpected, such as a
configuration change that will not persist after a reboot.
Caution
A caution indicates an act that would violate your support agreement, such as
recompiling the kernel.
Warning
A warning indicates potential data loss, as may happen when tuning hardware
for maximum performance.
4. Send in Your Feedback
If you spot a typo in the Global File System Configuration and Administration document or if you
have thought of a way to make this manual better, we would love to hear from you! Please
submit a report in Bugzilla (http://www.redhat.com/bugzilla) against the component rh-gfsg.
Be sure to mention the manual's identifier:
rh-gfsg
Send in Your Feedback
ix

If you mention this manual's identifier, we will know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If
you have found an error, please include the section number and some of the surrounding text
so we can find it easily.
5. Recommended References
For additional references about related topics, refer to the following table:
Topic Reference Comment
Shared Data Clustering and
File Systems Shared Data Clusters by Dilip M. Ranade. Wiley, 2002. Provides detailed technical information on cluster file
system and cluster
volume-manager design.
Storage Area Networks
(SANs) Designing Storage Area Networks: A Practical
Reference for Implementing
Fibre Channel and IP SANs,
Second Edition by Tom Clark.
Addison-Wesley, 2003.
Provides a concise summary
of Fibre Channel and IP SAN
Technology.
Building SANs with Brocade
Fabric Switches by C.
Beauchamp, J. Judd, and B.
Keo. Syngress, 2001.
Best practices for building
Fibre Channel SANs based
on the Brocade family of
switches, including core-edge
topology for large SAN
fabrics.
Building Storage Networks,
Second Edition by Marc
Farley. Osborne/McGraw-Hill,
2001.
Provides a comprehensive
overview reference on
storage networking
technologies.
Applications and High
Availability Blueprints for High Availability: Designing
Resilient Distributed Systems
by E. Marcus and H. Stern.
Wiley, 2000.
Provides a summary of best
practices in high availability.
Table 1. Recommended References Table
Introduction
x

GFS Overview
Red Hat GFS is a cluster file system that is available with Red Hat Cluster Suite. Red Hat GFS
nodes are configured and managed with Red Hat Cluster Suite configuration and management
tools. Red Hat GFS provides data sharing among GFS nodes in a Red Hat cluster. GFS
provides a single, consistent view of the file-system name space across the GFS nodes in a
Red Hat cluster. GFS allows applications to install and run without much knowledge of the
underlying storage infrastructure. GFS is fully compliant with the IEEE POSIX interface, allowing
applications to perform file operations as if they were running on a local file system. Also, GFS
provides features that are typically required in enterprise environments, such as quotas, multiple
journals, and multipath support.
GFS provides a versatile method of networking your storage according to the performance,
scalability, and economic needs of your storage environment. This chapter provides some very
basic, abbreviated information as background to help you understand GFS. It contains the
following sections:
•Section 1, “New and Changed Features”
•Section 2, “Performance, Scalability, and Economy”
•Section 3, “GFS Functions”
•Section 4, “GFS Software Subsystems”
•Section 5, “Before Setting Up GFS”
1. New and Changed Features
This section lists new and changed features included with the initial release of Red Hat
Enterprise Linux 5.
• GULM (Grand Unified Lock Manager) is not supported in Red Hat Enterprise Linux 5. If your
GFS file systems use the GULM lock manager, you must convert the file systems to use the
DLM lock manager. This is a two-part process.
• While running Red Hat Enterprise Linux 4, convert your GFS file systems to use the DLM
lock manager.
• Upgrade your operating system to Red Hat Enterprise Linux 5, converting the lock manager
to DLM when you do.
For information on upgrading to Red Hat Enterprise Linux 5 and converting GFS file systems
to use the DLM lock manager, see Configuring and Managing a Red Hat Cluster.
• Documentation for Red Hat Cluster Suite for Red Hat Enterprise Linux 5 has been expanded
Chapter 1.
1

and reorganized. For information on the available documents, see Section 2, “Related
Documentation”.
2. Performance, Scalability, and Economy
You can deploy GFS in a variety of configurations to suit your needs for performance,
scalability, and economy. For superior performance and scalability, you can deploy GFS in a
cluster that is connected directly to a SAN. For more economical needs, you can deploy GFS in
a cluster that is connected to a LAN with servers that use GNBD (Global Network Block Device).
The following sections provide examples of how GFS can be deployed to suit your needs for
performance, scalability, and economy:
•Section 2.1, “Superior Performance and Scalability”
•Section 2.2, “Performance, Scalability, Moderate Price”
•Section 2.3, “Economy and Performance”
Note
The deployment examples in this chapter reflect basic configurations; your needs
might require a combination of configurations shown in the examples.
2.1. Superior Performance and Scalability
You can obtain the highest shared-file performance when applications access storage directly.
The GFS SAN configuration in Figure 1.1, “GFS with a SAN” provides superior file performance
for shared files and file systems. Linux applications run directly on GFS nodes. Without file
protocols or storage servers to slow data access, performance is similar to individual Linux
servers with directly connected storage; yet, each GFS application node has equal access to all
data files. GFS supports over 300 GFS nodes.
Chapter 1. GFS Overview
2

Figure 1.1. GFS with a SAN
2.2. Performance, Scalability, Moderate Price
Multiple Linux client applications on a LAN can share the same SAN-based data as shown in
Figure 1.2, “GFS and GNBD with a SAN”. SAN block storage is presented to network clients as
block storage devices by GNBD servers. From the perspective of a client application, storage is
accessed as if it were directly attached to the server in which the application is running. Stored
data is actually on the SAN. Storage devices and data can be equally shared by network client
applications. File locking and sharing functions are handled by GFS for each network client.
Note
Clients implementing ext2 and ext3 file systems can be configured to access
their own dedicated slice of SAN storage.
Performance, Scalability, Moderate Price
3

Figure 1.2. GFS and GNBD with a SAN
2.3. Economy and Performance
Figure 1.3, “GFS and GNBD with Directly Connected Storage” shows how Linux client
applications can take advantage of an existing Ethernet topology to gain shared access to all
block storage devices. Client data files and file systems can be shared with GFS on each client.
Application failover can be fully automated with Red Hat Cluster Suite.
Chapter 1. GFS Overview
4

Figure 1.3. GFS and GNBD with Directly Connected Storage
3. GFS Functions
GFS is a native file system that interfaces directly with the VFS layer of the Linux kernel
file-system interface. GFS is a cluster file system that employs distributed metadata and multiple
journals for optimal operation in a cluster. Cluster management of GFS nodes is managed
through Red Hat Cluster Suite. Volume management is managed through CLVM (Cluster
Logical Volume Manager). For information about Red Hat Cluster Suite refer to Configuring and
Managing a Red Hat Cluster. For information about using CLVM, refer to LVM Administrator's
Guide.
Note
CLVM is a cluster-wide implementation of LVM, enabled by the CLVM daemon,
clvmd running in a Red Hat Cluster Suite cluster. The daemon makes it possible
to use LVM2 to manage logical volumes across a cluster, allowing all nodes in
the cluster to share the logical volumes.
GFS provides the following main functions:
GFS Functions
5

• Making a File System
• Mounting a File System
• Unmounting a File System
• GFS Quota Management
• Growing a File System
• Adding Journals to a File System
• Direct I/O
• Data Journaling
• Configuring atime Updates
• Suspending Activity on a File System
• Displaying Extended GFS Information and Statistics
• Repairing a File System
• Context-Dependent Path Names (CDPN)
4. GFS Software Subsystems
Table 1.1, “GFS Software Subsystem Components” summarizes the GFS Software subsystems
and their components.
Software Subsystem Components Description
GFS gfs.ko Kernel module that implements the GFS
file system and is loaded on GFS cluster
nodes.
gfs_fsck Command that repairs an unmounted
GFS file system.
gfs_grow Command that grows a mounted GFS file
system.
gfs_jadd Command that adds journals to a
mounted GFS file system.
gfs_mkfs Command that creates a GFS file system
on a storage device.
gfs_quota Command that manages quotas on a
mounted GFS file system.
gfs_tool Command that configures or tunes a GFS
file system. This command can also
gather a variety of information about the
Chapter 1. GFS Overview
6

Software Subsystem Components Description
file system.
lock_harness.ko Implements a pluggable lock module
interface for GFS that allows for a variety
of locking mechanisms to be used (for
example, the DLM lock module,
lock_dlm.ko).
lock_dlm.ko A lock module that implements DLM
locking for GFS. It plugs into the lock
harness, lock_harness.ko and
communicates with the DLM lock
manager in Red Hat Cluster Suite.
lock_nolock.ko A lock module for use when GFS is used
as a local file system only. It plugs into
the lock harness, lock_harness.ko and
provides local locking.
Table 1.1. GFS Software Subsystem Components
5. Before Setting Up GFS
Before you install and set up GFS, note the following key characteristics of your GFS file
systems:
Number of file systems
Determine how many GFS file systems to create initially. (More file systems can be added
later.)
File-system name
Determine a unique name for each file system. Each file-system name is required in the
form of a parameter variable. For example, this book uses file-system names gfs1 and
gfs2 in some example procedures.
Journals
Determine the number of journals for your GFS file systems. One journal is required for
each node that mounts a GFS file system. Make sure to account for additional journals
needed for future expansion.
GFS nodes
Determine which nodes in the Red Hat Cluster Suite will mount the GFS file systems.
GNBD server nodes
If you are using GNBD, determine how many GNBD server nodes are needed. Note the
Before Setting Up GFS
7

hostname and IP address of each GNBD server node for setting up GNBD clients later. For
information on using GNBD with GFS, see the Using GNBD with Global File System
document.
Storage devices and partitions
Determine the storage devices and partitions to be used for creating logical volumes (via
CLVM) in the file systems.
Chapter 1. GFS Overview
8

System Requirements
This chapter describes the system requirements for Red Hat GFS with Red Hat Enterprise Linux
5 and consists of the following sections:
•Section 1, “Platform Requirements”
•Section 2, “Red Hat Cluster Suite”
•Section 3, “Fencing”
•Section 4, “Fibre Channel Storage Network”
•Section 5, “Fibre Channel Storage Devices”
•Section 6, “Network Power Switches”
•Section 7, “Console Access”
1. Platform Requirements
Table 2.1, “Platform Requirements” shows the platform requirements for GFS.
Operating System Hardware Architecture RAM
Red Hat Enterprise Linux AS, ES, or
WS, Version 4 or later ia64, x86-64, x86
SMP supported
256 MB, minimum
Table 2.1. Platform Requirements
2. Red Hat Cluster Suite
Red Hat GFS runs with Red Hat Cluster Suite 4.0 or later. The Red Hat Cluster Suite software
must be installed on the cluster nodes before you can install and run Red Hat GFS.
Note
Red Hat Cluster Suite 4.0 and later provides the infrastructure for application
failover in the cluster and network communication among GFS nodes (and other
Red Hat Cluster Suite nodes).
3. Fencing
Chapter 2.
9

You must configure each GFS node in your Red Hat cluster for at least one form of fencing.
Fencing is configured and managed in Red Hat Cluster Suite. For more information about
fencing options, refer to Configuring and Managing a Red Hat Cluster.
4. Fibre Channel Storage Network
Table 2.2, “Fibre Channel Network Requirements” shows requirements for GFS nodes that are
to be connected to a Fibre Channel SAN.
Requirement Description
HBA (Host Bus Adapter) One HBA minimum per GFS node
Connection method Fibre Channel switch
Note: If an FC switch is used for fencing, you may want to
consider using Brocade, McData, or Vixel FC switches,
for which Red Hat Cluster Suite fencing agents exist.
Refer to Configuring and Managing a Red Hat Cluster for
more information about supported fencing agents.
Note: When a small number of nodes is used, it may be
possible to connect the nodes directly to ports on the
storage device.
Note: FC drivers may not work reliably with FC hubs.
Table 2.2. Fibre Channel Network Requirements
5. Fibre Channel Storage Devices
Table 2.3, “Fibre Channel Storage Device Requirements” shows requirements for Fibre Channel
devices that are to be connected to a GFS cluster.
Requirement Description
Device Type FC RAID array or JBOD
Note: Make sure that the devices can operate reliably
when heavily accessed simultaneously from multiple
initiators.
Note: Make sure that your GFS configuration does not
exceed the number of nodes an array or JBOD supports.
Size GFS is based on a 64-bit architecture, which can
theoretically accommodate an 8 EB file system. However,
the current supported maximum size of a GFS file system
is 25 TB. If your system requires GFS file systems larger
than 25 TB, contact your Red Hat service representative.
Chapter 2. System Requirements
10