Red Hat GLOBAL FILE SYSTEM 4.7 User manual

Global File System
Red Hat Global File
System
4.7
Global_File_System
ISBN: N/A
Publication date: July 2008

This book provides information about installing, configuring, and maintaining Red Hat GFS (Red
Hat Global File System) for Red Hat Enterprise Linux 4.7.
Global File System

Global File System: Red Hat Global File System
Copyright ©2008 Red Hat, Inc.
Copyright ©2008 Red Hat, Inc. This material may only be distributed subject to the terms and conditions set forth in the
Open Publication License, V1.0 or later with the restrictions noted below (the latest version of the OPL is presently
available at http://www.opencontent.org/openpub/).
Distribution of substantively modified versions of this document is prohibited without the explicit permission of the
copyright holder.
Distribution of the work or derivative of the work in any standard (paper) book form for commercial purposes is
prohibited unless prior permission is obtained from the copyright holder.
Red Hat and the Red Hat "Shadow Man" logo are registered trademarks of Red Hat, Inc. in the United States and other
countries.
All other trademarks referenced herein are the property of their respective owners.
The GPG fingerprint of the [email protected] key is:
CA 20 86 86 2B D6 9D FC 65 F6 EC C4 21 91 80 CD DB 42 A6 0E
1801 Varsity Drive
Raleigh, NC 27606-2072
USA
Phone: [phone]
Phone: [phone]
Fax: [phone]
PO Box 13588
Research Triangle Park, NC 27709
USA

Global File System

Introduction ..............................................................................................................vii
1. Audience ...................................................................................................... vii
2. Related Documentation ................................................................................. vii
3. Document Conventions ................................................................................ viii
4. Feedback .......................................................................................................x
5. Recommended References .............................................................................x
1. GFS Overview ....................................................................................................... 1
1. Performance, Scalability, and Economy ........................................................... 1
1.1. Superior Performance and Scalability ................................................... 2
1.2. Performance, Scalability, Moderate Price .............................................. 2
1.3. Economy and Performance .................................................................. 3
2. GFS Functions ............................................................................................... 4
3. GFS Software Subsystems ............................................................................. 5
4. Before Setting Up GFS ................................................................................... 6
2. System Requirements ............................................................................................ 9
1. Platform Requirements ................................................................................... 9
2. Red Hat Cluster Suite ..................................................................................... 9
3. Fencing ......................................................................................................... 9
4. Fibre Channel Storage Network .....................................................................10
5. Fibre Channel Storage Devices ......................................................................10
6. Network Power Switches ...............................................................................11
7. Console Access ............................................................................................11
8. Installing GFS ...............................................................................................11
3. Getting Started .....................................................................................................13
1. Prerequisite Tasks ........................................................................................13
2. Initial Setup Tasks .........................................................................................13
4. Managing GFS .....................................................................................................15
1. Making a File System ....................................................................................15
2. Mounting a File System .................................................................................18
3. Unmounting a File System .............................................................................20
4. GFS Quota Management ...............................................................................21
4.1. Setting Quotas ...................................................................................21
4.2. Displaying Quota Limits and Usage .....................................................23
4.3. Synchronizing Quotas ........................................................................24
4.4. Disabling/Enabling Quota Enforcement ................................................26
4.5. Disabling/Enabling Quota Accounting ..................................................27
5. Growing a File System ..................................................................................28
6. Adding Journals to a File System ...................................................................29
7. Direct I/O ......................................................................................................32
7.1. O_DIRECT ...........................................................................................32
7.2. GFS File Attribute ...............................................................................32
7.3. GFS Directory Attribute .......................................................................33
8. Data Journaling .............................................................................................34
9. Configuring atime Updates ............................................................................35
9.1. Mount with noatime ...........................................................................35
9.2. Tune GFS atime Quantum .................................................................36
v

10. Suspending Activity on a File System ...........................................................37
11. Displaying Extended GFS Information and Statistics ......................................38
12. Repairing a File System ...............................................................................39
13. Context-Dependent Path Names ..................................................................40
A. Upgrading GFS ....................................................................................................43
Index .......................................................................................................................47
Global File System
vi

Introduction
Welcome to the Global File System Configuration and Administration document. This book
provides information about installing, configuring, and maintaining Red Hat GFS (Red Hat
Global File System). Red Hat GFS depends on the cluster infrastructure of Red Hat Cluster
Suite. For information about Red Hat Cluster Suite refer to Red Hat Cluster Suite Overview and
Configuring and Managing a Red Hat Cluster.
HTML and PDF versions of all the official Red Hat Enterprise Linux manuals and release notes
are available online at http://www.redhat.com/docs/.
1. Audience
This book is intended primarily for Linux system administrators who are familiar with the
following activities:
• Linux system administration procedures, including kernel configuration
• Installation and configuration of shared storage networks, such as Fibre Channel SANs
2. Related Documentation
For more information about using Red Hat Enterprise Linux, refer to the following resources:
•Red Hat Enterprise Linux Installation Guide — Provides information regarding installation of
Red Hat Enterprise Linux.
•Red Hat Enterprise Linux Introduction to System Administration — Provides introductory
information for new Red Hat Enterprise Linux system administrators.
•Red Hat Enterprise Linux System Administration Guide — Provides more detailed information
about configuring Red Hat Enterprise Linux to suit your particular needs as a user.
•Red Hat Enterprise Linux Reference Guide — Provides detailed information suited for more
experienced users to reference when needed, as opposed to step-by-step instructions.
•Red Hat Enterprise Linux Security Guide — Details the planning and the tools involved in
creating a secured computing environment for the data center, workplace, and home.
For more information about Red Hat Cluster Suite for Red Hat Enterprise Linux, refer to the
following resources:
•Red Hat Cluster Suite Overview — Provides a high level overview of the Red Hat Cluster
Suite.
•Configuring and Managing a Red Hat Cluster — Provides information about installing,
vii

configuring and managing Red Hat Cluster components.
•LVM Administrator's Guide: Configuration and Administration — Provides a description of the
Logical Volume Manager (LVM), including information on running LVM in a clustered
environment.
•Using GNBD with Global File System — Provides an overview on using Global Network Block
Device (GNBD) with Red Hat GFS.
•Using Device-Mapper Multipath — Provides information about using the Device-Mapper
Multipath feature of Red Hat Enterprise Linux.
•Linux Virtual Server Administration — Provides information on configuring high-performance
systems and services with the Linux Virtual Server (LVS).
•Red Hat Cluster Suite Release Notes — Provides information about the current release of
Red Hat Cluster Suite.
Red Hat Cluster Suite documentation and other Red Hat documents are available in HTML and
PDF versions online at the following location:
http://www.redhat.com/docs
3. Document Conventions
Certain words in this manual are represented in different fonts, styles, and weights. This
highlighting indicates that the word is part of a specific category. The categories include the
following:
Courier font
Courier font represents commands,file names and paths, and prompts .
When shown as below, it indicates computer output:
Desktop about.html logs paulwesterberg.png
Mail backupfiles mail reports
bold Courier font
Bold Courier font represents text that you are to type, such as: service jonas start
If you have to run a command as root, the root prompt (#) precedes the command:
#gconftool-2
Introduction
viii

italic Courier font
Italic Courier font represents a variable, such as an installation directory:
install_dir/bin/
bold font
Bold font represents application programs and text found on a graphical interface.
When shown like this: OK , it indicates a button on a graphical application interface.
Additionally, the manual uses different strategies to draw your attention to pieces of information.
In order of how critical the information is to you, these items are marked as follows:
Note
A note is typically information that you need to understand the behavior of the
system.
Tip
A tip is typically an alternative way of performing a task.
Important
Important information is necessary, but possibly unexpected, such as a
configuration change that will not persist after a reboot.
Caution
A caution indicates an act that would violate your support agreement, such as
recompiling the kernel.
Warning
A warning indicates potential data loss, as may happen when tuning hardware
for maximum performance.
Feedback
ix

4. Feedback
If you spot a typo, or if you have thought of a way to make this manual better, we would love to
hear from you. Please submit a report in Bugzilla (http://bugzilla.redhat.com/bugzilla/) against
the component rh-cs.
Be sure to mention the manual's identifier:
rh-gfs(EN)-4.7 (2008-07-24T15:10)
By mentioning this manual's identifier, we know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If
you have found an error, please include the section number and some of the surrounding text
so we can find it easily.
5. Recommended References
For additional references about related topics, refer to the following table:
Topic Reference Comment
Shared Data Clustering and
File Systems Shared Data Clusters by Dilip M. Ranade. Wiley, 2002. Provides detailed technical information on cluster file
system and cluster
volume-manager design.
Storage Area Networks
(SANs) Designing Storage Area Networks: A Practical
Reference for Implementing
Fibre Channel and IP SANs,
Second Edition by Tom Clark.
Addison-Wesley, 2003.
Provides a concise summary
of Fibre Channel and IP SAN
Technology.
Building SANs with Brocade
Fabric Switches by C.
Beauchamp, J. Judd, and B.
Keo. Syngress, 2001.
Best practices for building
Fibre Channel SANs based
on the Brocade family of
switches, including core-edge
topology for large SAN
fabrics.
Building Storage Networks,
Second Edition by Marc
Farley. Osborne/McGraw-Hill,
2001.
Provides a comprehensive
overview reference on
storage networking
technologies.
Applications and High
Availability Blueprints for High Availability: Designing
Resilient Distributed Systems
Provides a summary of best
practices in high availability.
Introduction
x

Topic Reference Comment
by E. Marcus and H. Stern.
Wiley, 2000.
Table 1. Recommended References Table
Recommended References
xi

xii

GFS Overview
Red Hat GFS is a cluster file system that is available with Red Hat Cluster Suite. Red Hat GFS
nodes are configured and managed with Red Hat Cluster Suite configuration and management
tools. Red Hat GFS provides data sharing among GFS nodes in a Red Hat cluster. GFS
provides a single, consistent view of the file-system name space across the GFS nodes in a
Red Hat cluster. GFS allows applications to install and run without much knowledge of the
underlying storage infrastructure. GFS is fully compliant with the IEEE POSIX interface, allowing
applications to perform file operations as if they were running on a local file system. Also, GFS
provides features that are typically required in enterprise environments, such as quotas, multiple
journals, and multipath support.
GFS provides a versatile method of networking your storage according to the performance,
scalability, and economic needs of your storage environment. This chapter provides some very
basic, abbreviated information as background to help you understand GFS. It contains the
following sections:
•Section 1, “Performance, Scalability, and Economy”
•Section 2, “GFS Functions”
•Section 3, “GFS Software Subsystems”
•Section 4, “Before Setting Up GFS”
1. Performance, Scalability, and Economy
You can deploy GFS in a variety of configurations to suit your needs for performance,
scalability, and economy. For superior performance and scalability, you can deploy GFS in a
cluster that is connected directly to a SAN. For more economical needs, you can deploy GFS in
a cluster that is connected to a LAN with servers that use GNBD (Global Network Block Device).
The following sections provide examples of how GFS can be deployed to suit your needs for
performance, scalability, and economy:
•Section 1.1, “Superior Performance and Scalability”
•Section 1.2, “Performance, Scalability, Moderate Price”
•Section 1.3, “Economy and Performance”
Note
The deployment examples in this chapter reflect basic configurations; your needs
Chapter 1.
1

might require a combination of configurations shown in the examples.
1.1. Superior Performance and Scalability
You can obtain the highest shared-file performance when applications access storage directly.
The GFS SAN configuration in Figure 1.1, “GFS with a SAN” provides superior file performance
for shared files and file systems. Linux applications run directly on GFS nodes. Without file
protocols or storage servers to slow data access, performance is similar to individual Linux
servers with directly connected storage; yet, each GFS application node has equal access to all
data files. GFS supports over 300 GFS nodes.
Figure 1.1. GFS with a SAN
1.2. Performance, Scalability, Moderate Price
Multiple Linux client applications on a LAN can share the same SAN-based data as shown in
Figure 1.2, “GFS and GNBD with a SAN”. SAN block storage is presented to network clients as
block storage devices by GNBD servers. From the perspective of a client application, storage is
accessed as if it were directly attached to the server in which the application is running. Stored
data is actually on the SAN. Storage devices and data can be equally shared by network client
applications. File locking and sharing functions are handled by GFS for each network client.
Chapter 1. GFS Overview
2

Note
Clients implementing ext2 and ext3 file systems can be configured to access
their own dedicated slice of SAN storage.
Figure 1.2. GFS and GNBD with a SAN
1.3. Economy and Performance
Figure 1.3, “GFS and GNBD with Directly Connected Storage” shows how Linux client
applications can take advantage of an existing Ethernet topology to gain shared access to all
block storage devices. Client data files and file systems can be shared with GFS on each client.
Application failover can be fully automated with Red Hat Cluster Suite.
Economy and Performance
3

Figure 1.3. GFS and GNBD with Directly Connected Storage
2. GFS Functions
GFS is a native file system that interfaces directly with the VFS layer of the Linux kernel
file-system interface. GFS is a cluster file system that employs distributed metadata and multiple
journals for optimal operation in a cluster. Cluster management of GFS nodes is managed
through Red Hat Cluster Suite. Volume management is managed through CLVM (Cluster
Logical Volume Manager). For information about Red Hat Cluster Suite refer to Configuring and
Managing a Red Hat Cluster. For information about using CLVM, refer to LVM Administrator's
Guide.
Note
CLVM is a cluster-wide implementation of LVM, enabled by the CLVM daemon,
clvmd running in a Red Hat Cluster Suite cluster. The daemon makes it possible
to use LVM2 to manage logical volumes across a cluster, allowing all nodes in
the cluster to share the logical volumes.
GFS provides the following main functions:
Chapter 1. GFS Overview
4

• Making a File System
• Mounting a File System
• Unmounting a File System
• GFS Quota Management
• Growing a File System
• Adding Journals to a File System
• Direct I/O
• Data Journaling
• Configuring atime Updates
• Suspending Activity on a File System
• Displaying Extended GFS Information and Statistics
• Repairing a File System
• Context-Dependent Path Names (CDPN)
3. GFS Software Subsystems
Table 1.1, “GFS Software Subsystem Components” summarizes the GFS Software subsystems
and their components.
Software Subsystem Components Description
GFS gfs.ko Kernel module that implements the GFS
file system and is loaded on GFS cluster
nodes.
gfs_fsck Command that repairs an unmounted
GFS file system.
gfs_grow Command that grows a mounted GFS file
system.
gfs_jadd Command that adds journals to a
mounted GFS file system.
gfs_mkfs Command that creates a GFS file system
on a storage device.
gfs_quota Command that manages quotas on a
mounted GFS file system.
gfs_tool Command that configures or tunes a GFS
file system. This command can also
GFS Software Subsystems
5

Software Subsystem Components Description
gather a variety of information about the
file system.
lock_harness.ko Implements a pluggable lock module
interface for GFS that allows for a variety
of locking mechanisms to be used (for
example, the DLM lock module,
lock_dlm.ko).
lock_dlm.ko A lock module that implements DLM
locking for GFS. It plugs into the lock
harness, lock_harness.ko and
communicates with the DLM lock
manager in Red Hat Cluster Suite.
lock_gulm.ko A lock module that implements GULM
locking for GFS. It plugs into the lock
harness, lock_harness.ko and
communicates with the GULM lock
manager in Red Hat Cluster Suite.
lock_nolock.ko A lock module for use when GFS is used
as a local file system only. It plugs into
the lock harness, lock_harness.ko and
provides local locking.
Table 1.1. GFS Software Subsystem Components
4. Before Setting Up GFS
Before you install and set up GFS, note the following key characteristics of your GFS file
systems:
Number of file systems
Determine how many GFS file systems to create initially. (More file systems can be added
later.)
File-system name
Determine a unique name for each file system. Each file-system name is required in the
form of a parameter variable. For example, this book uses file-system names gfs1 and
gfs2 in some example procedures.
Journals
Determine the number of journals for your GFS file systems. One journal is required for
each node that mounts a GFS file system. Make sure to account for additional journals
needed for future expansion.
Chapter 1. GFS Overview
6

GFS nodes
Determine which nodes in the Red Hat Cluster Suite will mount the GFS file systems.
GNBD server nodes
If you are using GNBD, determine how many GNBD server nodes are needed. Note the
hostname and IP address of each GNBD server node for setting up GNBD clients later. For
information on using GNBD with GFS, see the Using GNBD with Global File System
document.
Storage devices and partitions
Determine the storage devices and partitions to be used for creating logical volumes (via
CLVM) in the file systems.
Before Setting Up GFS
7

8