Red Hat DEVICE-MAPPER MULTIPATH 5.2 Instruction Manual

Using Device-Mapper Multipath
Configuration and
Administration
5.2
DM_Multipath
ISBN: N/A
Publication date: July 2008

This book provides information on using the Device-Mapper Multipath feature of Red Hat
Enterprise Linux 5.2
Using Device-Mapper Multipath

Using Device-Mapper Multipath: Configuration and
Administration
Copyright ©2008 Red Hat, Inc.
Copyright ©2008 Red Hat, Inc. This material may only be distributed subject to the terms and conditions set forth in the
Open Publication License, V1.0 or later with the restrictions noted below (the latest version of the OPL is presently
available at http://www.opencontent.org/openpub/).
Distribution of substantively modified versions of this document is prohibited without the explicit permission of the
copyright holder.
Distribution of the work or derivative of the work in any standard (paper) book form for commercial purposes is
prohibited unless prior permission is obtained from the copyright holder.
Red Hat and the Red Hat "Shadow Man" logo are registered trademarks of Red Hat, Inc. in the United States and other
countries.
All other trademarks referenced herein are the property of their respective owners.
The GPG fingerprint of the [email protected] key is:
CA 20 86 86 2B D6 9D FC 65 F6 EC C4 21 91 80 CD DB 42 A6 0E
1801 Varsity Drive
Raleigh, NC 27606-2072
USA
Phone: [phone]
Phone: [phone]
Fax: [phone]
PO Box 13588
Research Triangle Park, NC 27709
USA

Using Device-Mapper Multipath

Preface ....................................................................................................................vii
1. Audience ...................................................................................................... vii
2. Related Documentation ................................................................................. vii
3. Feedback .................................................................................................... viii
4. Document Conventions ................................................................................ viii
1. Device Mapper Multipathing .................................................................................... 1
1. Overview of DM-Multipath ............................................................................... 1
2. Storage Array Support .................................................................................... 4
3. DM-Multipath Components ............................................................................. 4
4. DM-Multipath Setup Overview ......................................................................... 5
2. Multipath Devices ................................................................................................... 7
1. Multipath Device Identifiers ............................................................................. 7
2. Consistent Multipath Device Names in a Cluster .............................................. 7
3. Multipath Device Attributes ............................................................................. 8
4. Multipath Devices in Logical Volumes .............................................................. 8
3. Setting Up DM-Multipath ........................................................................................11
1. Setting Up DM-Multipath ................................................................................11
2. Ignoring Local Disks when Generating Multipath Devices ................................12
3. Adding Devices to the Multipathing Database .................................................14
4. The DM-Multipath Configuration File ......................................................................17
1. Configuration File Overview ...........................................................................17
2. Configuration File Blacklist .............................................................................18
2.1. Blacklisting by WWID .........................................................................18
2.2. Blacklisting By Device Name ...............................................................19
2.3. Blacklisting By Device Type ................................................................19
2.4. Blacklist Exceptions ............................................................................20
3. Configuration File Defaults .............................................................................20
4. Multipaths Device Configuration Attributes ......................................................23
5. Configuration File Devices .............................................................................25
5. DM-Multipath Administration and Troubleshooting ...................................................29
1. Multipath Command Output ...........................................................................29
2. Multipath Queries with multipath Command ....................................................30
3. Multipath Command Options ..........................................................................30
4. Determining Device Mapper Entries with the dmsetup Command .....................31
5. Troubleshooting with the multipathd Interactive Console ..................................31
Index .......................................................................................................................33
v

vi

Preface
This book describes the Device Mapper Multipath (DM-Multipath) feature of Red Hat Enterprise
Linux.
1. Audience
This book is intended to be used by system administrators managing systems running the Linux
operating system. It requires familiarity with Red Hat Enterprise Linux.
2. Related Documentation
For more information about using Red Hat Enterprise Linux, refer to the following resources:
•Red Hat Enterprise Linux Installation Guide — Provides information regarding installation of
Red Hat Enterprise Linux 5.
•Red Hat Enterprise Linux Deployment Guide — Provides information regarding the
deployment, configuration and administration of Red Hat Enterprise Linux 5.
For more information about Red Hat Cluster Suite for Red Hat Enterprise Linux 5, refer to the
following resources:
•Red Hat Cluster Suite Overview — Provides a high level overview of the Red Hat Cluster
Suite.
•Configuring and Managing a Red Hat Cluster — Provides information about installing,
configuring and managing Red Hat Cluster components.
•LVM Administrator's Guide: Configuration and Administration — Provides a description of the
Logical Volume Manager (LVM), including information on running LVM in a clustered
environment.
•Global File System: Configuration and Administration — Provides information about installing,
configuring, and maintaining Red Hat GFS (Red Hat Global File System).
•Using GNBD with Global File System — Provides an overview on using Global Network Block
Device (GNBD) with Red Hat GFS.
•Linux Virtual Server Administration — Provides information on configuring high-performance
systems and services with the Linux Virtual Server (LVS).
•Red Hat Cluster Suite Release Notes — Provides information about the current release of
Red Hat Cluster Suite.
Red Hat Cluster Suite documentation and other Red Hat documents are available in HTML,
PDF, and RPM versions on the Red Hat Enterprise Linux Documentation CD and online at
vii

http://www.redhat.com/docs/.
3. Feedback
If you spot a typo, or if you have thought of a way to make this manual better, we would love to
hear from you. Please submit a report in Bugzilla (http://bugzilla.redhat.com/bugzilla/) against
the component rh-cs.
Be sure to mention the manual's identifier:
Bugzilla component: Documentation-cluster
Book identifier: DM_Multipath(EN)-5.2 (2008-07-31T15:35)
By mentioning this manual's identifier, we know exactly which version of the guide you have.
If you have a suggestion for improving the documentation, try to be as specific as possible. If
you have found an error, please include the section number and some of the surrounding text
so we can find it easily.
4. Document Conventions
Certain words in this manual are represented in different fonts, styles, and weights. This
highlighting indicates that the word is part of a specific category. The categories include the
following:
Courier font
Courier font represents commands,file names and paths, and prompts .
When shown as below, it indicates computer output:
Desktop about.html logs paulwesterberg.png
Mail backupfiles mail reports
bold Courier font
Bold Courier font represents text that you are to type, such as: service jonas start
If you have to run a command as root, the root prompt (#) precedes the command:
#gconftool-2
italic Courier font
Italic Courier font represents a variable, such as an installation directory:
Preface
viii

/bin/
bold font
Bold font represents application programs and text found on a graphical interface.
When shown like this: OK , it indicates a button on a graphical application interface.
Additionally, the manual uses different strategies to draw your attention to pieces of information.
In order of how critical the information is to you, these items are marked as follows:
Note
A note is typically information that you need to understand the behavior of the
system.
Tip
A tip is typically an alternative way of performing a task.
Important
Important information is necessary, but possibly unexpected, such as a
configuration change that will not persist after a reboot.
Caution
A caution indicates an act that would violate your support agreement, such as
recompiling the kernel.
Warning
A warning indicates potential data loss, as may happen when tuning hardware
for maximum performance.
Document Conventions
ix

x

Device Mapper Multipathing
Device Mapper Multipathing (DM-Multipath) allows you to configure multiple I/O paths between
server nodes and storage arrays into a single device. These I/O paths are physical SAN
connections that can include separate cables, switches, and controllers. Multipathing
aggregates the I/O paths, creating a new device that consists of the aggregated paths.
1. Overview of DM-Multipath
DM-Multipath can be used to provide:
• Redundancy
DM-Multipath can provide failover in an active/passive configuration. In an active/passive
configuration, only half the paths are used at any time for I/O. If any element of an I/O path
(the cable, switch, or controller) fails, DM-Multipath switches to an alternate path.
• Improved Performance
DM-Multipath can be configured in active/active mode, where I/O is spread over the paths in a
round-robin fashion. In some configurations, DM-Multipath can detect loading on the I/O
paths and dynamically re-balance the load.
Figure 1.1, “Active/Passive Multipath Configuration with One RAID Device” shows an
active/passive configuration with two I/O paths from the server to a RAID device. There are 2
HBAs on the server, 2 SAN switches, and 2 RAID controllers.
Chapter 1.
1

Figure 1.1. Active/Passive Multipath Configuration with One RAID Device
In this configuration, there is one I/O path that goes through hba1, SAN1, and controller 1 and a
second I/O path that goes through hba2, SAN2, and controller2. There are many points of
possible failure in this configuration:
• HBA failure
• FC cable failure
• SAN switch failure
• Array controller port failure
With DM-Multipath configured, a failure at any of these points will cause DM-Multipath to switch
to the alternate I/O path.
Figure 1.2, “Active/Passive Multipath Configuration with Two RAID Devices” shows a more
complex active/passive configuration with 2 HBAs on the server, 2 SAN switches, and 2 RAID
devices with 2 RAID controllers each.
Chapter 1. Device Mapper Multipathing
2

Figure 1.2. Active/Passive Multipath Configuration with Two RAID Devices
In the example shown in Figure 1.2, “Active/Passive Multipath Configuration with Two RAID
Devices”, there are two I/O paths to each RAID device (just as there are in the example shown
in Figure 1.1, “Active/Passive Multipath Configuration with One RAID Device”). With
DM-Multipath configured, a failure at any of the points of the I/O path to either of the RAID
devices will cause DM-Multipath to switch to the alternate I/O path for that device.
Figure 1.3, “Active/Active Multipath Configuration with One RAID Device” shows an active/active
configuration with 2 HBAs on the server, 1 SAN switch, and 2 RAID controllers. There are four
I/O paths from the server to a storage device:
• hba1 to controller1
• hba1 to controller2
• hba2 to controller1
• hba2 to controller2
Overview of DM-Multipath
3

In this configuration, I/O can be spread among those four paths.
Figure 1.3. Active/Active Multipath Configuration with One RAID Device
2. Storage Array Support
By default, DM-Multipath includes support for the most common storage arrays that support
DM-Multipath. The supported devices can be found in the multipath.conf.defaults file. If
your storage array supports DM-Multipath and is not configured by default in this file, you may
need to add them to the DM-Multipath configuration file, multipath.conf. For information on
the DM-Multipath configuration file, see Chapter 4, The DM-Multipath Configuration File.
Some storage arrays require special handling of I/O errors and path switching. These require
separate hardware handler kernel modules.
3. DM-Multipath Components
Table 1.1, “DM-Multipath Components”. describes the components of DM-Multipath.
Component Description
dm-multipath kernel module Reroutes I/O and supports failover for paths and path groups.
Chapter 1. Device Mapper Multipathing
4

Component Description
multipath command Lists and configures multipath devices. Normally started up
with /etc/rc.sysinit, it can also be started up by a udev
program whenever a block device is added or it can be run by
the initramfs file system.
multipathd daemon Monitors paths; as paths fail and come back, it may initiate
path group switches. Provides for interactive changes to
multipath devices. This must be restarted for any changes to
the /etc/multipath.conf file.
kpartx command Creates device mapper devices for the partitions on a device
It is necessary to use this command for DOS-based partitions
with DM-MP. The kpartx is provided in its own package, but
the device-mapper-multipath package depends on it.
Table 1.1. DM-Multipath Components
4. DM-Multipath Setup Overview
DM-Multipath includes compiled-in default settings that are suitable for common multipath
configurations. Setting up DM-multipath is often a simple procedure.
The basic procedure for configuring your system with DM-Multipath is as follows:
1. Install device-mapper-multipath rpm.
2. Edit the multipath.conf configuration file:
• comment out the default blacklist
• change any of the existing defaults as needed
• save the configuration file
3. Start the multipath daemons.
4. Create the multipath device with the multipath command.
Detailed setup instructions for several example multipath configurations are provided in see
Chapter 3, Setting Up DM-Multipath.
DM-Multipath Setup Overview
5

6

Multipath Devices
Without DM-Multipath, each path from a server node to a storage controller is treated by the
system as a separate device, even when the I/O path connects the same server node to the
same storage controller. DM-Multipath provides a way of organizing the I/O paths logically, by
creating a single multipath device on top of the underlying devices.
1. Multipath Device Identifiers
Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally
unique and unchanging. By default, the name of a multipath device is set to its WWID.
Alternately, you can set the user_friendly_names option in the multipath configuration file,
which sets the alias to a node-unique name of the form mpathn.
For example, a node with two HBAs attached to a storage controller with two ports via a single
unzoned FC switch sees four devices: /dev/sda,/dev/sdb,dev/sdc, and /dev/sdd.
DM-Multipath creates a single device with a unique WWID that reroutes I/O to those four
underlying devices according to the multipath configuration. When the user_friendly_names
configuration option is set to yes, the name of the multipath device is set to mpathn.
When new devices are brought under the control of DM-Multipath, the new devices may be
seen in three different places under the /dev directory: /dev/mapper/mpathn,
/dev/mpath/mpathn, and /dev/dm-n.
• The devices in /dev/mapper are created early in the boot process. Use these devices to
access the multipathed devices, for example when creating logical volumes.
• The devices in /dev/mpath are provided as a convenience so that all multipathed devices
can be seen in one directory. These devices are created by the udev device manager and
may not be available on startup when the system needs to access them. Do not use these
devices for creating logical volumes or filesystems.
• Any devices of the form /dev/dm-nare for internal use only and should never be used.
For information on the multipath configuration defaults, including the user_friendly_names
configuration option, see Section 3, “Configuration File Defaults”.
You can also set the name of a multipath device to a name of your choosing by using the alias
option in the multipaths section of the multipath configuration file. For information on the
multipaths section of the multipath configuration file, see see Section 4, “Multipaths Device
Configuration Attributes”.
2. Consistent Multipath Device Names in a Cluster
When the user_friendly_names configuration option is set to yes, the name of the multipath
device is unique to a node, but it is not guaranteed to be the same on all nodes using the
Chapter 2.
7

multipath device. This should not cause any difficulties if you use LVM to create logical devices
from the multipath device, but if you require that your multipath device names be consistent in
every node in the cluster you perform one of the following procedures:
• Use the alias option in the multipaths section of the multipath configuration file to set the
name of the multipath device. The alias for the multipath device is consistent across all the
nodes in a cluster. For information on the multipaths section of the multipath configuration
file, see see Section 4, “Multipaths Device Configuration Attributes”.
• If you want the system-defined user-friendly names to be consistent across all nodes in the
cluster, set up all of the multipath devices on one machine. Then copy the
/var/lib/multipath/bindings file from that machine to all the other machines in the
cluster.
3. Multipath Device Attributes
In addition to the user_friendly_names and alias options, a multipath device has numerous
attributes. You can modify these attributes for a specific multipath device by creating an entry
for that device in the multipaths section of the multipath configuration file. For information on
the multipaths section of the multipath configuration file, see see Section 4, “Multipaths Device
Configuration Attributes”.
4. Multipath Devices in Logical Volumes
After creating multipath devices, you can use the multipath device names just as you would use
a physical device name when creating an LVM physical volume. For example, if
/dev/mapper/mpath0 is the name of a multipath device, the following command will mark
/dev/mapper/mpath0 as a physical volume.
pvcreate /dev/mapper/mpath0
You can use the resulting LVM physical device when you create an LVM volume group just as
you would use any other LVM physical device.
When you create an LVM logical volume that uses active/passive multipath arrays as the
underlying physical devices, you should include filters in the lvm.conf to exclude the disks that
underlie the multipath devices. This is because if the array automatically changes the active
path to the passive path when it receives I/O, multipath will failover and failback whenever LVM
scans the passive path if these devices are not filtered. For active/passive arrays that require a
command to make the passive path active, LVM prints a warning message when this occurs.
To filter all SCSI devices in the multipath configuration file (lvm.conf), include the following filter
in the devices section of the file.
filter = [ "r/disk/", "r/sd.*/", "a/.*/" ]
Chapter 2. Multipath Devices
8

Multipath Devices in Logical Volumes
9

10