Dell DX6000G User manual

DX Storage Cluster File Server (CFS)
Setup and Configuration Guide
Version 2.6

DX Storage Cluster File Server (CFS) Setup and Configuration
Guide: Version 2.6
Copyright © 2011 Caringo, Inc.
All Rights Reserved.
No part of this document may be reproduced, transmitted, or transcribed without the written consent of Caringo, Inc.

Copyright © 2011 Caringo, Inc.
All rights reserved iii Version 2.6 February 2011
Table of Contents
1. Welcome to DX Storage CFS ............................................................................................... 1
1.1. Overview of DX Storage CFS .................................................................................... 1
1.2. About this Document ................................................................................................. 1
1.2.1. Audience ....................................................................................................... 1
1.2.2. Scope ............................................................................................................ 1
2. Installation and Guided Configuration .................................................................................... 2
2.1. Requirements ........................................................................................................... 2
2.1.1. Operating System ........................................................................................... 2
2.1.2. Recommended Network Infrastructure ............................................................. 2
2.1.3. Protecting Local Disk Resources ..................................................................... 2
2.2. Installation Steps ...................................................................................................... 3
2.2.1. Installing the DX Storage CFS Server .............................................................. 3
2.2.2. Cluster Name Space (CNS) ............................................................................ 3
2.2.3. Configuring DX Storage CFS and its Mountpoints ............................................ 6
2.2.4. Post-configuration File System Options ............................................................ 8
2.3. Deleting a Mountpoint ............................................................................................... 9
2.4. Upgrading CFS ......................................................................................................... 9
2.4.1. Upgrading from Versions 1.x ........................................................................... 9
2.4.2. Upgrading from Version 2.0 ............................................................................ 9
2.4.3. Upgrading from Version 2.5 .......................................................................... 10
2.5. Removing CFS and CNS ......................................................................................... 10
2.6. Restoring CFS ........................................................................................................ 11
3. Running and Managing DX Storage CFS ............................................................................ 12
3.1. Starting DX Storage CFS and CNS .......................................................................... 12
3.2. DX Storage CFS and CNS Shutdown ...................................................................... 12
3.3. Timescapes ............................................................................................................ 13
3.3.1. Deleting a Timescape Mount ......................................................................... 14
3.3.2. Mounting a Timescape at an Alternate Time .................................................. 14
3.4. File Revisions and DX Storage Deletion ................................................................... 14
3.5. Support and Management Utilities ............................................................................ 14
3.5.1. cns-reporter .................................................................................................. 14
3.5.2. cfs-grab ....................................................................................................... 15
3.5.3. Configuration Modifications with cns-admin and cfs-admin ............................... 15
3.5.4. cfs-loglevel ................................................................................................... 16
3.6. DX Storage Metadata and Policies ........................................................................... 16
3.6.1. Custom Metadata ......................................................................................... 17
3.6.2. DX Storage Lifecycle Policies ........................................................................ 18
A. Temp and Logging Space Configuration ............................................................................. 20

Copyright © 2011 Caringo, Inc.
All rights reserved 1 Version 2.6 February 2011
Chapter 1. Welcome to DX Storage CFS
1.1. Overview of DX Storage CFS
DX Storage Cluster File Server (CFS) is a native DX Storage application that presents a DX Storage
cluster as a standard Linux file system, enabling the use of DX Storage by software that uses a file
system. You can also layer a network file service over DX Storage CFS using Samba or NFS. CFS
separates file data stored directly in DX Storage from file metadata stored in DX Storage via the
Cluster Name Space (CNS).
To enhance performance, the CFS process uses the local file system as a spool/cache for files. This
local cache is where user files are written before they get spooled to DX Storage and where they will
be accessed on subsequent reads if locally available. The cache/spool size is managed by evicting
files based on a least recently used basis when space needs to be reclaimed.
The Cluster Name Space (CNS) also uses DX Storage to store name space records for the file
system structure and metadata (owner, permissions, etc.), fulfilling the role typically performed by
a database but without the scalability and management concerns. CNS uses a RAM-based cache
to deliver metadata objects that are in high demand. Name space modifications and updates are
continuously flushed to DX Storage via a background process.
Recoverability for both CFS and CNS are enhanced by using the Linux machine's local hard disk or
shared storage for journaling of in process events. In the event of a power failure or system crash,
the processes are restarted automatically and any changes not recorded in the namespace or
spooled to DX Storage are replayed from the journal to prevent data loss. A process monitor runs
at all times to restart a CFS or CNS process in the unlikely event of a process crash. All process
monitoring and recovery activities appear in the syslog for management purposes.
1.2. About this Document
1.2.1. Audience
This document is intended for people in the following roles.
1. Storage system administrators
2. Network administrators
3. Technical architects
Throughout this document, the storage system administrator and network administrator roles will be
referred to as the administrator. The administrators are normally responsible for allocating storage,
managing capacity, monitoring storage system health, replacing malfunctioning hardware, and
adding additional capacity when needed.
1.2.2. Scope
This document covers the steps needed to deploy and configure DX Storage CFS. The reader is
expected to have a background in networking, basic knowledge of Linux operating systems, and
optional experience with NFS, CIFS, SAMBA or other file server clients.

Copyright © 2011 Caringo, Inc.
All rights reserved 2 Version 2.6 February 2011
Chapter 2. Installation and Guided Configuration
2.1. Requirements
2.1.1. Operating System
DX Storage CFS has been developed and tested with 64-bit Red Hat Enterprise Linux 6.0;
other RHEL versions or Linux distributions are not currently supported. Subsequent installation
instructions will assume a pre-installed RHEL environment with either internet connectivity or an
alternately configured RHEL yum repository for use in installing required 3rd party packages.
Warning
The recently added Red Hat 'hugepages' feature has been found to be incompatible with
CFS and other applications, often resulting in kernel hangs particularly under high stress
scenarios. As a result it is highly recommended that huge pages be disabled as follows
prior to installing and running CFS:
echo 'never' > /proc/sys/kernel/mm/redhat_transparent_hugepage/enabled
If the feature is not disabled, both CFS and CNS will log a warning at startup as an
additional reminder. For the changes to remain in effect with the next reboot, you must
add 'transparent_hugepage=never' to the existing kernel parameters in the /boot/grub/
menu.lst file prior to rebooting the CFS server.
Note
For optimal performance, it is highly recommended that the 'deadline' I/O scheduler be
utilized on the installed CFS server. The following command demonstrates how to enable
the deadline scheduler on an 'sda' disk:
echo deadline | sudo tee /sys/block/sda/queue/scheduler
For the changes to remain in effect with the next reboot, you must add 'elevator=deadline'
to the existing kernel parameters in the /boot/grub/menu.lst file prior to rebooting the CFS
server.
2.1.2. Recommended Network Infrastructure
A configured Network Time Protocol (NTP) server is strongly recommended to provide clock
synchronization services to DX Storage CFS, the name space server and the DX Storage cluster. A
common sense of time is especially important between CFS servers and the name space server if
the latter is configured remotely. To install a functional NTP configuration on RHEL, run the following
command as the root user:
# yum install ntp
Gigabit Ethernet or better is the recommended connection speed between DX Storage CFS and DX
Storage nodes.
2.1.3. Protecting Local Disk Resources
Red Hat 6 does not guarantee immediate write of data to disk with their default ext4 file system.
In order to ensure data security in the event of a system or power failure, it is therefore highly
recommended that administrators add the 'nodelalloc' mount option to several critical local

Copyright © 2011 Caringo, Inc.
All rights reserved 3 Version 2.6 February 2011
directories critical to recovering from a failure. Specifically, the following 3 directories should be
mounted with 'nodelalloc':
mount /dev/sdxx /var/lib/dht -o rw,nodelalloc,user_xattr
mount /dev/sdxx /var/cache/cns -o rw,nodelalloc,user_xattr
mount /dev/sdxx /var/spool/cfs -o rw,nodelalloc,user_xattr
2.2. Installation Steps
The following steps are necessary to install DX Storage CFS.
• Install DX Storage CFS
• Configure DX Storage CNS
• Configure DX Storage CFS
On an Red Hat Enterprise Linux system with a local name space, the following commands must be
run as root to install and setup the necessary components:
$ sudo su -
# ./installCFS.sh
# cns-admin
# cfs-admin mkfs --add
# /etc/init.d/caringo-cfs start
There are many valid ways to install each of these components and services. This is not a
comprehensive guide to NTP or other 3rd party package installation. For complete instructions on
installing, please see the documentation included with the operating system, from the software
project web sites, and other materials. The following is a list of examples with steps highlighted for
your reference. Please make sure that you understand the details of configuring these packages
and customize these steps to fit your IT environment.
For information on configuring CFS for High Availability, please reference the Linux-HA project.
2.2.1. Installing the DX Storage CFS Server
DX Storage CFS distribution is available as a Red Hat rpm package that is installed with a shell
script. The package and its dependencies must be installed as the 'root' user with the following
steps:
1. Install caringo-cfs package by running the following:
$ sudo su
# ./installCFS.sh
2.2.2. Cluster Name Space (CNS)
The file system structure, metadata and DX Storage UUIDs are stored in a journaling name space
that persists file metadata into DX Storage automatically.
After installing the DX Storage CFS server, you will need to configure a new name space. This is
only required once at installation time, subsequent CFS share definitions will not require a new
name space. You can initiate the configuration process by running the following command as the
root user:

Copyright © 2011 Caringo, Inc.
All rights reserved 4 Version 2.6 February 2011
# cns-admin
Note: Please make sure DX Storage is online and reachable prior to configuring the new name
space. cns-admin will prompt for the following values:
1. Log facility. Enter the logging facility to use. The default value is syslog.
a. Log filename. If the file logging facility was selected, enter the filename and location of
the logfile. By default the installer will use a /var/log/caringo/cns.log file. An additional file
in /var/log/caringo/cnsaudit.log will log the UUIDs for all successful DX Storage deletes for
audit purposes. With file logging, default log rotation for both files will be configured to keep
up to 8 log files, rotating weekly or at a max file size of 512 MB.
b. Syslog facility. If the syslog option was selected, enter the facility to log to. The default
value is local4. Configuration of a remote syslog host must be done in the syslogd
configuration file.
2. Log level. Enter the minimum level of log messages to record. The default value is info. Each
log level includes the following:
• info: errors, warnings and information messages like system start up and stop
• verbose: errors, warnings, and info messages plus high-level operational functions
• debug: errors, warnings, info and verbose messages plus lower level operational functions
• trace: all possible log messages
Note
Debug or trace logging are not recommended unless explicitly requested as part
of a support investigation due to the amount of disk space consumed. If debug
logging is required for any length of time, administrators must monitor space
availability carefully and are strongly encouraged to allocate a dedicated partition
for logging to protect other system services.
3. CNS host: The location of the name space server for the name space you are configuring. The
default is localhost (127.0.0.1), which will configure a name space on the local server. Enter an
external IP address if using a remote server separate from the CFS server for the name space.
You may also enter a 0.0.0.0 address if the name space should be created locally but will need
to connect to CFS shares on both the local server and remote servers.
4. DX Storage: use Zeroconf. This option allows you to specify whether or not Zeroconf should
be used to discover the list of nodes in the DX Storage cluster. The CNS server must be in the
same subnet with DX Storage in order to use Zeroconf. CFS 2.6 is compatible with DX Storage
versions 3.0.5 and later.
a. If 'No' is selected:
Cluster primary node address: Enter the IP address or hostname for a DX Storage node
in the target cluster. The target cluster must be the same as the one configured for all CFS
mounts.
Primary node SCSP port: Enter the port the DX Storage node uses for SCSP
communications. The default is '80'.

Copyright © 2011 Caringo, Inc.
All rights reserved 5 Version 2.6 February 2011
Cluster secondary node address: Enter the IP address or hostname for a second DX
Storage node in the target cluster for redundancy. The target cluster must be the same as
the one configured for all CFS mounts.
Secondary node SCSP port: Enter the port the secondary DX Storage node uses for
SCSP communications. The default is '80'.
b. If 'Yes' is selected:
Cluster name Enter the name of the DX Storage cluster. This must match the value of
the DX Storage "cluster" parameter in the node.cfg file and be the same target cluster as
the one configured for all CFS mounts. Note that if using Zeroconf for discovering the list
of nodes and using a cluster name with periods in it, the periods should be replaced with
underscores to ensure proper node discovery.
5. Do you want me to email you the Root UUID for CNS? Whether or not you would like the
generated root UUID for the name space in DX Storage emailed to you. The root id is critical
to recovering the name space in the event of a name space server failure and should be saved
in a safe location. The email function will only work with SMTP servers that do not require
authentication.
a. If you answer yes, you will be prompted to enter email server and address information:
i. SMTP email server: The SMTP email server that should be used to send the email.
Ex: mail.host.com. A mail account of 'admin' at the specified mail server is assumed
(i.e. [email protected]) as the sending account. Mail servers without an admin
account will not be able to send the root uuid email.
ii. Your email address: The email address to which the root id should be sent. Ex:
[email protected]
Note
Administrators who experience email send errors should consult /var/log/
caringo/smtp.log.
b. If you answer no, the utility will confirm that a new root id for the name space has been
created with a message similar to the following:
Generated New Root Anchor [f223011d1fa56f67cf79a87a5de45901]
Warning
If the configuration utility detects that a root id already exists it will ask you if you would
like to create a new one. Answering 'yes' to this question will create a new root and delete
all local CFS state information (spool, cache, etc.), as you will not be able to access the
information in the shares once the name space root has changed. The configuration files
for all shares will remain intact. You must stop CFS and CNS prior to attempting to create
a new root id.
The script will ask if you would like CNS to be started once you confirm the root id. 'Yes' is
recommended as a functioning namespace is required for CFS configuration. The created
configuration file will be located in: /opt/caringo/fabric/etc/cns.conf. Manual editing of this file is not
recommended unless under direct advice from your technical support contact.

Copyright © 2011 Caringo, Inc.
All rights reserved 6 Version 2.6 February 2011
Note
The name space must remain online at all times to ensure CFS can obtain the needed
file system information. If CFS cannot connect to the name space, it will start throwing
permission denied errors and block new file writes.
2.2.3. Configuring DX Storage CFS and its Mountpoints
After installing the DX Storage CFS server and creating the name space via cns-admin, you will
need to configure a mountpoint, logging and DX Storage cluster information. You can initiate the
configuration process by running the following command as the root user:
# cfs-admin mkfs --add
This will prompt you for the minimum required parameters needed to run DX Storage CFS. Most
fields have a default value with the exception of fields that require your input which will be blank. A
description of each of the configuration prompts is included below. To stop the configuration process
at any time, use Ctrl-C; no configuration files will be created until the utility completes successfully.
The utility can be run multiple times if you are creating multiple DX Storage CFS mountpoints. Each
mountpoint must have its own unique name, configuration file and spooler partition. All created
configuration files will be located in: /opt/caringo/fabric/etc/<mountid>.conf. Manual editing of this file
is not recommended unless under direct advice from your technical support contact.
1. ID to be used for this mount. Enter a name for the mountpoint. It must be unique on each
server and consist of letters and numbers without punctuation or spaces. For example:
"acctShare" or "archiveFS".
2. Mount directory path. Enter the location of the mountpoint. By default, the utility will use
the previously provided mount ID in the /mnt directory. Spaces in mountpoint paths are not
supported.
3. Spooler directory path. Enter the location where the spooler/cache directory will reside. The
location must be unique for each distinct read/write mount ID or the filesystem mount will fail.
By default, the installer will use the previously provided mount ID in the /var/spool/cfs directory.
Extended attributes must be enabled on the specified spooler partition to store the lifepoints
and custom and system metadata associated with each unspooled item.
Warning
A dedicated spooler partition is required for each share to ensure space monitoring is
accurate and the local cache is properly maintained.
4. Mount on boot. This determines whether or not this DX Storage CFS file system should be
mounted automatically when the server reboots. The default and recommended value is Yes.
5. Mount read-only. This value determines whether or not this DX Storage CFS file system
should be mounted in read-only mode. The default value is No.
6. Log facility. Enter the logging facility to use. The default value is syslog.
a. Log filename. If the file logging facility was selected, enter the filename and location of the
logfile. By default the installer will use a <mountid>.log file in the /var/log/caringo directory.
With file logging, default log rotation will be configured to keep up to 8 log files, rotating
weekly or at a max file size of 512 MB.

Copyright © 2011 Caringo, Inc.
All rights reserved 7 Version 2.6 February 2011
b. Syslog facility. If the syslog option was selected, enter the facility to log to. The default
value is local4. Configuration of a remote syslog host must be done in the syslogd
configuration file.
7. Log level. Enter the minimum level of log messages to record. The default value is info. Debug
is not recommended unless explicitly requested as part of a support investigation due to the
amount of disk space consumed that will not then be available for core processes. Each log
level includes the following:
• info: errors, warnings and information messages like system start up and stop
• verbose: errors, warnings, and info messages plus high-level operational functions
• debug: errors, warnings, info and verbose messages plus lower level operational functions
• trace: all possible log messages
Note
Debug or trace logging are not recommended unless explicitly requested as part
of a support investigation due to the amount of disk space consumed. If debug
logging is required for any length of time, administrators must monitor space
availability carefully and are strongly encouraged to allocate a dedicated partition
for logging to protect other system services.
8. CNS Host: The IP address or hostname of the name space server that will store the filesystem
metadata for the mount you are configuring. The default is localhost (127.0.0.1), which will
configure the share to use a name space on the local server. To use a name space configured
on a remote server, enter the external IP address for the previously created name space.
Note
If configuring a multi-server environment where more than one CFS server is sharing
a single CNS instance, you must enter the same CNS host information that was
entered in the cns-admin utility for each share. cfs-admin will generate a world
unique serial number for every created CFS share for internal use when requesting
exclusive file locks across all servers in the name space. This serial number is
stored with the CFS configuration files so they must not be copied from one server to
another to ensure the CFS share serial number is not duplicated.
9. CFS Time Horizon (Default=0d means no Timescapes): The time that deleted file system
objects will be kept in order to be available for a timescape mount of the file system at a time
in the past. Times are accepted in hours (i.e. 2h), days (i.e. 1d), months (i.e. 2m), or years (i.e.
1y). After the time horizon, deleted files are garbage collected from DX Storage and can no
longer be accessed.
10. Use Zeroconf for DX Storage configuration?. This option allows you to specify whether
or not Zeroconf should be used to discover the list of nodes in the DX Storage cluster. The
CFS server must be in the same subnet with DX Storage in order to use Zeroconf. CFS 2.6 is
compatible with DX Storage versions 3.0.5 and later.
a. If 'No' is selected:

Copyright © 2011 Caringo, Inc.
All rights reserved 8 Version 2.6 February 2011
Cluster primary node address: Enter the IP address or hostname of a DX Storage node
in the target cluster. The target cluster must be the same as the one configured for the
Cluster Name Space (CNS).
Primary node SCSP port: Enter the port the DX Storage node uses for SCSP
communications. The default is '80'.
Cluster secondary node address: Enter the IP address or hostname of a second DX
Storage node (different from the primary) in the target cluster for redundancy. If sub-
clusters are configured, a node in a sub-cluster that is not the same as the primary's is
recommended. The target cluster must be the same as the one configured for the Cluster
Name Space (CNS).
Secondary node SCSP port: Enter the port the secondary DX Storage node uses for
SCSP communications. The default is '80'.
b. If 'Yes' is selected:
Cluster name: Enter the name of the DX Storage cluster. This must match the value of the
DX Storage "cluster" parameter in the node.cfg file and be the same as the one configured
for the Cluster Name Space (CNS). Note that if using Zeroconf for discovering the list of
nodes and using a cluster name with periods in it, the periods should be replaced with
underscores to ensure proper node discovery.
11. If the Spooler/cache and mount directories do not already exist, you will be asked if you would
like them to be created. The recommended response is 'Yes', as DX Storage CFS cannot
start without these directories. You will also be asked if you would like to mount the configured
mountpoint immediately.
12. After installing DX Storage CFS initially, running the following command will start the CFS
monitoring process. This will be started automatically on subsequent reboots.
# /etc/init.d/caringo-cfs start
Note
If you attempt to start CFS without first starting the Cluster Name Space, the
initialization will fail after 60 seconds. This may leave the filesystem mounted without
a name space. To unmount the filesystem, execute the following command and then
start the name space and CFS sequentially:
fusermount -uz /mnt/mountname
2.2.4. Post-configuration File System Options
In addition to the required parameters gathered by the cfs-admin mkfs utility, there are some
optional mountpoint settings that can be set in the /etc/fstab file to modify certain file system
behaviors. The cfs-admin mkfs utility will automatically create an entry in /etc/fstab each time the
utility is run for a unique mount ID. Prior to committing any changes to this file, creating a backup
of the previous version of the file is highly recommended. To modify this file, simply open it in a
text editor like vi and find the entry for the mountpoint you would like to update. For instance, the
'testMount1' mountpoint point would display as follows using the default parameters from the cfs-
admin mkfs utility:
testMount1 /mnt/testMount1 fuse.cfs config=testMount1.conf,noatime

Copyright © 2011 Caringo, Inc.
All rights reserved 9 Version 2.6 February 2011
2.2.4.1. ACLs
The cfs-admin mkfs utility does not by default enable Access Control Lists or ACLs. ACLs provide
more fine-grained file and directory access control than the basic user, group, owner permissions
and can be very useful in complex permission scenarios. For more information on ACLs functionality
and syntax, please refer to the ACL manpage. To enable ACLs for a CFS mountpoint, you will
need to edit as root the /etc/fstab file and add 'acl' to the mount options as shown in the following
example:
testMount1 /mnt/testMount1 fuse.cfs config=testMount1.conf,noatime,acl
2.2.4.2. Access Time Updates
The cfs-admin mkfs utility by default disables updates to the last access time for both a file and
a directory by adding the 'noatime' options to the fstab entry for each mountpoint. Access time or
atime can have a significant impact on file system performance and is often not needed. This setting
does not impact update times when a file or directory is modified. If this information is required
for a particular mountpoint, the options can be modified to reenable access time by removing the
'noatime' value since access time updates are enabled by default. An updated fstab entry for the
'testMount1' mountpoint above would appear as:
testMount1 /mnt/testMount1 fuse.cfs config=testMount1.conf
2.3. Deleting a Mountpoint
In the event that you need to delete a mountpoint, the 'cfs-admin mkfs' utility can also be used to
specify the mountpoint and issue the delete. Prior to deleting the mountpoint, you should delete all
contents from the mountpoint and then unmount it to be sure there are no in process activities (see
Section 3.2). As a root user, type the following command (where 'testMount1' equals the name of
the mountpoint to be deleted):
# cfs-admin mkfs --delete --cfs-id=testMount1
The delete command should be used only when the mountpoint is no longer needed both currently
and historically. It may take some time to delete all the files and resources associated with a
mountpoint. This will not remove the name space nor will it remove the files' streams from the DX
Storage cluster.
2.4. Upgrading CFS
2.4.1. Upgrading from Versions 1.x
The 2.6 release now supports data migration from the previous 1.x versions of CFS that utilized a
MySQL database. Please contact your support resource for migration instructions.
2.4.2. Upgrading from Version 2.0
2.0 Ubuntu-based installations can migrate to 2.6 by installing a new Red Hat server with
configuration data and shares that exactly mirror the current installation and then switching out the
newly created CNS rootID with the previous Ubuntu-based one. The same DX Storage cluster must
be used for both the old and the new installation. Please reference the 'Restoring CFS' section of
the Setup and Configuration Guide for complete details.

Copyright © 2011 Caringo, Inc.
All rights reserved 10 Version 2.6 February 2011
2.4.3. Upgrading from Version 2.5
Version 2.5 Red Hat installations can upgrade to 2.6 by stopping all services, installing the new
version and then restarting services. After copying the 2.6 distribution file to the server and
unzipping it, run the following commands as root:
Note
Any configuration file backups previously taken for disaster recovery purposes must be
discarded and a new backup taken, as previous configuration files are not compatible
with the new software version.
1. Stop all client activity on the server. Stopping remote Samba or NFS access to the share is an
easy way to do this.
2. Monitor the local spooler cache draining to zero by using the following script:
./cfs-checkspool
The script will monitor the spool cache and any active writes for all configured shares in a loop
and print a statement when all activity has completed and it is safe to shutdown CFS.
3. Stop CFS: /etc/init.d/caringo-cfs stop
4. Stop CNS: /etc/init.d/caringo-cns stop
5. Install the new software: ./installCFS.sh
6. Start CNS: /etc/init.d/caringo-cns start
7. Start CFS: /etc/init.d/caringo-cfs start
2.5. Removing CFS and CNS
The following commands should be run to remove CNS and CFS if the data and metadata are no
longer needed:
1. Prior to removing CFS, the timescape horizon for all shares should be set to 0 days and then
all files and folders should be manually deleted from all mountpoints to ensure content is not
orphaned in DX Storage. This will trigger the namespace garbage collection process to delete
the associated content from DX Storage. When the CNS log at debug level consistently shows
that no files have been deleted as part of the garbage collection process, it is safe to proceed
with removal of the software.
2. Stop CFS: /etc/init.d/caringo-cfs stop
3. Stop CNS: /etc/init.d/caringo-cns stop
4. To remove all mounts, do the following for each mount: cfs-admin mkfs --delete --
cfs-id=testMount1 (where 'testMount1' equals the name of the mountpoint to be deleted)
5. Purge CFS code: yum remove caringo-cfs
Note
Removing CFS without first unmounting will not succeed and will print a scriptlet error
to the command line. Due to a defect in the Red Hat 6 package manager (yum) a more

Copyright © 2011 Caringo, Inc.
All rights reserved 11 Version 2.6 February 2011
descriptive error is not passed through to the client. Check the syslog for a full description
of the possible errors if the uninstall is unsuccessful.
2.6. Restoring CFS
Should a CFS server ever fail or need to be repurposed, all persisted data can be restored or moved
onto a new CFS server by configuring the server as you would a new server but then replacing the
CNS rootUUID with the previously configured one. Recovering with the rootUUID will not restore
files that had not yet been spooled into DX Storage or that were written to the namespace in the 60
seconds prior to a server failure. For planned moves, administrators should ensure that writes are
quiesced for a period of time prior to the restore.
To re-install or move CFS onto a new server, execute the following steps:
1. If migrating from version 2.0, install Red Hat 6.0 on a new server.
2. Install and configure CNS as it was configured previously using the installation instructions
above. The DX Storage cluster MUST be the same as the previous configuration in order to
retrieve the previously stored data.
3. Stop all client activity on the old server. Stopping remote Samba or NFS access to the share is
an easy way to do this.
4. Monitor the local spooler cache draining to zero by using the following script:
./cfs-checkspool
The script will monitor the spool cache and any active writes for all configured shares in a loop
and print a statement when all activity has completed and it is safe to shutdown CFS.
5. Stop CFS on the old server: /etc/init.d/caringo-cfs stop
6. Stop CNS on the old server: /etc/init.d/caringo-cns stop
7. Edit the CNS configuration file on the new server (nano /opt/caringo/fabric/etc/
cns.conf) and replace the existing root_anchor parameter with the previous server's
rootUUID.
8. Recreate the CFS shares using the cfs-admin mkfs script. The DX Storage cluster and the
mount IDs for the shares MUST be same as the ones prior to the upgrade.
9. Start CNS and CFS as normal.
If you are restoring a system after a failure, the process is the same except steps 2-5 above will not
be necessary.

Copyright © 2011 Caringo, Inc.
All rights reserved 12 Version 2.6 February 2011
Chapter 3. Running and Managing DX Storage CFS
3.1. Starting DX Storage CFS and CNS
The Cluster Name Space should always be started prior to CFS. The name space can be started
using the following command:
$ /etc/init.d/caringo-cns start
CNS will fail to start and print an appropriate error message if one of several critical configuration
parameters is missing or invalid. Once the configuration is corrected via the cns-admin script, CNS
should start correctly.
DX Storage CFS will start automatically when the DX Storage CFS server boots if the "mount on
boot" option was selected during the configuration process for each mountpoint. If the process was
stopped for any reason it can be manually started with a standard mount command. To mount all
configured DX Storage CFS mountpoints at once, type in the following command:
$ sudo /etc/init.d/caringo-cfs start
To mount a single mountpoint that was previously defined using the cfs-admin mkfs script, run a
command similar to the following (where /mnt/testMount1 is the mounted destination for the desired
mountpoint)
$ sudo mount /mnt/testMount1
CFS will fail to mount a share and will print an appropriate error message if one of several critical
configuration parameters is missing or invalid. Once the configuration is corrected via the cfs-admin
mkfs script, the share should mount correctly.
Before writing any data to a mountpoint, run a 'mount' command with no options to ensure the
desired mountpoints started correctly and are present in the mounted list. If the mountpoint is not in
the mounted list, the install was not successful and you should not write any data to the mountpoint.
3.2. DX Storage CFS and CNS Shutdown
To stop DX Storage CFS and unmount all configured shares, use the following command:
$ sudo /etc/init.d/caringo-cfs stop
To stop and/or unmount a specific configured share, use a command similar to the following
(where /mnt/testMount1 is the mounted destination for the mountpoint):
$ sudo umount /mnt/testMount1
For file systems shared via Samba, NFS, etc, administrators may need to stop associated file
system services before they can cleanly unmount CFS.
Note
If DX Storage CFS is stopped using a kill command, a 'fusermount -u /mnt/mountpoint'
command must be executed before restarting to ensure the mountpoint is properly
released and remounted. If the remount option is utilized, the mountpoint will be
unmounted and then immediately mounted.

Copyright © 2011 Caringo, Inc.
All rights reserved 13 Version 2.6 February 2011
After CFS has been stopped, CNS may also be stopped using the following command:
$ /etc/init.d/caringo-cns stop
3.3. Timescapes
The DX Storage CFS Timescape continuous snapshot feature allows an administrator to mount a
historical, read-only view of a file system at any past time, providing a mechanism to both view and/
or copy previous versions of files that might otherwise be lost.
To utilize the feature, you must first create a timescape mount at the point in time you would like to
view, using the cfs-admin mktimescape script. The script can be initiated by running the following
command:
sudo cfs-admin mktimescape --add
This will bring up a wizard similar to the ones used to initially create a CFS share that will collect the
information necessary to create the timescape mount. It includes the following prompts:
1. TCFSID to be used for this timescape mount: Enter a name for the timescape mountpoint.
It must be unique on each server and consist of letters and numbers without punctuation or
spaces. For example: "acctShareT20100826" or "TimescapearchiveFS20100826".
2. CFSID of the CFS share you wish to make a timescape mount for: Enter the name of the
local CFS share for which you are creating a timescape mount. i.e. "acctShare" or "archiveFS"
from the initial installation example. The share must be on the same server as the timescape
mount you are creating to ensure the configuration data can be shared. The share must be in
existence for the entire time the timescape share is mounted for the timescape to work.
3. CFS mount directory path: Enter the location of the timescape mountpoint. By default, the
utility will use the previously provided TCFS ID in the /mnt directory. Spaces in mountpoint
paths are not supported.
4. Spooler directory path: Enter the location where the spooler/cache directory will reside. The
location must be unique for each distinct read/write TCFS ID or the filesystem mount will fail.
By default, the installer will use the previously provided TCFS ID in the /var/spool/cfs directory.
Extended attributes must be enabled on the specified spooler partition to store the lifepoints
and custom and system metadata associated with each unspooled item.
Warning
A dedicated spooler partition is required for each share to ensure space monitoring is
accurate and the local cache is properly maintained.
5. CFS Timescape (ISO8601 Format ex. 2010-08-26T13:47:42): Enter the time at which to
mount the filesystem specified in ISO 8601 format (ex. 9:38 am US/Pacific on 14 June, 2010
would be written 2010-06-14T09:38:00, with the date prior to T and the time following). The
local time zone offset of the DX Storage CFS server will be used. The wizard will return an
error if you attempt to configure a time period either earlier than the current time minus the
configured timescape horizon or in the future. A timescape share mounted when the configured
horizon passes may result in inconsistent results and an inability to access files that have been
garbage collected from the namespace while the share is mounted.
6. If the Spooler/cache and mount directories do not already exist, you will be asked if you would
like them to be created. The recommended response is 'Yes', as DX Storage CFS cannot

Copyright © 2011 Caringo, Inc.
All rights reserved 14 Version 2.6 February 2011
start without these directories. You will also be asked if you would like to mount the configured
mountpoint immediately.
Once mounted, you will be able to view the specified CFS share at the time in the past specified by
your configured Timescape time. All timescape mounts are read-only but you may copy a deleted
file from a timescape mount to an active, current mount to recreate the file in the present.
Timescape mounts log to the same configured logging facility as the CFS share they are based on.
By default, timescape shares are configured to require less RAM, as a read-only file system requires
less resources than a current file system.
3.3.1. Deleting a Timescape Mount
Similar to an active mount, the 'cfs-admin mktimescape' utility can also be used to specify a
mountpoint to be deleted. As a root user, type the following command (where 'ttestMount1' equals
the name of the timescape mountpoint to be deleted):
# cfs-admin mktimescape --delete --tcfs-id=ttestMount1
Removing the timescape mount will not remove the name space nor will it remove the files' streams
from the DX Storage cluster.
3.3.2. Mounting a Timescape at an Alternate Time
Administrators needing to quickly mount a different time period for the same CFS share may do so
by manually mounting the timescape share with the time overridden by a timescape mount option
similar to the following:
mount -o remount,timescape=2010-09-15T12:34:20 /mnt/ttestMount1/
where the timescape value is an ISO 8601 time. This time is not persisted into the timescape share
configuration file, but prevents creation of a multiple timescape shares with different time periods for
the same CFS share. This mount option only applies to timescape mounts and cannot be used for
active, current mounts.
3.4. File Revisions and DX Storage Deletion
Each modification to a file stored in CFS creates a new object in DX Storage, with CNS keeping
track of the modification revision that is current for each file. Old revisions are deleted from DX
Storage via a background garbage collection process based on the configured time horizon.
Files that are explicitly deleted from the CFS mount are removed from DX Storage via the same
background process. After the time horizon, deleted files are no longer accessible via a timescape
mount.
3.5. Support and Management Utilities
3.5.1. cns-reporter
The cns-reporter utility provides basic capacity and content information for the Cluster Name Space.
It takes the following arguments:
Argument Definition and usage
-a ADDRESS or --address=ADDRESS Address of a DX Storage node used by the
CFS share

Copyright © 2011 Caringo, Inc.
All rights reserved 15 Version 2.6 February 2011
Argument Definition and usage
-p PORT or --port=PORT SCSP Port for the configured DX Storage node
(default=80)
-r ROOTUUID or --uuid=ROOTUUID The CNS rootID you received during initial
configuration
-c CFSID or --cfsid=CFSID The CFSID of the share for which you want to
report
-l list or --list=LIST Option that returns the list of CFSIDs in the
designated rootUUID. Output returns the
Namespace ID, CFSID, Time Horizon, Creation
Time and Anchor Stream of the toplevel
directory.
-s SUBTREE or --subtree=SUBTREE Relative path of the subtree for which you want
to report ( default is the entire share )
-t HISTORY or --timescape=HISTORY Whether or not timescape data should be
included in the report. 0=no(default) 1=yes.
This argument does not work with the subtree
option.
The following shows an example command for a CFS share named 'Accounting' with a subtree
named '2010/taxes':
cns-reporter -a 192.168.1.220 -r f4f5388cb8b21ebb917c332625f75fca -c
Accounting -s "2010/taxes"
Example output:
Namespace Capacity Report for [Accounting/2010/taxes] created
in elapsed: 330.703432 s
Total Folders: 11258
Total Files: 6026
Total Data Capacity Used: 10.73 Gb or [11521613767] bytes
Total Metadata objects: 22552
Total Metadata Capacity Used: 0.029100 Gb or [31245858] bytes
Total CAStor UUID's found:6026
Total Folder attributes: 90530
Total File attributes: 54234
3.5.2. cfs-grab
cfs-grab is a command-line tool that collects runtime environment information to assist with
support ticket submissions. From a command line as a root user, simply type 'cfs-grab' to utilize
the tool. The tool will generate a gzipped tar file with the current date timestamp (example:
grab-1272553621.tgz ) with a collection of relevant data that can be attached to a support ticket or
email.
3.5.3. Configuration Modifications with cns-admin and cfs-admin
3.5.3.1. cfs-admin config
Administrators who need to update a configuration parameter for a previously configured CFS share
can do so using the command:

Copyright © 2011 Caringo, Inc.
All rights reserved 16 Version 2.6 February 2011
sudo cfs-admin config
The script will prompt for the CFSID you wish to modify and then display the same prompts from
initial configuration but with the current configuration displayed for each prompt. If you do not wish to
modify a particular parameter, simple press enter and the current value will remain in place. Some
parameters like the configured namespace that might strand or corrupt data are view only.
Note
The share you are attempting to edit must have been mounted at least once prior to
running cfs-admin config to ensure all configuration data is completely populated.
3.5.3.2. cns-admin
If the cns-admin script is re-run after initial configuration, it will prompt for the same questions again
but pre-populate the previously provided values and allow for modifications. The script can be run
using the same command as initial configuration:
sudo cns-admin
At the end, the script will prompt to restart CNS for the new parameters to take effect.
3.5.4. cfs-loglevel
cfs-loglevel is a tool that allows easy access to update the log level for all CFS and CNS logs. From
a command line as a root user, type 'cfs-loglevel', then select the desired log level from the dialog
box and click 'Ok'. The log level will be updated in the configuration files for both CFS and CNS and
will take effect immediately with no required process restart.
3.6. DX Storage Metadata and Policies
Along with the standard file system attributes that are associated with a file stored in DX Storage
CFS, additional metadata and lifecycle policies can be stored with files when they are stored in DX
Storage. This metadata can assist with storage management and data movement through various
life cycles. All metadata and policies are specific to DX Storage and are not visible from within the
file system.
Metadata and policies are specific to each mountpoint so it is advisable to configure separate
mountpoints in cases where distinct metadata values would be helpful. For instance, to distribute
content based on the branch office a file originated from, set up a mountpoint for each branch office
and add a custom metadata attribute with the branch office name that would be added to each file
stored. Similarly, to store more replicas of files needed by the Media department due to frequent
read activity, setup a separate mountpoint for the Media department with a Lifecycle policy for
additional replicas and leave all other departments utilizing a mountpoint with a lifepoint policy for
fewer replicas.
As a baseline for file metadata, DX Storage CFS will automatically add the following system
attributes on all streams stored in DX Storage:
• Castor-CFS-Server: the name of the server from which the file was originally stored
• Castor-CFS-Version: the version of the CFS software that originally stored the file
• Content-Type: the file's mimetype
• CAStor-CFS-CFSID: the name of the mountpoint id through which the file was originally stored

Copyright © 2011 Caringo, Inc.
All rights reserved 17 Version 2.6 February 2011
• Castor-CFS-FileId: the filesystem id for the file in the CFS name space
• Castor-CFS-Uid: the file's user id at the time it was stored
• Castor-CFS-Gid: the file's group id at the time it was stored
• Castor-CFS-Mode: the file's permission mode in decimal at the time it was stored
The mechanisms to add both custom metadata and life cycle retention policies to files stored in DX
Storage via DX Storage CFS are outlined in the sections below.
3.6.1. Custom Metadata
The 'cfs-admin metadata' utility allows a root user to administer the custom metadata that is
attached to newly created files when they are ready to be stored in DX Storage (last file close plus
5 seconds). Custom metadata applies to all files stored in a particular mountpoint and, like all DX
Storage metadata, is immutable after the file is stored in DX Storage. Thus, changing the custom
metadata for a CFS file system will affect only new or modified files and will not change metadata for
files that were previously written. To add metadata to a mountpoint, execute the following command
as the root user:
$ sudo su -
# cfs-admin metadata --set [--name=NAME] [--value=VALUE] <mount-root>
where <mount-root> is the full path for the CFS mountpoint to which the metadata should be
attached. For example, to add an attribute of 'Department' with a value of 'Legal' to all files in the
legal mountpoint, you would execute the following command:
# cfs-admin metadata --set --name=Department --value=Legal /mnt/legal
Metadata names must be unique within a mountpoint and may contain up to 100 characters. A '--
set' command that uses a pre-existing metadata name with new or different values will overwrite the
previous values. By default, metadata is cached for 60 seconds so attachment of new or updated
metadata to new files written to the mountpoint will be delayed until the cache is refreshed.
To add multiple values for a name, simply append them with a separator. For instance, a mountpoint
that had files that applied to both the legal department and the accounting department might
have a metadata value of '--value=Legal, Accounting'. Metadata values may contain up to 1536
characters. A "no space left on device" error will be returned if the size limit is exceeded. ASCII
control characters are not supported and any preceding or trailing whitespace will be discarded from
both the name and the value.
To view all the existing metadata name/value pairs for a mountpoint, the 'get' action for the cfs-
admin metadata utility can be utilized. For instance, to view all metadata for the legal mountpoint the
following command would be executed:
# cfs-admin metadata --get /mnt/legal
Similarly, if you wanted to see the value for a particular metadata name, you would include the
name in the command. For example:
# cfs-admin metadata --get --name=Department /mnt/legal
To delete a metadata attribute for a mountpoint, the 'delete' action for the cfs-admin metadata utility
can be executed. For instance, to delete the Department=Legal attribute example added to the legal
mountpoint above, the following command could be executed: