AWS Storage Gateway User manual

AWS Storage Gateway
User Guide
API Version 2013-06-30

AWS Storage Gateway User Guide
AWS Storage Gateway: User Guide

AWS Storage Gateway User Guide
Table of Contents
What Is AWS Storage Gateway? ........................................................................................................... 1
Are You a First-Time AWS Storage Gateway User? .......................................................................... 2
How AWS Storage Gateway Works ............................................................................................... 2
File Gateways .................................................................................................................... 3
Volume Gateways ............................................................................................................... 3
Tape Gateways ................................................................................................................... 6
Pricing ...................................................................................................................................... 8
Plan Your Gateway Deployment ................................................................................................... 8
Getting Started ................................................................................................................................ 10
Sign Up for AWS Storage Gateway ............................................................................................. 10
AWS Regions ........................................................................................................................... 10
Requirements ........................................................................................................................... 10
Hardware and Storage Requirements .................................................................................. 11
Network and Firewall Requirements .................................................................................... 12
Supported Hypervisors and Host Requirements .................................................................... 20
Supported NFS Clients for a File Gateway ........................................................................... 21
Supported SMB Clients for a File Gateway ........................................................................... 21
Supported File System Operations for a File Gateway ........................................................... 21
Supported iSCSI Initiators .................................................................................................. 22
Supported Third-Party Backup Applications for a Tape Gateway ............................................. 22
Accessing AWS Storage Gateway ................................................................................................ 23
Using the Hardware Appliance ........................................................................................................... 24
Supported AWS Regions ............................................................................................................ 24
Setting Up Your Hardware Appliance .......................................................................................... 25
Rack-Mount and Plug In Your Hardware Appliance ....................................................................... 25
Configure Network Parameters .................................................................................................. 27
Activate Your Hardware Appliance .............................................................................................. 29
Launching a Gateway ................................................................................................................ 32
Configuring an IP Address for the Gateway .................................................................................. 33
Configuring Your Gateway ......................................................................................................... 34
Removing a Gateway ................................................................................................................ 34
Deleting Your Hardware Appliance ............................................................................................. 34
Creating Your Gateway ...................................................................................................................... 36
Creating a File Gateway ............................................................................................................ 36
Creating a Gateway .......................................................................................................... 36
Creating a File Share ........................................................................................................ 42
Using Your File Share ........................................................................................................ 51
Creating a Volume Gateway ....................................................................................................... 56
Creating a Gateway .......................................................................................................... 57
Creating a Volume ............................................................................................................ 63
Using Your Volume ........................................................................................................... 65
Backing Up Your Volumes ................................................................................................. 71
Creating a Tape Gateway ........................................................................................................... 75
Creating a Gateway .......................................................................................................... 75
Creating Tapes ................................................................................................................. 82
Using Your Tape Gateway .................................................................................................. 83
Activating a Gateway in a Virtual Private Cloud .......................................................................... 133
Creating a Gateway Using a VPC Endpoint ......................................................................... 134
Managing Your Gateway .................................................................................................................. 144
Managing Your File Gateway .................................................................................................... 144
Adding a File Share ........................................................................................................ 144
Deleting a File Share ....................................................................................................... 146
Editing Storage Settings for Your File Share ....................................................................... 148
Editing Metadata Defaults for Your NFS File Share .............................................................. 149
API Version 2013-06-30
iii

AWS Storage Gateway User Guide
Editing Access Settings for Your NFS File Share .................................................................. 150
Editing Access Settings for Your SMB File Share ................................................................. 150
Refreshing Objects in Your Amazon S3 Bucket ................................................................... 153
Using S3 Object Lock with File Gateway ............................................................................ 154
Understanding File Share Status ....................................................................................... 154
File Share Best Practices .................................................................................................. 155
Managing Your Volume Gateway .............................................................................................. 156
Adding a Volume ............................................................................................................ 156
Expanding the Size of a Volume ....................................................................................... 157
Cloning a Volume ........................................................................................................... 157
Viewing Volume Usage .................................................................................................... 159
Deleting a Volume .......................................................................................................... 160
Moving Your Volumes to a Different Gateway .................................................................... 160
Reducing the Amount of Billed Storage on a Volume .......................................................... 162
Creating a One-Time Snapshot ......................................................................................... 162
Editing a Snapshot Schedule ............................................................................................ 162
Deleting Snapshots ......................................................................................................... 163
Understanding Volume Status and Transitions .................................................................... 171
Managing Your Tape Gateway .................................................................................................. 178
Adding Tapes ................................................................................................................. 178
Archiving Tapes .............................................................................................................. 180
Moving a Tape from Glacier to Deep Archive ...................................................................... 180
Retrieving Archived Tapes ................................................................................................ 181
Viewing Tape Usage ........................................................................................................ 181
Deleting Tapes ............................................................................................................... 182
Disabling Your Tape Gateway ........................................................................................... 182
Understanding Tape Status .............................................................................................. 183
Monitoring Your Gateway and Resources ........................................................................................... 185
Understanding Gateway Metrics ............................................................................................... 185
AWS Storage Gateway Metrics .......................................................................................... 185
Dimensions for AWS Storage Gateway Metrics .................................................................... 196
Monitoring the Upload Buffer .................................................................................................. 196
Monitoring Cache Storage ....................................................................................................... 198
Monitoring Your File Share ...................................................................................................... 199
Getting Notified About File Operations ............................................................................. 199
Understanding File Share Metrics ..................................................................................... 203
Monitoring Your Volume Gateway ............................................................................................ 204
Using Amazon CloudWatch Metrics ................................................................................... 205
Measuring Performance Between Your Application and Gateway ........................................... 206
Measuring Performance Between Your Gateway and AWS .................................................... 207
Understanding Volume Metrics ......................................................................................... 210
Monitoring Your Tape Gateway ................................................................................................ 212
Using Amazon CloudWatch Metrics ................................................................................... 213
Measuring Performance Between Your Tape Gateway and AWS ............................................ 213
Logging Storage Gateway API Calls with AWS CloudTrail ............................................................. 215
Storage Gateway Information in CloudTrail ........................................................................ 216
Understanding Storage Gateway Log File Entries ................................................................ 216
Maintaining Your Gateway ............................................................................................................... 219
Shutting Down Your Gateway VM ............................................................................................. 219
Starting and Stopping a Volume or Tape Gateway .............................................................. 219
Managing Local Disks .............................................................................................................. 220
Deciding the Amount of Local Disk Storage ....................................................................... 220
Sizing the Upload Buffer ................................................................................................. 221
Sizing Cache Storage ...................................................................................................... 222
Configuring an Upload Buffer and Cache Storage ............................................................... 223
Using Ephemeral Storage With EC2 Gateways .................................................................... 223
Managing Bandwidth .............................................................................................................. 224
API Version 2013-06-30
iv

AWS Storage Gateway User Guide
Changing Bandwidth Throttling Using the Storage Gateway Console ..................................... 224
Using the AWS SDK for Java ............................................................................................ 225
Using the AWS SDK for .NET ............................................................................................ 226
Using the AWS Tools for Windows PowerShell .................................................................... 227
Managing Gateway Updates ..................................................................................................... 228
Performing Maintenance Tasks on the Local Console ................................................................... 229
Performing Tasks on the VM Local Console (File Gateway) ................................................... 229
Performing Tasks on the EC2 Local Console (File Gateway) .................................................. 244
Performing Tasks on the VM Local Console (Volume and Tape Gateways) ............................... 252
Performing Tasks on the EC2 Local Console (Volume and Tape Gateways) .............................. 267
Accessing the Gateway Local Console ................................................................................ 273
Configuring Network Adapters for Your Gateway ................................................................ 275
Deleting Your Gateway and Removing Resources ........................................................................ 281
Deleting Your Gateway by Using the AWS Storage Gateway Console ...................................... 282
Removing Resources from a Gateway Deployed On-Premises ............................................... 282
Removing Resources from a Gateway Deployed on an Amazon EC2 Instance ........................... 284
Performance .................................................................................................................................. 285
Performance Guidance for File Gateways ................................................................................... 285
Performance Guidance for Tape Gateways ................................................................................. 286
Optimizing Gateway Performance ............................................................................................. 287
Add Resources to Your Gateway ....................................................................................... 287
Use a Larger Block Size for Tape Drives ............................................................................. 288
Optimize the Performance of Virtual Tape Drives ............................................................... 288
Add Resources to Your Application Environment ................................................................. 288
Security ......................................................................................................................................... 290
Creating a gateway in a VPC .................................................................................................... 290
Create a VPC Endpoint .................................................................................................... 290
Configuring CHAP Authentication ............................................................................................. 291
Viewing and Editing CHAP Credentials .............................................................................. 292
Encrypting Your Data Using AWS KMS ...................................................................................... 293
Authentication and Access Control ............................................................................................ 294
Authentication ............................................................................................................... 294
Access Control ................................................................................................................ 295
Overview of Managing Access .......................................................................................... 296
Using Identity-Based Policies (IAM Policies) ........................................................................ 299
Using Tags to Control Access to File Gateway Resources ...................................................... 305
Using ACLs for SMB File Share Access ............................................................................... 307
Storage Gateway API Permissions Reference ...................................................................... 309
Troubleshooting Your Gateway ......................................................................................................... 316
Troubleshooting On-Premises Gateway Issues ............................................................................ 316
Enabling AWS Support To Help Troubleshoot Your Gateway ................................................. 318
Troubleshooting Your Microsoft Hyper-V Setup .......................................................................... 320
Troubleshooting Amazon EC2 Gateway Issues ............................................................................ 323
Your Gateway Activation Hasn't Occurred After a Few Moments ............................................ 323
You Can't Find Your EC2 Gateway Instance in the Instance List ............................................. 323
You Created an Amazon EBS Volume But Can't Attach it to Your EC2 Gateway Instance ............ 324
You Can't Attach an Initiator to a Volume Target of Your EC2 Gateway .................................. 324
You Get a Message That You Have No Disks Available When You Try to Add Storage Volumes .... 324
You Want to Remove a Disk Allocated as Upload Buffer Space to Reduce Upload Buffer Space ... 324
Throughput to or from Your EC2 Gateway Drops to Zero ..................................................... 324
You Want Your File Gateway to Use a C5 or M5 EC2 Instance Type Instead of C4 or M4 ............. 324
Get AWS Support to Help Troubleshoot Your Gateway ........................................................ 325
Troubleshooting Hardware Appliance Issues ............................................................................... 326
You Can't Determine the Service IP Address ....................................................................... 326
How Do You Perform a Factory Reset? .............................................................................. 327
Where Do You Obtain Dell iDRAC Support? ........................................................................ 327
You Can't Find the Hardware Appliance Serial Number ........................................................ 327
API Version 2013-06-30
v

AWS Storage Gateway User Guide
Where to Obtain Hardware Appliance Support ................................................................... 327
Troubleshooting File Share Issues ............................................................................................. 328
Your File Share Is Stuck in CREATING Status ...................................................................... 328
You Can't Create a File Share ........................................................................................... 328
SMB File Shares Do Not Allow Multiple Different Access Methods .......................................... 329
Multiple File Shares Can't Write to the Mapped Amazon S3 Bucket ........................................ 329
You Can't Upload Files into Your S3 Bucket ........................................................................ 329
Can't Change the Default Encryption to Use SSE-KMS to Encrypt Objects Stored in My Amazon
S3 Bucket. ..................................................................................................................... 329
Object Versioning Might Affect What You See in Your File System ......................................... 330
ACL Permissions Aren't Working as Expected ...................................................................... 330
Your Gateway Performance Declined After You Performed a Recursive Operation .................... 331
Troubleshooting Volume Issues ................................................................................................ 331
The Console Says That Your Volume Is Not Configured ........................................................ 331
The Console Says That Your Volume Is Irrecoverable ........................................................... 331
Your Cached Gateway is Unreachable And You Want to Recover Your Data ............................. 332
The Console Says That Your Volume Has PASS THROUGH Status .......................................... 332
You Want to Verify Volume Integrity and Fix Possible Errors ................................................. 333
Your Volume's iSCSI Target Doesn’t Appear in Windows Disk Management Console .................. 333
You Want to Change Your Volume's iSCSI Target Name ....................................................... 333
Your Scheduled Volume Snapshot Did Not Occur ................................................................ 333
You Need to Remove or Replace a Disk That Has Failed ....................................................... 333
Throughput from Your Application to a Volume Has Dropped to Zero .................................... 333
A Cache Disk in Your Gateway Encounters a Failure ............................................................. 334
A Volume Snapshot Has PENDING Status Longer Than Expected ........................................... 334
Troubleshooting Virtual Tape Issues .......................................................................................... 334
Recovering a Virtual Tape From An Unrecoverable Gateway ................................................. 335
Troubleshooting Irrecoverable Tapes ................................................................................. 337
Recovering Your Data: Best Practices ......................................................................................... 338
Recovering from an Unexpected Virtual Machine Shutdown ................................................. 338
Recovering Your Data from a Malfunctioning Gateway or VM ............................................... 339
Retrieving Your Data from an Irrecoverable Volume ............................................................ 339
Recovering Your Data from an Irrecoverable Tape ............................................................... 340
Recovering Your Data from a Malfunctioning Cache Disk ...................................................... 340
Recovering Your Data from a Corrupted File System ............................................................ 340
Recovering Your Data From An Inaccessible Data Center ...................................................... 341
Additional Resources ....................................................................................................................... 343
Host Setup ............................................................................................................................ 343
Configuring VMware for Storage Gateway ......................................................................... 343
Synchronizing Your Gateway VM Time .............................................................................. 348
Volume or Tape Gateway on Amazon EC2 Host .................................................................. 349
File Gateway on EC2 Host ............................................................................................... 351
Volume Gateway .................................................................................................................... 354
Removing Disks from Your Gateway .................................................................................. 354
EBS Volumes for EC2 Gateways ........................................................................................ 356
Tape Gateway ........................................................................................................................ 357
Working with VTL Devices ............................................................................................... 357
Working with Tapes ........................................................................................................ 361
Getting Activation Key ............................................................................................................ 362
AWS CLI ........................................................................................................................ 363
Linux (bash/zsh) ............................................................................................................. 363
Microsoft Windows PowerShell ......................................................................................... 363
Connecting iSCSI Initiators ....................................................................................................... 364
Connecting to Your Volumes to a Windows Client ............................................................... 365
Connecting to VTL Devices .............................................................................................. 368
Connecting Your Volumes or VTL Devices to a Linux Client .................................................. 372
Customizing iSCSI Settings .............................................................................................. 374
API Version 2013-06-30
vi

AWS Storage Gateway User Guide
Configuring CHAP Authentication ..................................................................................... 377
Using AWS Direct Connect with Storage Gateway ....................................................................... 386
Port Requirements .................................................................................................................. 386
Connecting to Your Gateway .................................................................................................... 391
Getting an IP Address from an Amazon EC2 Host ............................................................... 391
Understanding Resources and Resource IDs ................................................................................ 392
Working with Resource IDs .............................................................................................. 393
Tagging Your Resources ........................................................................................................... 393
Working with Tags .......................................................................................................... 394
See Also ........................................................................................................................ 395
Open-Source Components ....................................................................................................... 395
Storage Gateway Limits ........................................................................................................... 395
Limits for File Shares ...................................................................................................... 395
Limits for Volumes ......................................................................................................... 396
Limits for Tapes ............................................................................................................. 396
Recommended Local Disk Sizes For Your Gateway .............................................................. 397
Using Storage Classes ............................................................................................................. 397
Using Infrequent Access Storage Class With File Gateway .................................................... 397
Using GLACIER Storage Class With File Gateway ................................................................. 398
API Reference ................................................................................................................................. 399
Required Request Headers ....................................................................................................... 399
Signing Requests .................................................................................................................... 400
Example Signature Calculation ......................................................................................... 401
Error Responses ...................................................................................................................... 402
Exceptions ..................................................................................................................... 403
Operation Error Codes .................................................................................................... 404
Error Responses .............................................................................................................. 416
Operations ............................................................................................................................. 418
Document History .......................................................................................................................... 419
Earlier Updates ....................................................................................................................... 421
API Version 2013-06-30
vii

AWS Storage Gateway User Guide
What Is AWS Storage Gateway?
AWS Storage Gateway connects an on-premises software appliance with cloud-based storage to provide
seamless integration with data security features between your on-premises IT environment and the AWS
storage infrastructure. You can use the service to store data in the AWS Cloud for scalable and cost-
effective storage that helps maintain data security.
AWS Storage Gateway offers file-based, volume-based, and tape-based storage solutions:
File Gateway – A file gateway supports a file interface into Amazon Simple Storage Service (Amazon S3)
and combines a service and a virtual software appliance. By using this combination, you can store and
retrieve objects in Amazon S3 using industry-standard file protocols such as Network File System (NFS)
and Server Message Block (SMB). The software appliance, or gateway, is deployed into your on-premises
environment as a virtual machine (VM) running on VMware ESXi or Microsoft Hyper-V hypervisor. The
gateway provides access to objects in S3 as files or file share mount points. With a file gateway, you can
do the following:
• You can store and retrieve files directly using the NFS version 3 or 4.1 protocol.
• You can store and retrieve files directly using the SMB file system version, 2 and 3 protocol.
• You can access your data directly in Amazon S3 from any AWS Cloud application or service.
• You can manage your Amazon S3 data using lifecycle policies, cross-region replication, and versioning.
You can think of a file gateway as a file system mount on S3.
A file gateway simplifies file storage in Amazon S3, integrates to existing applications through industry-
standard file system protocols, and provides a cost-effective alternative to on-premises storage. It also
provides low-latency access to data through transparent local caching. A file gateway manages data
transfer to and from AWS, buffers applications from network congestion, optimizes and streams data in
parallel, and manages bandwidth consumption. File gateways integrate with AWS services, for example
with the following:
• Common access management using AWS Identity and Access Management (IAM)
• Encryption using AWS Key Management Service (AWS KMS)
• Monitoring using Amazon CloudWatch (CloudWatch)
• Audit using AWS CloudTrail (CloudTrail)
• Operations using the AWS Management Console and AWS Command Line Interface (AWS CLI)
• Billing and cost management
Volume Gateway – A volume gateway provides cloud-backed storage volumes that you can mount as
Internet Small Computer System Interface (iSCSI) devices from your on-premises application servers. The
gateway supports the following volume configurations:
•Cached volumes – You store your data in Amazon Simple Storage Service (Amazon S3) and retain a
copy of frequently accessed data subsets locally. Cached volumes offer a substantial cost savings on
primary storage and minimize the need to scale your storage on-premises. You also retain low-latency
access to your frequently accessed data.
•Stored volumes – If you need low-latency access to your entire dataset, first configure your on-
premises gateway to store all your data locally. Then asynchronously back up point-in-time snapshots
of this data to Amazon S3. This configuration provides durable and inexpensive offsite backups that
you can recover to your local data center or Amazon EC2. For example, if you need replacement
capacity for disaster recovery, you can recover the backups to Amazon EC2.
API Version 2013-06-30
1

AWS Storage Gateway User Guide
Are You a First-Time AWS Storage Gateway User?
Tape Gateway – With a tape gateway, you can cost-effectively and durably archive backup data in
GLACIER or DEEP_ARCHIVE. A tape gateway provides a virtual tape infrastructure that scales seamlessly
with your business needs and eliminates the operational burden of provisioning, scaling, and maintaining
a physical tape infrastructure.
You can run AWS Storage Gateway either on-premises as a VM appliance, as a hardware appliance, or in
AWS as an Amazon Elastic Compute Cloud (Amazon EC2) instance. You deploy your gateway on an EC2
instance to provision iSCSI storage volumes in AWS. You can use gateways hosted on EC2 instances for
disaster recovery, data mirroring, and providing storage for applications hosted on Amazon EC2.
For an architectural overview, see How AWS Storage Gateway Works (Architecture) (p. 2). To see the
wide range of use cases that AWS Storage Gateway helps make possible, see the AWS Storage Gateway
detail page.
To get started with Storage Gateway, see the following.
Topics
•Are You a First-Time AWS Storage Gateway User? (p. 2)
•How AWS Storage Gateway Works (Architecture) (p. 2)
•AWS Storage Gateway Pricing (p. 8)
•Plan Your Storage Gateway Deployment (p. 8)
Are You a First-Time AWS Storage Gateway User?
In the following documentation, you can find a Getting Started section that covers setup information
common to all gateways and also gateway-specific setup sections. The Getting Started section shows
you how to deploy, activate, and configure storage for a gateway. The management section shows you
how to manage your gateway and resources:
•Creating a File Gateway (p. 36) provides instructions on how to create and use a file gateway. It
shows you how to create a file share, map your drive to an Amazon S3 bucket, and upload files and
folders to Amazon S3.
•Creating a Volume Gateway (p. 56) describes how to create and use a volume gateway. It shows you
how to create storage volumes and back up data to the volumes.
•Creating a Tape Gateway (p. 75) provides instructions on how to create and use a tape gateway. It
shows you how to back up data to virtual tapes and archive the tapes.
•Managing Your Gateway (p. 144) describes how to perform management tasks for all gateway types
and resources.
In this guide, you can primarily find how to work with gateway operations by using the AWS
Management Console. If you want to perform these operations programmatically, see the AWS Storage
Gateway API Reference.
How AWS Storage Gateway Works (Architecture)
Following, you can find an architectural overview of the available AWS Storage Gateway solutions.
Topics
API Version 2013-06-30
2

AWS Storage Gateway User Guide
File Gateways
•File Gateways (p. 3)
•Volume Gateways (p. 3)
•Tape Gateways (p. 6)
File Gateways
To use a file gateway, you start by downloading a VM image for the file gateway. You then activate the
file gateway from the AWS Management Console or through the Storage Gateway API. You can also
create a file gateway using an Amazon EC2 image.
After the file gateway is activated, you create and configure your file share and associate that share with
your Amazon S3 bucket. Doing this makes the share accessible by clients using either the NFS or SMB
protocol. Files written to a file share become objects in Amazon S3, with the path as the key. There is a
one-to-one mapping between files and objects, and the gateway asynchronously updates the objects in
Amazon S3 as you change the files. Existing objects in the bucket appear as files in the file system, and
the key becomes the path. Objects are encrypted with Amazon S3–server-side encryption keys (SSE-S3).
All data transfer is done through HTTPS.
The service optimizes data transfer between the gateway and AWS using multipart parallel uploads or
byte-range downloads, to better use the available bandwidth. Local cache is maintained to provide low
latency access to the recently accessed data and reduce data egress charges. CloudWatch metrics provide
insight into resource use on the VM and data transfer to and from AWS. CloudTrail tracks all API calls.
With file gateway storage, you can do such tasks as ingesting cloud workloads to S3, performing backup
and archive, tiering and migrating storage data to the AWS Cloud. The following diagram provides an
overview of file storage deployment for Storage Gateway.
Volume Gateways
For volume gateways, you can use either cached volumes or stored volumes.
Topics
•Cached Volumes Architecture (p. 3)
•Stored Volumes Architecture (p. 5)
Cached Volumes Architecture
By using cached volumes, you can use Amazon S3 as your primary data storage, while retaining
frequently accessed data locally in your storage gateway. Cached volumes minimize the need to scale
your on-premises storage infrastructure, while still providing your applications with low-latency access
to their frequently accessed data. You can create storage volumes up to 32 TiB in size and attach to them
as iSCSI devices from your on-premises application servers. Your gateway stores data that you write to
these volumes in Amazon S3 and retains recently read data in your on-premises storage gateway's cache
and upload buffer storage.
Cached volumes can range from 1 GiB to 32 TiB in size and must be rounded to the nearest GiB. Each
gateway configured for cached volumes can support up to 32 volumes for a total maximum storage
volume of 1,024 TiB (1 PiB).
API Version 2013-06-30
3

AWS Storage Gateway User Guide
Volume Gateways
In the cached volumes solution, AWS Storage Gateway stores all your on-premises application data in
a storage volume in Amazon S3. The following diagram provides an overview of the cached volumes
deployment.
After you install the Storage Gateway software appliance—the VM—on a host in your data center and
activate it, you use the AWS Management Console to provision storage volumes backed by Amazon S3.
You can also provision storage volumes programmatically using the AWS Storage Gateway API or the
AWS SDK libraries. You then mount these storage volumes to your on-premises application servers as
iSCSI devices.
You also allocate disks on-premises for the VM. These on-premises disks serve the following purposes:
•Disks for use by the gateway as cache storage – As your applications write data to the storage
volumes in AWS, the gateway first stores the data on the on-premises disks used for cache storage.
Then the gateway uploads the data to Amazon S3. The cache storage acts as the on-premises durable
store for data that is waiting to upload to Amazon S3 from the upload buffer.
The cache storage also lets the gateway store your application's recently accessed data on-premises for
low-latency access. If your application requests data, the gateway first checks the cache storage for the
data before checking Amazon S3.
You can use the following guidelines to determine the amount of disk space to allocate for cache
storage. Generally, you should allocate at least 20 percent of your existing file store size as cache
storage. Cache storage should also be larger than the upload buffer. This guideline helps make sure
that cache storage is large enough to persistently hold all data in the upload buffer that has not yet
been uploaded to Amazon S3.
•Disks for use by the gateway as the upload buffer – To prepare for upload to Amazon S3, your
gateway also stores incoming data in a staging area, referred to as an upload buffer. Your gateway
uploads this buffer data over an encrypted Secure Sockets Layer (SSL) connection to AWS, where it is
stored encrypted in Amazon S3.
You can take incremental backups, called snapshots, of your storage volumes in Amazon S3. These
point-in-time snapshots are also stored in Amazon S3 as Amazon EBS snapshots. When you take a new
snapshot, only the data that has changed since your last snapshot is stored. You can initiate snapshots
on a scheduled or one-time basis. When you delete a snapshot, only the data not needed for any other
snapshots is removed. For information about Amazon EBS snapshots, see Amazon EBS Snapshots.
You can restore an Amazon EBS snapshot to a gateway storage volume if you need to recover a backup
of your data. Alternatively, for snapshots up to 16 TiB in size, you can use the snapshot as a starting
API Version 2013-06-30
4

AWS Storage Gateway User Guide
Volume Gateways
point for a new Amazon EBS volume. You can then attach this new Amazon EBS volume to an Amazon
EC2 instance.
All gateway data and snapshot data for cached volumes is stored in Amazon S3 and encrypted at rest
using server-side encryption (SSE). However, you can't access this data with the Amazon S3 API or other
tools such as the Amazon S3 Management Console.
Stored Volumes Architecture
By using stored volumes, you can store your primary data locally, while asynchronously backing up that
data to AWS. Stored volumes provide your on-premises applications with low-latency access to their
entire datasets. At the same time, they provide durable, offsite backups. You can create storage volumes
and mount them as iSCSI devices from your on-premises application servers. Data written to your stored
volumes is stored on your on-premises storage hardware. This data is asynchronously backed up to
Amazon S3 as Amazon Elastic Block Store (Amazon EBS) snapshots.
Stored volumes can range from 1 GiB to 16 TiB in size and must be rounded to the nearest GiB. Each
gateway configured for stored volumes can support up to 32 volumes and a total volume storage of 512
TiB (0.5 PiB).
With stored volumes, you maintain your volume storage on-premises in your data center. That is, you
store all your application data on your on-premises storage hardware. Then, using features that help
maintain data security, the gateway uploads data to the AWS Cloud for cost-effective backup and rapid
disaster recovery. This solution is ideal if you want to keep data locally on-premises, because you need to
have low-latency access to all your data, and also to maintain backups in AWS.
The following diagram provides an overview of the stored volumes deployment.
After you install the AWS Storage Gateway software appliance—the VM—on a host in your data center
and activated it, you can create gateway storage volumes. You then map them to on-premises direct-
attached storage (DAS) or storage area network (SAN) disks. You can start with either new disks or disks
already holding data. You can then mount these storage volumes to your on-premises application servers
as iSCSI devices. As your on-premises applications write data to and read data from a gateway's storage
volume, this data is stored and retrieved from the volume's assigned disk.
To prepare data for upload to Amazon S3, your gateway also stores incoming data in a staging area,
referred to as an upload buffer. You can use on-premises DAS or SAN disks for working storage. Your
gateway uploads data from the upload buffer over an encrypted Secure Sockets Layer (SSL) connection
API Version 2013-06-30
5

AWS Storage Gateway User Guide
Tape Gateways
to the AWS Storage Gateway service running in the AWS Cloud. The service then stores the data
encrypted in Amazon S3.
You can take incremental backups, called snapshots, of your storage volumes. The gateway stores these
snapshots in Amazon S3 as Amazon EBS snapshots. When you take a new snapshot, only the data that
has changed since your last snapshot is stored. You can initiate snapshots on a scheduled or one-time
basis. When you delete a snapshot, only the data not needed for any other snapshot is removed.
You can restore an Amazon EBS snapshot to an on-premises gateway storage volume if you need to
recover a backup of your data. You can also use the snapshot as a starting point for a new Amazon EBS
volume, which you can then attach to an Amazon EC2 instance.
Tape Gateways
Tape Gateway offers a durable, cost-effective solution to archive your data in the AWS Cloud. With its
virtual tape library (VTL) interface, you use your existing tape-based backup infrastructure to store data
on virtual tape cartridges that you create on your tape gateway. Each tape gateway is preconfigured with
a media changer and tape drives. These are available to your existing client backup applications as iSCSI
devices. You add tape cartridges as you need to archive your data.
The following diagram provides an overview of tape gateway deployment.
The diagram identifies the following tape gateway components:
•Virtual tape – A virtual tape is like a physical tape cartridge. However, virtual tape data is stored in
the AWS Cloud. Like physical tapes, virtual tapes can be blank or can have data written on them. You
can create virtual tapes either by using the Storage Gateway console or programmatically by using the
Storage Gateway API. Each gateway can contain up to 1500 tapes or up to 1 PiB of total tape data at a
time. The size of each virtual tape, which you can configure when you create the tape, is between 100
GiB and 2.5 TiB.
API Version 2013-06-30
6

AWS Storage Gateway User Guide
Tape Gateways
•Virtual tape library (VTL) – A VTL is like a physical tape library available on-premises with robotic
arms and tape drives. Your VTL includes the collection of stored virtual tapes. Each tape gateway
comes with one VTL.
The virtual tapes that you create appear in your gateway's VTL. Tapes in the VTL are backed up by
Amazon S3. As your backup software writes data to the gateway, the gateway stores data locally and
then asynchronously uploads it to virtual tapes in your VTL—that is, Amazon S3.
•Tape drive – A VTL tape drive is analogous to a physical tape drive that can perform I/O and seek
operations on a tape. Each VTL comes with a set of 10 tape drives, which are available to your
backup application as iSCSI devices.
•Media changer – A VTL media changer is analogous to a robot that moves tapes around in a physical
tape library's storage slots and tape drives. Each VTL comes with one media changer, which is
available to your backup application as an iSCSI device.
•Archive – Archive is analogous to an offsite tape holding facility. You can archive tapes from your
gateway's VTL to the archive. If needed, you can retrieve tapes from the archive back to your gateway's
VTL.
•Archiving tapes – When your backup software ejects a tape, your gateway moves the tape to the
archive for long-term storage. The archive is located in the AWS Region in which you activated the
gateway. Tapes in the archive are stored in the virtual tape shelf (VTS). The VTS is backed by S3
Glacier or S3 Glacier Deep Archive, low-cost storage service for data archiving, backup, and long-
term data retention.
•Retrieving tapes – You can't read archived tapes directly. To read an archived tape, you must first
retrieve it to your tape gateway either by using the Storage Gateway console or by using the Storage
Gateway API. When you retrieve a tape that is archived in GLACIER, it becomes available in your VTL
in about three to five hours after you start retrieval. When you retrieve a tape that is archived in
DEEP_ARCHIVE, it becomes available in your VTL in about 12 hours after you start retrieval.
After you deploy and activate a tape gateway, you mount the virtual tape drives and media changer on
your on-premises application servers as iSCSI devices. You create virtual tapes as needed. Then you use
your existing backup software application to write data to the virtual tapes. The media changer loads
and unloads the virtual tapes into the virtual tape drives for read and write operations.
Allocating Local Disks for the Gateway VM
Your gateway VM needs local disks, which you allocate for the following purposes:
•Cache storage – The cache storage acts as the durable store for data that is waiting to upload to
Amazon S3 from the upload buffer.
If your application reads data from a virtual tape, the gateway saves the data to the cache storage. The
gateway stores recently accessed data in the cache storage for low-latency access. If your application
requests tape data, the gateway first checks the cache storage for the data before downloading the
data from AWS.
•Upload buffer – The upload buffer provides a staging area for the gateway before it uploads the data
to a virtual tape. The upload buffer is also critical for creating recovery points that you can use to
recover tapes from unexpected failures. For more information, see You Need to Recover a Virtual Tape
from a Malfunctioning Tape Gateway (p. 335).
As your backup application writes data to your gateway, the gateway copies data to both the cache
storage and the upload buffer. It then acknowledges completion of the write operation to your backup
application.
For guidelines on the amount of disk space to allocate for the cache storage and upload buffer, see
Deciding the Amount of Local Disk Storage (p. 220).
API Version 2013-06-30
7

AWS Storage Gateway User Guide
Pricing
AWS Storage Gateway Pricing
For current information about pricing, see Pricing on the AWS Storage Gateway details page.
Plan Your Storage Gateway Deployment
By using the AWS Storage Gateway software appliance, you can connect your existing on-premises
application infrastructure with scalable, cost-effective AWS cloud storage that provides data security
features.
To deploy Storage Gateway, you first need to decide on the following two things:
1. Your storage solution – Choose from one of the following storage solutions:
•File Gateway – You can use a file gateway to ingest files to Amazon S3 for use by object-based
workloads and for cost-effective storage for traditional backup applications. You can also use it to
tier on-premises file storage to S3. You can cost-effectively and durably store and retrieve your on-
premises objects in Amazon S3 using industry-standard file protocols.
•Volume Gateway – Using volume gateways, you can create storage volumes in the AWS Cloud.
Your on-premises applications can access these as Internet Small Computer System Interface (iSCSI)
targets. There are two options—cached and stored volumes.
With cached volumes, you store volume data in AWS, with a small portion of recently accessed data
in the cache on-premises. This approach enables low-latency access to your frequently accessed
dataset. It also provides seamless access to your entire dataset stored in AWS. By using cached
volumes, you can scale your storage resource without having to provision additional hardware.
With stored volumes, you store the entire set of volume data on-premises and store periodic point-
in-time backups (snapshots) in AWS. In this model, your on-premises storage is primary, delivering
low-latency access to your entire dataset. AWS storage is the backup that you can restore in the
event of a disaster in your data center.
For an architectural overview of volume gateways, see Cached Volumes Architecture (p. 3) and
Stored Volumes Architecture (p. 5).
•Tape Gateway – If you are looking for a cost-effective, durable, long-term, offsite alternative for
data archiving, deploy a tape gateway. With its virtual tape library (VTL) interface, you can use
your existing tape-based backup software infrastructure to store data on virtual tape cartridges
that you create. For more information, see Supported Third-Party Backup Applications for a
Tape Gateway (p. 22). When you archive tapes, you don't worry about managing tapes on
your premises and arranging shipments of tapes offsite. For an architectural overview, see Tape
Gateways (p. 6).
2. Hosting option – You can run Storage Gateway either on-premises as a VM appliance, or as hardware
appliance or in AWS as an Amazon EC2 instance. For more information, see Requirements (p. 10).
If your data center goes offline and you don't have an available host, you can deploy a gateway on an
EC2 instance. Storage Gateway provides an Amazon Machine Image (AMI) that contains the gateway
VM image.
Additionally, as you configure a host to deploy a gateway software appliance, you need to allocate
sufficient storage for the gateway VM.
Before you continue to the next step, make sure that you have done the following:
1. For a gateway deployed on-premises, you chose the type of host, VMware ESXi Hypervisor or
Microsoft Hyper-V. and set it up. For more information, see Requirements (p. 10). If you deploy
API Version 2013-06-30
8

AWS Storage Gateway User Guide
Plan Your Gateway Deployment
the gateway behind a firewall, make sure that ports are accessible to the gateway VM. For more
information, see Requirements (p. 10).
2. For a tape gateway, you have installed client backup software. For more information, see Supported
Third-Party Backup Applications for a Tape Gateway (p. 22).
API Version 2013-06-30
9

AWS Storage Gateway User Guide
Sign Up for AWS Storage Gateway
Getting Started
In this section, you can find instructions about how to get started with AWS Storage Gateway. To get
started, you first sign up for AWS. If you are a first-time user, we recommend that you read the regions
and requirements section.
Topics
•Sign Up for AWS Storage Gateway (p. 10)
•AWS Regions (p. 10)
•Requirements (p. 10)
•Accessing AWS Storage Gateway (p. 23)
Sign Up for AWS Storage Gateway
To use AWS Storage Gateway, you need an AWS account that gives you access to all AWS resources,
forums, support, and usage reports. You aren't charged for any of the services unless you use them. If
you already have an AWS account, you can skip this step.
To sign up for AWS account
1. Open https://portal.amazonaws.cn/billing/signup.
2. Follow the online instructions.
Part of the sign-up procedure involves receiving a phone call and entering a verification code on the
phone keypad.
For information about pricing, see AWS Storage Gateway Pricing on the AWS Storage Gateway detail
page.
AWS Regions
AWS Storage Gateway stores volume, snapshot, tape, and file data in the AWS Region in which your
gateway is activated. File data is stored in the AWS Region where your Amazon S3 bucket is located. You
select an AWS Region at the upper right of the AWS Storage Gateway Management Console before you
start deploying your gateway.
• Storage Gateway—For supported AWS Regions and a list of AWS service endpoints you can use with
Storage Gateway, see Regions and Endpoints in the AWS General Reference.
Note
Tape gateway is not available in the South America (São Paulo) Region.
• AWS Storage Gateway Hardware Appliance—For supported AWS Regions you can use with the
hardware appliance, see AWS Storage Gateway Hardware Appliance Regions in the AWS General
Reference.
Requirements
Unless otherwise noted, the following requirements are common to all gateway configurations.
API Version 2013-06-30
10

AWS Storage Gateway User Guide
Hardware and Storage Requirements
Topics
•Hardware and Storage Requirements (p. 11)
•Network and Firewall Requirements (p. 12)
•Supported Hypervisors and Host Requirements (p. 20)
•Supported NFS Clients for a File Gateway (p. 21)
•Supported SMB Clients for a File Gateway (p. 21)
•Supported File System Operations for a File Gateway (p. 21)
•Supported iSCSI Initiators (p. 22)
•Supported Third-Party Backup Applications for a Tape Gateway (p. 22)
Hardware and Storage Requirements
In this section, you can find information about the minimum hardware and settings for your gateway
and the minimum amount of disk space to allocate for the required storage. For information about best
practices for file gateway performance, see Performance Guidance for File Gateways (p. 285).
Hardware Requirements for On-Premises VMs
When deploying your gateway on-premises, you must make sure that the underlying hardware on which
you deploy the gateway VM can dedicate the following minimum resources:
• Four virtual processors assigned to the VM.
• 16 GiB of reserved RAM assigned to the VM.
• 80 GiB of disk space for installation of VM image and system data.
For more information, see Optimizing Gateway Performance (p. 287). For information about how your
hardware affects the performance of the gateway VM, see AWS Storage Gateway Limits (p. 395).
Requirements for Amazon EC2 Instance Types
When deploying your gateway on Amazon EC2, the instance size must be at least xlarge for your
gateway to function. However, for the compute-optimized instance family the size must be at least
2xlarge. Use one of the following instance types recommended for your gateway type.
Recommended for file gateway types
• General-purpose instance family— m4 or m5 instance type.
• Compute-optimized instance family— c4 or c5 instance types. Select the 2xlarge instance size or
higher to meet the required RAM requirements.
• Memory-optimized instance family—r3 instance types.
• Storage-optimized instance family— i3 instance types.
Note
When you launch your gateway in EC2, and the instance type you’ve selected supports
ephemeral storage, the disks will be listed automatically. To learn more about Amazon EC2
instance storage, see here. Note that application writes are stored in the cache synchronously,
and then asynchronously uploaded to durable storage in Amazon S3. If the ephemeral
storage is lost because an instance stops before the upload is complete, then the data that
still resides in cache and has not yet written to S3 can be lost. Before you stop the instance
that hosts the gateway make sure the CachePercentDirty CloudWatch metric is 0. For more
information about monitoring metrics for your storage gateway, see storage gateway metrics
and dimensions.
API Version 2013-06-30
11

AWS Storage Gateway User Guide
Network and Firewall Requirements
If you have more than 5 million objects in your Amazon S3 bucket and you are using a General
Purposes SSD volume, a minimum root EBS volume of 350 GiB is needed for acceptable
performance of your gateway during start up. For information about how to increase your
volume size, see Modifying an EBS Volume from the Console.
Recommended for cached volumes and tape gateway types
• General-purpose instance family—m4 or m5 instance types. We don't recommend using the
m4.16xlarge instance type.
• Compute-optimized instance family—c4 or c5 instance types. Select the 2xlarge instance size or
higher to meet the required RAM requirements.
• Storage-optimized instance family—d2, i2, or i3 instance types
Note
When you create any gateway type using the c4 or m4 instance type, it can't be changed to the
c5 or m5 instance type. For information about how to upgrade your instance to the c5 or m5
instance type, see You Want Your File Gateway to Use a C5 or M5 EC2 Instance Type Instead of
C4 or M4 (p. 324).
Storage Requirements
In addition to 80 GiB disk space for the VM, you also need additional disks for your gateway.
The following table recommends sizes for local disk storage for your deployed gateway.
Gateway Type Cache
(Minimum)
Cache
(Maximum)
Upload Buffer
(Minimum)
Upload Buffer
(Maximum)
Other
Required Local
Disks
File gateway 150 GiB 16 TiB — — —
Cached volume
gateway
150 GiB 16 TiB 150 GiB 2 TiB —
Stored volume
gateway
— — 150 GiB 2 TiB 1 or more for
stored volume
or volumes
Tape gateway 150 GiB 16 TiB 150 GiB 2 TiB —
Note
You can configure one or more local drives for your cache and upload buffer, up to the
maximum capacity.
When adding cache or upload buffer to an existing gateway, it's important to create new disks
in your host (hypervisor or Amazon EC2 instance). Don't change the size of existing disks if the
disks have been previously allocated as either a cache or upload buffer.
For information about gateway limits, see AWS Storage Gateway Limits (p. 395).
Network and Firewall Requirements
Your gateway requires access to the internet, local networks, Domain Name Service (DNS) servers,
firewalls, routers, and so on. Following, you can find information about required ports and how to allow
access through firewalls and routers.
API Version 2013-06-30
12

AWS Storage Gateway User Guide
Network and Firewall Requirements
Note
In some cases, you might deploy AWS Storage Gateway on Amazon EC2 or use other types of
deployment (including on-premises) with network security policies that restrict AWS IP address
ranges. In these cases, your gateway might experience service connectivity issues when the
AWS IP range values changes. The AWS IP address range values that you need to use are in the
Amazon service subset for the AWS Region that you activate your gateway in. For the current IP
range values, see AWS IP Address Ranges in the AWS General Reference.
Topics
•Port Requirements (p. 13)
•Networking and Firewall Requirements for the AWS Storage Gateway Hardware Appliance (p. 17)
•Allowing AWS Storage Gateway Access Through Firewalls and Routers (p. 19)
•Configuring Security Groups for Your Amazon EC2 Gateway Instance (p. 20)
Port Requirements
AWS Storage Gateway requires certain ports to be allowed for its operation. The following illustrations
show the required ports that you must allow for each type of gateway. Some ports are required by all
gateway types, and others are required by specific gateway types. For more information about port
requirements, see Port Requirements (p. 386).
Common ports for all gateway types
The following ports are common to all gateway types and are required by all gateway types.
Protocol Port Direction Source Destination How Used
TCP 443 (HTTPS) Outbound Storage
Gateway
AWS For
communication
from AWS
Storage
Gateway to the
AWS service
endpoint. For
information
about service
endpoints,
see Allowing
AWS Storage
Gateway
Access
Through
Firewalls and
Routers (p. 19).
TCP 80 (HTTP) Inbound AWS
Management
Console
Storage
Gateway
By local
systems
to obtain
the storage
gateway
activation key.
Port 80 is only
used during
activation of
the Storage
API Version 2013-06-30
13