Upload to Glacier with Amazon S3 Lifecycle Rules

  

There are a number of storage classes for every need and budget in Amazon Web Services cloud storage. The cheapest method to store the backup for a long time is Amazon Glacier. However, there are two different ways to upload files to that storage: direct and using the lifecycle policy. In this article, we will demonstrate why the lifecycle policy method is more practical.  

Why Uploading to Amazon Glacier?

AWS Glacier is the most cost-effective storage solution for the infrequently accessed data Amazon Web Services has to offer. You can store any amounts of data for $0.004 per gigabyte per month, so it proves itself cheaper than the on-premises solutions.

It is also secure: by default, only you can access your data. Additionally, you can use the IAM policies to define the additional circle of users who can access your data. All the uploaded data is encrypted on the server side using 256-bit Advanced Encryption Standard (AES-256). Besides, you can also set up the client-side encryption on your side (check out how to setup client-side encryption with CloudBerry Backup here).

With that said, Amazon Glacier is one of the best options for the data archival.

What is Amazon S3 Lifecycle Policies?

AWS lifecycle rules are a set of options aimed to manage the data stored within the S3. It allows to move or to delete the objects after the certain number of days by configuring your own lifecycle rules.

Previously, the lifecycle configuration could only be implemented by creating an XML file, but nowadays it is possible to create it using a pretty intuitive web GUI within the AWS management console.

Upload to Glacier through S3 Lifecycle Policies

Amazon Glacier offers a few data upload options:

  • AWS control web interface.
  • AWS (SDK) software or Amazon Glacier API.
  • Amazon S3 allows setting lifecycle management rules to archive unused data from Amazon S3 to Glacier.
  • Own solutions of AWS (APN) network partners that are already embedded in Glacier.

Let’s consider archiving using Lifecycle Policy in S3 in detail.

The cost of data storage in S3 or Glacier is just a part of the price for these services. An extra pay is charged for requests for data stored in S3 and download traffic. In Glacier, this list also includes the cost of data retrieval and the price also depends on the retrieval speed.

The point is that all information in Glacier is stored on reasonably priced tape drives, but this information is copied to more expensive media before its transfer to the user. So, when data is archived directly to Glacier, it takes half a day to index it and 3-5 hours more to make it available for retrieval (if standard retrieval is used). If data is uploaded to S3, all files will be instantly available.

It is possible to store several data versions and manage their lifecycle in S3. Upon expiration, data is deleted or it can be transferred to Glacier. If you set a storage class equal to 0 days, information will be immediately sent to Glacier. It is of use when information is rarely accessed in everyday life but its storage life is limited.

Though, it might seem that uploading data to S3 first and go with it to Glacier afterward might be more expensive, AWS has ensured that this exact scenario leads to no more expenses than direct Glacier upload.

Applying AWS Lifecycle Rules via AWS Management Console

The lifecycle rules can be set up in the AWS management console directly.

Here is our step-by-step guide:

  1. Open  your AWS console, click “Services” in the right upper corner and select “S3”
    AWS console S3 screenshot
  2. Select the bucket you want to set up lifecycle rule for and click on it. You will see the following menu on the right to it.
    S3 bucket screenshot
  3. Click “Management”, and then “ Add lifecycle rule” button.
    Applying lifecycle rule to the S3 bucket
  4. Select the name and add the tags/prefixes for your lifecycle rule if necessary.
    amazon s3 lifecycle policy naming
  5. Check the versions of data you want to be transferred with the lifecycle rule, then select “Transition to Amazon Glacier after” and specify the required amount of days (note that if you set different setting from "0", the costs of storage and file transfer will be bigger).
    Transition setup for the Amazon S3 lifecycle rule
  6. After that comes the “Expiration” step. Since the lifecycle is set up for the bucket in S3, you can just skip it and click “Next”.
    Expiration setup for the Amazon S3 lifecycle rule
  7. See the “Review” step and click “Save”.
    Amazon S3 lifecycle rule review step

Automating AWS Lifecycle Rules with CloudBerry Backup

Lifecycle policy can also be set up in a couple of clicks within CloudBerry Backup.

After setting up the backup plan and cloud storage in S3, you should enable Lifecycle Policy in CloudBerry Backup. Go to Tools and click Lifecycle Policy or use the same parameters in the left pane. For the step-by-step guide please check out this article.

In the dialog box select the source of data that will be transferred to the storage.

As a result of such transfer, the files would first get to S3 and after that would get to Glacier in 60 days. If you set 0 days - they will be archived automatically.

Please keep in mind that if you set different setting from "0", the costs of storage and file transfer will be bigger. AWS will charge additional costs for storing files in S3 storage.

This approach has an advantage over the pure Glacier transfer. With the pure Glacier transfer, you have to wait 3-5 hours for the initial inventory to complete. With the S3-Glacier lifecycle policy set to 0 days, the inventory happens as soon as the files are in the storage.

Summary

The need to use Amazon S3 and Glacier individually or in combination depends on the objectives of each separate project. We prepared a comparison table for you to evaluate expenses for both methods of data archiving:

Amazon S3 Amazon Glacier
Data transfer to Amazon (per GB) $0.000 $0.000
Data transfer from S3 to Glacier (1,000 requests) $0.055 X
PUT (1,000 requests) $0.0055 Free
COPY (1,000 requests) $0.0055 Free
POST (1,000 requests) $0.0055 Free
LIST (1,000 requests) $0.0055 Free
GET (10,000 requests) $0.0044 Free
UPLOAD (1,000 requests) $0.0044 $0.055
RETRIEVAL (1,000 requests) $0.0044 $0.055
Other requests $0.0044 Free
Data storage (first 50 TB / month) $0.025 per GB $0.0045 per GB
Data access rate Instantly Price
Data transfer from Amazon Price Price

Amazon Web Services provides a possibility to use 5 GB of S3 storage (unfortunately without Glacier) for FREE for 12 months – this time period is enough to get acquainted with system features. We recommend trying CloudBerry Backup available for FREE for 15 days to assess archiving features in full.

You are welcome to share your experience and ask questions in the comments section below.

Related Links:

 

Featured Product