What Is S3 Intelligent Tiering and When Should I Use It?

In the whirlwind of re:Invent 2018, AWS announced a number of new features and improvements, including new database options, S3 Batch operations, and machine learning services.

One of the announcements that flew under the radar was the introduction of S3 Intelligent-Tiering. S3 Intelligent-Tiering is a new storage class in S3 that is designed to save you money while still maintaining the performance you need.

In this post, we will first cover the basics of S3 Intelligent-Tiering: what it is, how it works, and when you should or shouldn’t use it. Then, we’ll do a cost comparison to see how S3 Intelligent-Tiering can save you money compared to other storage classes.

Everything about backing up to Amazon S3:

Let’s get started.

Background

Before we discuss what S3 Intelligent-Tiering is, let’s start with some background on S3 storage classes so you know why S3 Intelligent-Tiering is useful.

Amazon S3 is an object-based storage service provided by Amazon Web Services (AWS). It is used for a wide variety of storage needs, including saving server backups, serving static files for web pages, or keeping long-term archives for compliance purposes.

S3 provides different levels of availability and reliability through its different storage classes. Each storage class has a different cost structure and guarantees around how quickly you can access your data.

Further reading S3 Storage Classes Explained

For example, two of the more popular storage tiers are S3 Standard and S3 Standard Infrequent Access (or S3 Standard-IA). Both are designed to provide rapid access to your objects when requested. S3 Standard is for objects which are accessed frequently. The storage cost for the objects is fairly high, but there is a very low charge for accessing the objects.

Further reading Amazon S3 Pricing Explained

Conversely, S3 Standard-IA is for objects which are not expected to be accessed frequently. The storage cost is much cheaper than S3 Standard -- almost half the price -- but you are charged more heavily for accessing your objects.

Often, your objects will be accessed less frequently the older they get. Because of this, users use object lifecycle policies to move objects from one tier, such as S3 Standard, to a different tier, such as S3 Standard-IA, after a specified amount of time.

Why Use S3 Intelligent-Tiering?

Object lifecycle policies are helpful, but they’re a blunt instrument. Lifecycle policies apply to all objects in a given S3 prefix, but the access patterns for objects in a prefix can vary widely. Perhaps some objects are accessed frequently for months after creation, while others are rarely accessed after the first few days. This can result in unnecessary storage or access bills.

Here’s where S3 Intelligent-Tiering comes in. If you put objects in the S3 Intelligent-Tiering storage class, AWS will monitor and move your data on a per-object level to the proper storage tier. If your object hasn’t been accessed in 30 days, AWS will move it to the infrequent access storage tier. If the object is then accessed after being moved to infrequent access, AWS will move it back to the frequent access storage class for cheaper subsequent accesses.

Let’s see how this works with an example. Imagine you have three objects in S3 -- a hot object, a cool object, and a cold object.

The hot object is accessed daily over the course of a year. Because of this, S3 Intelligent-Tiering will keep it in the frequent access storage tier. You will pay slightly more for storage on this object, but you won’t have to pay the hefty cost for access that comes with the infrequent access tier.

In contrast, the cool object is not accessed daily at first. In fact, it isn’t accessed at all for the first three months of its life. For this object, S3 Intelligent-Tiering would shift it to the infrequent access tier after its first month to save on storage costs. Imagine the object is needed now after three months and is accessed once a week for the following six months. S3 Intelligent-Tiering would move it back to frequent access after the first access and would keep it there until it had another 30 day period of non-access.

Finally, imagine your cold object is accessed once in the first week and then never accessed again. S3 Intelligent-Tiering would move it to an infrequent access tier 30 days after its last access, saving you money on storage. The object would still be immediately accessible if needed, in contrast to something like the S3 Glacier storage class.

The big benefit of this is that you don’t need to manually track the access patterns of your objects on a granular basis. AWS will handle this for you in a way that can save you significant amounts on your storage bill.

Now that we know why S3 Intelligent-Tiering is useful, let’s look at the pricing details around S3 Intelligent-Tiering.

What Is the Pricing for S3 Intelligent-Tiering?

There are three main pricing axes for S3 Intelligent-Tiering:

• Storage
• Requests
• Object monitoring

Let’s look at each of these individually.

First, storage prices are equal to the S3 Standard and S3 Standard-IA storage prices within a given region. When an object is first placed in S3 Intelligent-Tiering, it will be in the frequent access tier and is charged at the S3 Standard rate ($0.023 per GB in US East).

If the object has not been accessed for 30 consecutive days, it will be moved to the infrequent access tier and charged at the S3 Standard-IA rate ($0.0125 per GB in US East).

The second pricing axis is on requests. S3 charges you based on the number of API requests and the amount of data returned by these requests.

Requests prices are much cheaper for S3 Standard than for S3 Standard-IA. The great part about S3 Intelligent-Tiering is that you pay S3 Standard request prices, even when your object is stored in an infrequent access tier.

By way of example, S3 Standard and S3 Intelligent-Tiering charge $0.0004 per 1000 GET requests, while S3 Standard-IA charges $0.001 per 1000 GET requests -- 2.5x as much. S3 Standard-IA also charges $0.01 per GB of data retrieved by a request, while there is no retrieval cost for S3 Standard or S3 Intelligent-Tiering.

Finally, S3 Standard and S3 Intelligent-Tiering have much cheaper rates for data returned by S3 Select. These storage classes charge $0.0007 per GB for data returned by S3 Select, while S3 Standard-IA charges $0.01 per GB -- 14 times as much!

One other great perk is that accessing your S3 Intelligent-Tiering data via S3 Select does not trigger a move from the infrequent access storage tier to the frequent access storage tier. This is a huge perk, as you can pay S3 Standard-IA storage costs but S3 Standard request costs for S3 Select.

The final price aspect of S3 Intelligent-Tiering is an object monitoring cost. AWS charges you $0.0025 per 1,000 objects in the S3 Intelligent-Tiering storage tier.

Now that we know the pricing around S3 Intelligent-Tiering, let’s see a few areas where it might not be a good fit.

When Shouldn’t I Use S3 Intelligent-Tiering?

S3 Intelligent-Tiering can save a nice amount on your AWS bill if it fits your use cases. However, it’s not right for all situations. Below are a few situations where S3 Intelligent-Tiering may not be for you.

• Predictable access patterns: If your objects have very predictable access patterns, you could handle them via object lifecycle rules rather than S3 Intelligent-Tiering. By doing this, you would avoid the monitoring charge of $0.0025 per 1,000 objects.
• Very small objects: If your objects are smaller than 128KB, they will never be moved from the frequent access tier to the infrequent access tier.
• Short-lived objects: S3 Intelligent-Tiering has a minimum storage duration charge of 30 days. If your objects are deleted before that time, you should use a different storage class.

In the next section, we’ll do some cost comparisons across the S3 storage classes.

S3 Intelligent-Tiering Cost Comparisons

In the previous section, we reviewed the S3 Intelligent-Tiering storage class and pricing. Let’s bring it to life with some cost comparisons

There are a lot of factors to consider with a cost comparison, so these will be necessarily simplified. Your results will differ based on your use case.

Non-Standard Access Patterns

For our first comparison, assume that you have a large number of objects under a particular prefix. Some are accessed frequently and some are not. However, it’s hard to distinguish between the two in a way that works for an object lifecycle policy.

Let’s use the following assumptions:

• 5000000 objects of 1MB each are placed into S3 on January 1.
• Each month, exactly half of the objects are modified via a PUT operation. Each modified object is modified five times per month.
• Each month, exactly half of the objects are read via a GET operation. Each accessed object is read 50 times per month.

With these assumptions, let’s break down how much it would cost for one year each of S3 Standard, S3 Standard-IA, and S3 Intelligent-Tiering.

For each comparison, we will use the US East region for pricing.

S3 Standard:

• Storage:

5000 GB X $0.023 per GB/month X 12 months = $1380

 

• PUT requests:

2,500,000 objects X 5 requests X 12 months = 150000000 requests

$0.005 per 1000 requests = $750

 

• GET requests:

2,500,000 objects X 50 requests X 12 months = 1500000000 requests

0.0004 per 1,000 requests = $600

Total cost: $2730

S3 Standard-IA

• Storage:

5000 GB X $0.0125 per GB/month X 12 months = $750

 

• PUT requests:

2,500,000 objects X 5 requests X 12 months = 150000000 requests

$0.01 per 1000 requests = $1500

 

• GET requests:

2,500,000 objects X 50 requests X 12 months = 1500000000 requests

$0.001 per 1,000 requests = $1500

 

• Data retrieval:

2,500,000 objects X 50 requests X 12 months = 1500000000 requests

1500000000 requests at 1MB per request = 1,500,000 GB retrieved

1,500,000 GB retrieved at $0.01 per GB = $15000

Total cost: $18750

S3 Intelligent-Tiering

• Storage:

First month (all at frequent access tier): 5000 GB X $0.023 per GB/month = $115

Eleven months for half at frequent access: 2500GB X $0.023 per GB/month X 11 months = $632.50

Eleven months for half at infrequent access: 2500GB X $0.0125 per GB/month X 11 months = $343.75

 

• PUT requests:

2,500,000 objects X 5 requests X 12 months = 150000000 requests

$0.005 per 1000 requests = $750

 

• GET requests:

2,500,000 objects X 50 requests X 12 months = 1500000000 requests

$0.0004 per 1,000 requests = $600

 

• Object monitoring:

5000000 objects at $0.0025 per 1,000 objects = $12.50

Total cost: $2453.75

S3 Select

For our second use case, let’s imagine we’re heavy users of S3 Select.

Using S3 Intelligent-Tiering with S3 Select can be a great way to save money, as using S3 Select does not transfer your S3 Intelligent-Tiering objects from infrequent access to the frequent access storage tier. This means you get the low S3 Standard-IA storage rates combined with low S3 Select retrieval rates of S3 Standard.

Let’s use the following assumptions:

• We have 10000 objects of 100MB each (1000 GB) of data in S3 for a year.
• We access each object with S3 Select twice daily. Each S3 Select request scans the full object and returns about 10% of the data.

For each comparison, we will use the US East region for pricing.

S3 Standard:

• Storage:

1000 GB X $0.023 per GB/month X 12 months = $276

 

• SELECT requests:

10000 objects X 2 requests X 365 days = 7,300,000 requests

$0.0004 per 1,000 requests = $2.92

 

• Data returned by S3 Select:

100 GB X 2 requests X 365 days = 73,000 GB returned

73,000 GB returned at $0.0007 per GB = $51.10

 

• Data scanned by S3 Select:

1000 GB X 2 requests X 365 days = 730,000 GB scanned

730,000 GB scanned at $0.002 per GB = $1460

Total cost: $1790.02

S3 Standard-IA

• Storage:

1000 GB X $0.0125 per GB/month X 12 months = $150

 

• SELECT requests:

10000 objects X 2 requests X 365 days = 7,300,000 requests

$0.001 per 1,000 requests = $7.30

 

• Data returned by S3 Select:

100 GB X 2 requests X 365 days = 73,000 GB returned

73,000 GB returned at $0.01 per GB = $730

 

• Data scanned by S3 Select:

1000 GB X 2 requests X 365 days = 730,000 GB scanned

730,000 GB scanned at $0.002 per GB = $1460

Total cost: $2347.30

S3 Intelligent-Tiering

• Storage:

First month (frequent access tier): 1000 GB X $0.023 per GB/month = $23

Eleven months at infrequent access: 1000GB X $0.0125 per GB/month X 11 months = $150

 

• SELECT requests:

10000 objects X 2 requests X 365 days = 7,300,000 requests

$0.0004 per 1,000 requests = $2.92

 

• Data returned by S3 Select:

100 GB X 2 requests X 365 days = 73,000 GB returned

73,000 GB returned at $0.0007 per GB = $51.10

 

• Data scanned by S3 Select:

1000 GB X 2 requests X 365 days = 730,000 GB scanned

730,000 GB scanned at $0.002 per GB = $1460

 

• Object monitoring:

10000 objects at $0.0025 per 1,000 objects = $0.02

Total cost: $1687.04

In both of these examples, S3 Intelligent-Tiering ended up being the cheapest option.

Your results may vary depending on how frequently you access your objects, how consistent your access patterns are, and whether you’re using S3 Select.

Conclusion

In this post, we learned about S3 Intelligent-Tiering, a new S3 storage class that AWS announced at re:Invent 2018. S3 Intelligent-Tiering helps you smartly manage your objects between frequent access and infrequent access storage tiers based on each object’s access patterns.

We reviewed why S3 Intelligent-Tiering can be useful, and we broke down its pricing components. Further, we looked at times when S3 Intelligent-Tiering would not be a good fit for you. Finally, we did two price comparisons across storage classes to see how S3 Intelligent-Tiering compares.