Aws Cost Optimization
Aws Cost Optimization
Aws Cost Optimization
Save
Big
on Your AWS Bill
A Practical Guide to Cloud Cost Optimization
www.n2ws.com
Overview
The public cloud offers many advantages over the traditional model of IT – through low-cost
infrastructure, fast provisioning and access to cutting-edge cloud-based technologies.
But only if you adapt to the new on-demand pay-as-you-go model of the cloud and learn how to manage
your resources efficiently.
You can miss out on opportunities for significant discounts on your monthly cloud bills. And you may not
choose the most cost-efficient solution for your applications.
This paper provides you with a strategy for reducing your costs on leading cloud platform AWS. But it also
serves as a practical guide to cloud cost optimization – with diagrams, screenshots and links to additional
resources, so you can start taking cost-saving measures straight away.
Let's start by looking at the optimization tools you'll need for the job.
6. Adopt an RI Strategy 26
Use it to identify the services that are costing you the most, as these are the low-hanging fruit where you
can make the biggest savings.
You can also use it to check whether you're making good use of any Reserved Instances (RIs) – by
comparing the total usage available with the total number of hours you actually consume.
What's more, you can track spending and monitor the progress of your cost-optimization regime.
In the case of consolidated bills, line items are broken down by account, product and region. Bills for
individual accounts are simply organized by product and region.
AWS CloudWatch
An infrastructure monitoring service that provides insights for managing your
AWS resources.
You can use it to visualize log data from your EC2 deployments. You can monitor estimated charges for
individual AWS services and set up email alerts that notify you when costs exceed a certain threshold.
You can also create alarms and take automated actions whenever resource utilization reaches predefined
thresholds.
CloudWatch provides CPU utilization data for EC2 instances by default. But you can also set up custom
metrics to monitor memory utilization.
Cost Explorer includes a suite of default reports that help you quickly compare costs between different
products, accounts or regions.
You can also dig deeper by filtering and grouping your data as well as creating customized reports.
It also makes RI recommendations based on your last month's instance usage patterns and alerts you to
RIs that have recently expired or due to expire in the next 30 days.
The solution is based on a CloudFormation template, which launches a series of components that form
the application stack. Setup requires technical knowledge of working on the AWS platform.
It filters your data into a series of predefined age groups, starting from objects less than 15 days old
through to objects 730 days and older, and presents a visualization for each group showing how much
data you stored during that time period and how much of it you retrieved.
Storage Class Analysis is intended as a visual aid for making decisions on when to transition objects
from S3 Standard to the lower-cost S3 Standard – IA tier.
You can switch between different visualization formats, such as multi-line graphs, pie charts and bar
charts, and also create your own visualizations.
The solution helps you reduce your monthly bills by only running instances during the periods you actually
need them. For example, Dev/Test resources are rarely used outside regular 9–5 hours.
So you can define a schedule that closes them down in the evenings, at weekends and during public
holidays. Setup is relatively simple but requires some technical knowledge of working on the AWS platform.
AWS Lambda
A fully managed event-driven serverless service, which you can use to deploy code that
runs in response to CloudWatch events.
This gives you the scope to automate some of your cost-optimization processes. For example, you could set
up a Lambda function to automatically find and delete unused EBS volumes or delete snapshots when they
reach the end of a specified retention period.
It is particularly useful if you're relatively new to Spot Instances, as it can help you identify low-risk
instance types for applications that don't yet have sufficient flexibility or fault tolerance to cope with
higher levels of interruption.
Tag Editor
Tags facilitate visibility over your infrastructure by helping you quickly identify who is
responsible for each of your resources what they use them for. They also play an
important role in lifecycle management.
You can tag resources as and when you create them. You can also manage tags for an individual resource
within the associated AWS console for that resource.
But if you need to centrally manage a number of tags at any one time, it's easier to use the AWS Tag
Editor. With the Tag Editor, you can search through your inventory of taggable resources and then
change, remove or apply new tags.
You can perform many of your cost-optimization tasks quickly and efficiently by running a single command
or script. For example, you can view Spot Instance pricing history, scale elastic EBS volumes and delete
old EBS snapshots without leaving the same terminal session. Although the AWS CLI is a very powerful
tool, it takes a lot of time and learning to reach a useful level of proficiency.
Third-Party Tools
In addition to Amazon's own set of cost-management solutions, you should also con-
sider third-party tools, which typically offer additional functionality, such as enhanced
best-practice checks, multi-cloud capabilities and higher levels of automation.
These include conventional cost-optimization tools, which provide features such as right-sizing, RI recom-
mendations and budget reporting.
But other types of third-party software can also help you reduce your monthly AWS bills. For example,
data management tools can help you take advantage of the most cost-efficient storage methods by giving
you the flexibility and control to move data around your AWS environment. Moreover, a solid tool will
allow you to stop and start Amazon EC2 instances at predefined times to reduce the unnecessary costs of
idle servers.
Nurture Cost
Awareness
"The ability to scale up and down on demand has allowed resource procurement to
transition from being solely owned by the finance team to being the responsibility of
stakeholders across IT, engineering, finance, and beyond. This democratization of resource
procurement has initiated an ever-growing group of cost-conscious stakeholders."
With the on-demand utility model of the cloud, it's easy to provision computing resources. But, at
the same time, it's just as easy to leave them running when you no longer need them.
Not only that, but you no longer need large upfront investments to fund new infrastructure. As a
result, where IT spending was previously the sole responsibility of central finance, it's now
devolved to a much broader spectrum of departments.
This calls for a change in culture towards a far more cost-aware mindset. It requires governance
over and accountability for cloud usage. But it also requires a cost-aware approach to application
architecture.
Enforcing tagging policies so everyone knows what each resource is used for.
Accurately allocating costs to individual business units.
Regularly reviewing your AWS costs and the success of your cloud transformation.
Cost-Aware Architecture
Your application design should take advantage of the most cost-efficient AWS services
wherever possible.
You should also make full use of the cloud's elastic features by breaking your applications up into smaller,
individual, loosely coupled components.
This will help you scale your applications more efficiently, as you can increase and decrease the capacity
allocated to each component independently, providing finer grained matching of discrete application
functions to resource requirements.
Similarly, you can deploy different components to AWS instances in different families. That way, you can
achieve more granular control by matching each component to an instance type with just the right mix of
CPU and memory.
What's more, you're better able to scale your applications horizontally. This is the method whereby you
deploy each component to a cluster of virtual machines (VMs) and scale in or out by removing or adding
machines. It is a cost-efficient way to scale your infrastructure because you can use a network of smaller
instances and adjust it by smaller, more precise increments.
Ditch Unused
Resources
Now we're onto the cost-saving measures that can make an immediate impact on
your AWS bill. The best place to start is to clean up all unused resources, as these
eat away at your budgets without serving any purpose.
The following core AWS services should be the first on your agenda:
EC2 Instances
Idle EC2 instances are amongst the biggest causes of inflated AWS bills.
But you can prevent unnecessary waste by using CloudWatch alarms to automatically stop or terminate
instances that have been idle for longer than a specified period. You can only terminate machines backed
by Instance Store. But you can either stop or terminate instances backed by EBS – depending on whether
you're likely to need them again.
For example, you may want to stop instances where you only need them during normal working hours.
However, you may want to terminate others altogether if they remain unused for a longer period of time.
This gives you the scope to automate some of your cost-optimization processes. For example, you could set
up a Lambda function to automatically find and delete unused EBS volumes or delete snapshots when they
reach the end of a specified retention period. Lambda is also a component of the automated AWS Instance
Scheduler deployment.
EBS Volumes
By default, when you terminate an EBS-backed instance, only the EBS volumes that were
attached at launch are deleted. Any additional volumes you attached after you launched
the instance continue to run.
This can cost you big if you leave unattached EBS volumes running for longer than necessary –
especially where you use SSD-backed volumes, which cost more than twice as much as their
HDD counterparts.
You can manually find and remove unattached EBS volumes in the:
Don't forget to take a snapshot of your volume before you terminate it. This gives you the option to
recover it should you need it again in the future. You can take snapshots in the EC2 dashboard from the
same menu you use to delete volumes.
Although snapshots are incremental, your first backup captures the entire volume. What's more, if your
retention period is long, you may need to store as much incremental data as the initial snapshot.
In other words, your snapshots may be costing you far more than you think. So, unless you want to use
your orphaned snapshots to create new EBS volumes, you should delete them as part of your
cost-optimization routine. You can manually find and remove orphan snapshots in the:
Elastic IPs
Public Internet addresses are a limited resource. So, to discourage waste, AWS restricts
each account to five Elastic IP addresses per region.
It also levies a small hourly charge on addresses that aren't associated with a running instance. The same
goes for Elastic IPs associated with a stopped instance or unattached network interface. To keep costs
down, you should release Elastic IPs if you no longer need them. You can manually find and release
unattached Elastic IPs in the:
Fine-Tune Your
Compute Resources
Size matters when it comes to cloud cost efficiency. If your instances provide far
more CPU and memory than your workloads need then you're simply pouring
money down the drain.
But if they don't provide enough then your application performance will suffer.
Fine-Tune Your Compute Resources
Size matters when it comes to cloud cost efficiency.
What capacity do you need to meet average consumption? And how much additional power do you need
to cope with peaks in demand?
The answers to these questions will help you determine the optimum utilization level of your instances.
But, as a rule of thumb, you should generally aim for 70–80% utilization if you want to minimize costs
without affecting latency.
But right-sizing your instances shouldn't end there. You can fine-tune your resources by switching to
instances with different performance characteristics.
The following table highlights common examples where a different instance family may be a better fit for
your application:
Memory-intensive workloads requiring a higher Memory-optimized family of instances (R5, R5a, R4,
proportion of memory X1e, X1, High Memory and z1d)
Small and moderate workloads requiring occasional Burstable Performance family of instances
additional capacity to meet peaks in demand (T2, T3 and T3a)
You should also collect and analyze usage statistics on a regular basis and review your EC2 capacity over
time. And, where possible, you should deploy your application to a fleet of smaller instances.
So, instead of scaling vertically by changing the size of your instance, you can scale your infrastructure
horizontally a little at a time by adding or removing lower-cost machines.
Baseline
Price / Hour Price / Hour
Name vCPUs Performance Memory
(Linux) (Windows)
/ vCPU
You can access statistics for CPU and memory consumption of a specific instance in the:
CloudWatch console
AWS CLI by running the get-metric-statistics command
If you're an Amazon RDS user, you should be aware that DB instances are decoupled from storage. So,
from time to time, you may need to switch to a different DB instance type to meet changing demand on
your database system.
Optimize Your
Storage
“Organizations tend to think of data storage as an ancillary service and do not optimize
storage after data is moved to the cloud. Many also fail to clean up unused storage and
let these services run for days, weeks, and even months at significant cost.”
This now makes it much easier to optimize your block storage capacity and manage costs. To monitor
the disk utilization of your EBS volumes, you'll need to publish custom usage metrics to CloudWatch.
You can modify the size of an EBS volume in the:
EBS Snapshots
In addition to removing orphan snapshots, you should maintain a snapshot retention
policy that properly balances your data backup requirements with cost.
Deleting an old snapshot won't always necessarily reduce your storage overhead. This is because it may
contain referenced data that's required to restore the volume. If this is the case then the data is preserved
and continues to rack up charges.
However, if a good proportion of the data on your volumes changes frequently then deleting old snapshots
will remove duplicate files and help to keep your storage costs down.
AWS compresses your snapshots and stores them in S3. So your snapshot costs may seem relatively
inexpensive. But the vendor uses a different pricing system from its regular S3 tiers. This is more than
twice the Standard S3 rate.
Not only that, but the buckets AWS uses to store snapshots are not publicly visible. So you cannot retrieve
them using the standard S3 methods.
Currently, the only way to retain snapshots at a lower cost is to use a third-party cloud data management
tool that's able to copy them to a standard S3 bucket.
Amazon S3
Amazon's object storage offering S3 is the cloud provider's most cost-efficient way to
store data. So, to keep your bills down, you should make use of it wherever possible.
This means limiting your use of more expensive services, such as EBS and EFS, to only those use cases
where they're strictly necessary. S3 is not only the least expensive storage solution, but it also
automatically scales with you. So you don't have to worry about wasting unused provisioned storage.
Standard
Low-latency storage option with 99.999999999% durability and 99.99% availability.
Designed for frequently accessed data.
Standard – IA
The same low latency and durability as S3 Standard but availability slightly lower at 99.9%. Standard
– IA is designed for infrequently accessed data, as storage charges are lower but data retrieval
charges higher.
Objects can exist in the same bucket as those set as S3 Standard. This means you can transition data
between classes without making application changes.
Glacier
The most economical storage option. However, data retrieval is slow and expensive, making it
suitable only for data archiving and long-term backups.
Efficient lifecycle management of your S3 objects is key to lowering your storage costs. So AWS provides a
number of tools to help you manage your buckets and take advantage of its different storage classes.
Storage Class Analysis is a simple data visualization tool that can play a useful role in your S3 optimization
workflow. Although it provides a basic overview of retrieval patterns, it may be all you need to decide at
what age your data is ready for transition between tiers.
It provides more detailed views of your storage retrieval patterns for fine-tuning your S3 lifecycle policies.
If you have Storage Class Analysis enabled, you can access Amazon QuickSight via the:
Dynamic scaling is a particularly effective cost-optimization technique, as it can take almost immediate
action to changes in demand. By contrast, it can take time to manually respond to scaling events – while, in
the meantime, your application can suffer performance issues or rack up unnecessary costs.
Minimum size
The baseline number of instances below which the size of your group should never fall.
Desired capacity
The number of instances you want to run based on your scaling policy. Your desired capacity moves up
and down dynamically as the load on your application changes. Desired capacity adjustments are
triggered by CloudWatch alarms.
Maximum size
The ceiling figure above which the number of instances in your group should never exceed.
Desired capacity
Maximum size
Top Tip
Auto Scaling is also available for other AWS services, including ECS, EC2 Spot Fleets,
Aurora, DynamoDB and EMR.
You can configure all of these resources using AWS Auto Scaling – a single unified interface that simplifies
Auto Scaling management across your services without having to navigate to other consoles.
Adopt an
RI Strategy
Reserved Instances are an alternative billing mechanism to the on-demand
pricing model, where you purchase a discounted usage credit for a specific type
of instance over a fixed term.
Adopt an RI Strategy
They can deliver savings of up to 75% compared with standard on-demand charges, where the maximum
discount you can expect depends on the following:
RI specifications
The instance type, operating system (OS), tenancy and Availability Zone (AZ).
RI term
Either one or three years
Payment option
All upfront, partial upfront or no upfront payment at all.
For maximum return on your RI purchases, your running instances should consume your usage credits
as much of the time as possible. This is because, during any period where you have no active matching
instances, your credit will remain unused and go to waste. This is why RIs tend to suit steady predict-
able workloads.
RIs are a more effective cost-optimization strategy when you use consolidated billing. This is because
an RI credit can apply to a matching instance in any of a company's linked accounts. However, the
account that buys the reservation is always first in line to the discount.
Another strategy for maximizing RI savings is to use them to cover the baseline capacity of an Auto
Scaling group.
Desired capacity
Maximum size
Use RIs to cover the capacity required for your minimum group size
As your Auto Scaling group will always run at or above your minimum number of instances, you can effec-
tively guarantee full utilization of your matching reservations.
Other AWS services with reservation models include RDS, Redshift, ElastiCache and DynamoDB.
Top Tip
Don't forget to monitor your RIs so you can allocate other workloads to them if
they're unused or underutilized. And keep track of their expiry date, as AWS doesn't
renew them automatically.
Purchase
Spot Instances
Spot Instances are a disposable class of instances, which are priced based on
supply of and demand for EC2 spare capacity.
Purchase Spot instances
Using Spot Instances can potentially bring the biggest savings to your AWS monthly bill, with discounts
of up to 90% compared with standard on-demand machines.
When you purchase a Spot Instance, you're simply charged the current market price for the machine type
you need – subject to an optional maximum price you're willing to pay.
However, AWS can interrupt your Spot Instance if it doesn't have enough capacity to meet demand. It
can also do so if the current Spot price is greater than your maximum price or your request includes a
constraint, such as a launch group or an AZ group, it cannot meet.
Although interruptions are relatively infrequent, they can still happen at any time. So Spot Instances
aren't suitable for mission-critical applications that rely on continuous availability.
Interruption Behavior
You can specify how EC2 interrupts your Spot Instances.
The default interruption behavior is to terminate them. But you also have the option to either stop or
hibernate them.
When the Spot service stops or hibernates your instances, it preserves your EBS volumes. However,
hibernation also preserves instance memory on the root volume. This means your workloads can resume
automatically when your Spot capacity becomes available again.
Currently, hibernation is supported by a limited number of instance types. It also requires installation of
the EC2 Hibernation Agent on your guest OS.
Spot Instance prices are less volatile under the new pricing model
Downtime can end up being a far greater cost to your business – through reputational damage, loss of
productivity and loss of revenue. So it's equally essential to have backup and recovery measures in place to
help you maintain business continuity and keep downtime costs down.
Third-party data management tools are designed to simplify the backup and recovery process, so you're
not scrambling for the AWS documentation whenever you need to restore your cloud infrastructure.
But data management tools don't just help in potential disaster situations. They also help you reduce your
AWS bills by providing continuous cloud storage optimization.
instant recovery button to protect your data from any outage any time.
Savvy Data Lifecycle Management to Save Your AWS (and Your Budget)
• Save up to 60% on long-term retention costs by decoupling and transferring your EBS
snapshots to the N2WS S3 repository, with the ability to recover to any region or account
• Start, stop and hibernate groups of Amazon EC2 or Amazon RDS instances and save on
computing costs with N2WS LightSwitch
About N2WS
environment, providing backup and disaster recovery functionality across regions and accounts.