Stop Wasting Money, Start Cost Optimization for AWS!

Generally, organizations move towards AWS to furnish their foundation with the capacity to develop and extend their abilities and because they can only pay for the resources they use. An unfortunate side effect of this methodology is that little costs regularly go unnoticed and can add up over time, prompting high monthly bills. The monetary effect of the current pandemic is forcing the world to adjust spending within their organizations. Everybody is turning over every rock to discover approaches to cut waste without impacting business. One way to get fast cost savings is to eliminate wasted spend on cloud services.

Organizations waste an average of about 35% of their cloud spend. With the implementation of a focused effort can save an organization as much as 30% to 35% on their cloud spend within a few months. Unlike numerous other IT costs tied to long-term contracts, cloud spend can provide quick wins in any cost-savings initiative. Getting solid cloud cost-saving practices in place will also help ensure ongoing efficiencies as cloud use accelerates even faster in a post-pandemic world. If you’re trying to reduce expenses the following practices help limit your AWS expenditures and protect your bottom line.

Recently I was working on a project where the previous application team was handling the infrastructure and maintenance activities without any DevOps perspective. Therefore, the client wanted to maximize automation and reduce operational and maintenance costs wherever possible. The initial step to conquer any issue is to comprehend our concern and afterward investigating our approaches to determine it. So in the very first step, we started analyzing our expenditure by bills. For analyzing our spending on resources we used AWS cost explorer.

After analyzing, we were able to create a list of our expenses for AWS services that were impacting our bill.

1. EC2 = 30%
2. RDS = 20%
3. CloudWatch = 9%
4. Cloudfront = 6%
5. Other = 35%

After shortlisting our pillars to be worked on we started analyzing where and how we can save our costs. As per shortlisting, we found out that 50% of our cost impact is from EC2 and RDS alone. So the first step which we have taken to reduce our expenses is to minimize the cost of EC2, RDS, and then others. The following are a few practices that helped me while reducing our cloud expenses.

EC2 Savings

So we started analyzing the utilization of our resources on EC2. For identifying the instances that were not optimized fully, we have used the AWS compute optimizer tool.

Further digging in Over-provisioned instances we can get the recommended instance type which we can use in place of over-provisioned EC2s.

After analyzing reports for utilization we came to know that 60% of our resources were underutilized and using only 30-40% of CPU and memory. After optimizing the EC2 resources, we have acquired a few things which can be done to save almost 40-50% on the cost of our EC2 service.

- - Savings Plan: It’s a discount plan, where purchasing in bulk leads to cost savings. To participate, the program requires customers to commit to spending a specific dollar amount per hour over a one- or three-year term. AWS will offer discounts that can be as much as 72% off of the original price. The longer the commitment, the higher the discount. And for customers that choose to pay all upfront will have the benefit of further discounts. It also has a drawback that you can’t purchase any savings plan for ECS, RDS, Redshift, and other services. Once you are past your Saving Plan limits, you will be charged On-Demand prices. We can also visualize our coverage by savings plan using the AWS utilization report and coverage report in our billing section which will help us better utilize our savings plan.

- - Reserved Instance: Reserved instances (RI) is also a good way of saving our expenditure on EC2. If we have predictable workloads it makes sense to opt RI as an option to choose.RI has the same fundamentals as a savings plan to commit over a one or three year term along with utilization rather than hourly commits. It is a little less flexible than Savings Plan but offers higher discounts. We can choose between standard or convertible RIs or a mix of both according to our needs and workload predictions. We can track our coverage of RI under RI Coverage reports provided by AWS.

- - Spot Instance: Spot instances are a type of AWS instances that allows us to take advantage of unused EC2 capacity and reduce the expense. These instances can save up to 70%-80% of the cost as compared to On-Demand instances which fluctuate depending upon the demand and supply of spot instances. But we have to keep in mind that spot instances can be lost within 2 minutes of notice. This is an ideal use case if your application is fault-tolerant and can handle the interruption. Spot instance can also be launched for 1-6 hours of duration which is also known as spot blocks. Spot fleet can also be used as a combination of On-Demand and Spot instances for specific use cases.

RDS Savings
For cost optimization on RDS, the key factor is to choose the correct instance type. We can analyze the usage of our RDS by using CloudWatch metrics for the past few months or weeks. After this, we can opt for the right instance size for our workload. One more factor affecting our bill is RDS snapshots which act as a backup of our databases. We can remove old snapshots that were not required to cut a few more slices from the RDS cost. Another thing to look in RDS is the I/O cost which is also a part of RDS billing. If we do not require Multi-AZ deployment then we should restrict it to a single AZ as I/O will be doubled if we are using Multi-AZ because Amazon RDS will synchronously replicate our data to the standby database instance. The biggest rock to turn in RDS cost optimization after this is to use Reserved instances which will reduce our RDS cost by almost 50%-60%. For using Reserved instance we need to commit over a one or three year term along with the utilization.

We Can also track the coverage of our RDS Reserved Instances under

Cost Management -> Reservation -> Coverage Report

CloudWatch Savings
AWS CloudWatch charges consist of a lot of components such as custom metrics, dashboards, alarms, events, etc. We were able to operate many services for free within its free tier limit

Generally, our CloudWatch bill goes up unexpectedly over time. To manage our CloudWatch spends we can keep track of a few things which can help us reduce our bill.

- - Delete unused alarms and keep the threshold at a justified level so that it shouldn’t invoke notifications unnecessarily.

- - We must always configure the log retention period rather than never expire so that they don’t pile up over time.

- - Make sure you are not using duplicate metrics in different dashboards and widgets.

- - Don’t unnecessarily streamline your logs. We might need to keep our production logs but might our non-production environments logs are not worth keeping.

Toppings
Apart from the above major impacting components, we can do some cherry toppings to remove a few more cuts from our bill by using tad keenness.

- - S3: AWS Simple Storage Service is object-based storage that offers a wide range of storage classes such as S3 Glacier, S3 Standard-IA, S3 Glacier Deep Archive, S3 Intelligent-Tiering, etc. Every storage class has different pricing, we can choose our storage class based on our use case and needs. By utilizing S3 storage classes intelligently with lifecycle and versioning policy and we can set aside to 40%-half of our S3 cost.

- - Cloudfront: The price that was charged for CloudFront fluctuates relying upon the edge location from which CloudFront serves your requests and the amount of data transferred. First of all, we should wisely choose our edge location and also we can use whitelisting or blacklisting for countries using Geo-Restriction settings to save some money. Sometimes files at your origins were rarely updated for such scenarios we should increase the cache time at our edge location and should invalidate our CloudFront cache after modifying our data origins.
  - Terminate idle resources: As infrastructure grows we might have missed some resources which were not in use but occupying a little part of our infrastructure along with our bill. We should always try to keep an eye on them and terminate such resources which might be some EC2, RDS instances, or some old log groups in CloudWatch could be in another region created for some testing at initial times. We can use CloudWatch for finding such resources, such as we can look for RDS instances with 0 connections or EC2 with CPU utilization < 1% for a certain period.
  - Infrastructure on-demand: We are paying for any resources on AWS that are up and running whether we are actually utilizing them or not. This option helps to pay for only what we need and consume. Our non-production environments are not required all the time, this strategy will help us reduce our bill by 60-70%. There is a huge number of options that can help us achieve this such as a combination of Jenkins and terraform code for scheduling our non-production environments to be stopped at night and make available in the morning or we can use AWS Instance Scheduler or any other such tools.
  - Billing Alarms: Billing Alarms assist us with following along and cutoff points of our consumption on resources. It’s a smart thing to do if need to know whether we are crossing the lines. For billing alarms first, we have to enable billing alerts in our billing console then we can use the SNS notification list to receive alerts on mail.