17th May, 2023
11 Min read
Book a Demo
While AWS services provide a wealth of mission-critical services – storing over 2.2 trillion objects in S3 – many organizations are left floundering in the solution’s complex pricing structures. Spanning transfer types and geographies, data transfer costs can be hugely unpredictable and rapidly get out of hand.
Below, we leverage decades of industry experience to show the major drivers behind uncontrolled AWS spend. From there, we can establish a comprehensive list of best practices for reducing AWS data transfer costs.
Cloud computing enables businesses to transfer vast swathes of data between users and applications. The data lifecycle begins with data ingress – where GBs and TBs worth of information is sent to the cloud’s secure and global platform (aka AWS s3). However, the value of data is only truly realized when used outside of storage. Data egress is the data transfer lifecycle’s final step. Given its importance, data exiting the cloud is one of the biggest influences on your AWS bill.
While the AWS pricing calculator gives businesses a helping hand, many customers find their real-world data transfer charges to be hugely unpredictable. Amazon lists over a dozen reasons why costs can vary from the calculator’s quotes. Due to the sheer quantity of variables, keeping control over data transfer costs requires a thorough understanding of just how your data is being transferred – as well as the charges associated with each type of transfer. Starting with the broader architecture, here are the three primary transfer routes – and their unique pricing quirks.
The public internet provides rapid object delivery between Amazon S3 buckets and the millions of apps and websites available today. Amazon S3 provides 100GB of data transfer for free each month. Exceeding that, additional fees apply: for the first 50TB, AWS charges $0.09 per GB. This means – for an organization transferring 50TB of data between AWS via the public internet every month – their theoretical cost would be $4,491. For the large corporations handling even more data, an extra 100TB is charged at $0.07 per GB.
Data egress for sites and applications is only one small part of the pricing puzzle. While many organizations rely on the free data transfer AWS offers between its services, it’s vital to note that these services are only free of charge when within the same Availability Zone and held by the same account.
This pricing structure directly contradicts architectural best practice. Always-on availability relies on workloads being spread across different regions in case of localized outage. Placing even further weight onto AWS’ pricing complexity is the fact that the same region can encompass multiple availability zones. AWS’ highly geographical pricing is discussed in further detail below. These surprise costs add up rapidly, as data transfer fees between different regions start at $0.02 per GB, and at $0.01 per GB between different accounts within the same region. As every layer adds up, so too does the total bill for each month.
AWS offers a number of different methods for transferring data to on-prem. To make it even more interesting, each has its own unique pricing model. The three most common options are Direct Connect, Storage Gateway, and Snowball.
While the monthly charges for data transfer seem fair in a vacuum, it’s vital that your cost review process takes a macro view of your entire architecture. By understanding the context of each transfer, it becomes possible to answer the following core question.
With data transfer fees ranging from $0.025 to $0.85 per gigabyte – and the average company handling petabytes of data – it’s easy to see how costs can spiral rapidly out of control. One of the biggest factors to AWS’ unpredictability is their concept of Regions. Regions are physical locations around the world where AWS’ data centers are clustered. Drilling down deeper, each Region is composed of at least three data centers, grouped into Availability Zones. These AZs are physically separated to help ensure high availability. Sao Paulo is the most expensive AWS region, with a basic transfer fee that reaches $0.15 per GB.
On AWS’ side, building and maintaining data centers is hugely expensive, with each region fronting its own labor, real estate, and energy costs. The demand placed on each location is unique and fluctuating, leading to prices that quickly become varied. On the other hand, organizations are rapidly developing a global mindset. In the pursuit of rapid global growth, data transfer patterns can rapidly become a tangled and unmanaged mess, leading to uncontrolled AWS pricing. With Regions holding a major sway over the cost of every single AWS service, they also offer a unique opportunity to bring AWS costs back under control.
Reducing data transfer costs is a multi-phase process. From building a foundation of visibility to actioning immediate cost savings, here are the first 10 steps toward AWS cost optimization.
Cost allocation tags in AWS are a way to categorize and track resource usage and costs across your AWS account. They allow you to tag your AWS resources (such as EC2 instances, S3 buckets, and RDS databases) with custom metadata that describes their purpose, owner, or any other relevant attribute.
Taking the extra time to assign tags to instances and load balancers enables a culture of cloud visibility. By tracking the resources being used by each project, it becomes possible to discover over-utilization.
Actively reviewing your AWS cost distribution doesn’t have to be a painstaking process. The most basic tool in your arsenal is AWS’ inbuilt Cost Explorer. By navigating to its usage page, you’re offered a view of how much data each AWS service is transferring. It’s then possible to rejig your architecture to reduce this. For example, many larger organizations make use of NAT gateways, allowing instances in a private subnet to communicate with the internet without being directly exposed. NAT gateways charge based on how much data is being transferred. If this is eating away at cloud spend, however, NAT instances can offer the same functionality at far lower cost.
For a wider view of how much data is flowing throughout an organization, AWS CloudTrail logs all API calls that reach an AWS service. Together, you’re granted insight into which apps and services are most data transfer-hungry. This essentially creates a heatmap of inefficient hotspots throughout your organization’s stack, defining your priority list.
Once you’ve gained a clear understanding of where your current AWS budget is going, it’s time to begin making changes. First and foremost: the quantity of data leaving your cloud. Data compression can offer immediate benefits to suitable use-cases, with reduced latency and bandwidth requirements. If data compression isn’t suitable, then optimizing application code – with a focus on reducing file sizes and minimizing reliance on external libraries – can also help drop transfer costs.
Not all S3 buckets are created equal. Optimizing S3 storage can focus on individual instances – S3 Select, for instance, allows a workload to retrieve only the data you need from an object, rather than retrieving the entire object. At the same time, S3 Intelligent Tiering offers automatic changes to each object’s access tier. This is based on access patterns as they change in real-time. This can help to optimize your storage costs by automatically moving objects to the most cost-effective access tier without any impact on performance. Finally, lifecycle policies can automate the process of transitioning objects between classes or deleting them when no longer needed. Comprehensively addressing unoptimized S3 storage allows you to pay only for the storage you actually need.
CloudFront caches frequently-accessed content at edge locations, reducing the need to retrieve content from the origin server after every request. Caching rules are able to be modified based on specific criteria such as URL patterns, HTTP headers, and cookies, further optimizing cache utilization and cutting the amount of data being transferred over the internet.
CloudFront integrates with Amazon S3 and EC2, allowing users to serve content directly from these services and dodging additional data transfer costs from external services. Its pay-as-you-go pricing means you only pay for the data transfer and requests actually used. It’s worth mentioning that as an Advanced Tier Services Partner of AWS, GlobalDots can provide superior CloudFront pricing.
And if you’re considering alternatives to Amazon CloudFront, GlobalDots also partners with Akamai and Cloudflare, two of the largest CDN providers worldwide. Providing bargain rates for both, customers can make use of efficient architecture and reduced bandwidth demands at an even lower cost than AWS CloudFront.
AWS charges data transfer fees for data transferred between different regions or availability zones. By keeping data transfer within a single region, these fees can be avoided. One way to support this is with region-specific endpoints. For example, using S3, utilize a specific endpoint for the region in which you are storing your data. In the same vein, Virtual Private Cloud (VPC) peering is a networking feature that allows you to connect two VPCs within the same region. This routes traffic directly between the VPCs, without going over the public internet.
Note that high-availability and mission-critical functions may require a higher degree of fault tolerance. This is typically achieved by sharing these workloads across multiple regions. Make sure to identify which multi-region workloads are necessary, and restrict those that aren’t.
When you launch EC2 instances, ensure that they are in the same VPC, and subnet within the same region. To help reinforce this boundary, consider configuring security groups: these act as virtual firewalls and allow you to specify which traffic is allowed to and from each instance.
Furthermore, some AWS services are specific to particular regions, and using them can result in lower costs. For an S3-specific example, issuing a Standard-Infrequent Access (IA) storage class in the same region as the EC2 instance can reduce the data transfer costs associated with infrequently accessed data.
AWS regions can vary dramatically in price. However, a major benefit of cloud computing is its global accessibility: make the most of this by tailoring your cost optimization approach around this. With a cohesive cloud restructure that streamlines all services into this cheaper region, cost savings can be truly transformative.
It’s important to note that choosing the cheapest region isn’t always the best option, as other factors like latency and data transfer costs should also be considered. Other factors, such as proximity to users, compliance requirements, and service availability, may also play a role in choosing a region. Perform a cost-benefit analysis to determine the most cost-effective solution for you.
Too many AWS accounts provision resources on a worst-case-scenario approach. This leads to chronic overprovisioning, and one of the biggest cost realization actions can be achieved by identifying off-peak hours. Take AWS Redshift: its tremendous data warehousing ability offers real computing power. However, when Redshift clusters are run on-demand, idle clusters are still being paid for. Off-hour shutdown is vital for Redshift and EC2. This change can run in parallel with a culture of cloud cost optimization, where DevOps are shown the true cost-saving potential they wield.
Although it’s normal to experience small seasonal fluctuations in cloud usage, it’s important to remain vigilant for any abrupt and unforeseen spikes in usage. To manage cost drivers proactively, set benchmarks and monitor them regularly with AWS Cost Explorer. Make accurate spending forecasts to stay on track with your KPIs and avoid any unpleasant surprises.
While you can independently begin laying a cultural foundation of cost optimization, projects executed by FinOps specialists can ensure deeper cost reduction thanks to accurate planning and implementation that stretch across entire organizations.
For example, one large eCommerce company had been shifting its operations to the cloud, but this transition was done without any clear methodology or proper architectural planning. Consequently, the company was suffering a disproportionate cloud bill and limited visibility into their own resource usage.
Counting a total of 74 AWS accounts across 16 business units, this optimization process represented a complex challenge. Alongside this, there was – understandably – some resistance to external intervention. Working to gain trust with the pre-established teams, GlobalDots was soon able to lead fruitful and open conversations around cloud usage and implementation decisions. From there, action could be taken to address the more egregious inefficiencies. See here for a full run-down of GlobalDots’ approach and techniques.
Cost optimization in the cloud refers to the process of minimizing the amount of money spent on cloud services while ensuring that the organization’s needs are met. It involves identifying opportunities to reduce costs and then taking steps to implement cost-saving measures.
Cloud cost optimization is a crucial component to organizational buoyancy, particularly in today’s wider context of financial instability. By helping organizations forecast and budget for their cloud spending more accurately, the risk of unexpected costs is drastically reduced. The benefits go deeper, too, as organizations are empowered to scale up or down without incurring unnecessary expenses, allowing full architectural agility. Ultimately, cloud cost optimization is crucial for maximizing the value of cloud services and achieving business goals while minimizing expenses.
Cloud Cost Management refers to the set of practices and tools that organizations use to monitor, analyze, and optimize their cloud spending. It involves tracking cloud usage and spending, identifying areas of inefficiency or waste, and implementing strategies to reduce costs – all while maintaining a high level of performance and functionality.
Cutting-edge solutions allow for close tracking and monitoring of all cost drivers, while insight-rich tools can support your team via automated changes in provisioning.
Prioritizing cloud spending requires identifying and categorizing all cloud services used by the organization, evaluating their importance, and allocating resources accordingly.
Schedule a call with our experts. Discover new technology and get recommendations to improve your performance.