Recent AWS Outages: Should You Panic?

Shalom Carmel Chief Information Officer at GlobalDots
6 Min read

Amazon cloud services ran into significant difficulties in December 2021. Three major AWS outages affected financial institutions, airline reservation systems, online retailers, and dating apps. The incidents were global phenomenons. Companies like Spotify, Venmo, Netflix, Tinder, and Roku experienced service disruptions. One of the outages took down Amazon’s own delivery app, which led to packages piling up in Amazon warehouses. Small businesses came to a standstill.

With every outage in December, businesses worldwide waited for AWS to find the underlying root cause and resolve the issues. Organizations realized they were totally dependent on Amazon. It brought to light the vulnerability of relying on a single cloud provider. In the aftermath of the havoc, businesses want to figure out how to safeguard themselves from similar situations in the future.

How One AI-Driven Media Platform Cut EBS Costs for AWS ASGs by 48%

How One AI-Driven Media Platform Cut EBS Costs for AWS ASGs by 48%

Cost of Cloud Outages

The adoption of public cloud services has seen an upward trajectory since the emergence of the big three providers, Amazon Web Services (AWS), Microsoft Azure, and Google Cloud. Businesses have good reasons to embrace cloud computing. Public clouds are high speed, low maintenance, cost-effective, scalable, and secure. To learn more about public cloud advantages, read our article 7 Key benefits of cloud computing for Your Business.

However, the dark side of the public cloud is outages. When an organization relies on a sole cloud provider, a cloud outage can threaten the whole business. Cloud outages expose enterprises to significant financial risks. A 2020 Global Survey of Data Center and IT Managers conducted by Uptime Institute found that two-thirds of outage incidents were more than $100,000 while the rest of the reported outages cost $1 million or more. So, it makes sense for businesses to worry about cloud outages.

The Recent Major AWS Outages

The end of 2021 was not a good time for AWS. The major outages were as follows:

Impairment of Network Devices (December 7, 2021) – The AWS outage began around 12:30 pm Eastern time in US-EAST-1 region. It took AWS about 5 hours to resolve the issue. Several impaired network devices caused the outage. The network devices that handle traffic forwarding and network address translation (NAT) were overwhelmed. It caused disruptions for Netflix, Disney+, Fidelity Investments, Roku, and Ring. The service interruption halted Amazon package deliveries because the scanning and routing apps were unavailable.

Network Congestion and Packet Loss (December 15, 2021) – The AWS incident began around 7:15 am Pacific time in the US-WEST-2 region. The root cause was defined as network congestion due to internal engineers unexpectedly moving traffic to the AWS backbone. Due to packet loss, applications like Okta, Workday, and Slack were unreachable. By 12:00 pm Pacific time, the issue was resolved.

Power Outage in Data Center (December 22, 2021) – The outage started around 4:11 am PST. The reason was determined as a power outage in a single data center in the USE1-AZ4 availability zone of the US-EAST-1 region. The power loss shutdown EC2 and EBS instances. It affected services like Slack, Imgur, EPIC games, and Asana. It took 12 hours to resolve the issue.

Cloud Outages: A Part of DevOps Life

Businesses want reliable infrastructure and 100% outage-free environments. Unfortunately, there is no way to design a system or architecture with 100% uptime. So, cloud outages will always be a part of the IT and DevOps life.

Even though the December outages created an uproar, AWS has gone through many incidents over the years. Here are a few highlights:

Notably, cloud outages can be caused by natural disasters, human errors, and unpredictable network conditions. It’s impossible to design a cloud service that can avoid all calamities.

Cloud Outage Risk Mitigation Strategies

The risk of cloud outages is unavoidable. However, businesses can take steps to mitigate the risks. Here are a few steps you can consider:

Use Multi-Cloud and Hybrid Cloud Environments

Replicating infrastructure across multiple clouds, also known as a multi-cloud strategy, can help you avoid falling into the single cloud dependency trap. During an outage of one cloud provider, you can use your backup provider to keep services running. Another strategy is to split your resources between on-premises and public cloud to create a hybrid cloud approach. You can use your on-premises resources during public cloud outages. To learn more about hybrid cloud, download Survey Report: Hybrid Cloud Management 2020.

Do Not Skimp on Multi-Region Redundancy

If multi-cloud and hybrid cloud strategies are out of your price range, you can still benefit from multi-region redundancy. Even if a region goes down, you will have access to other regions to keep your applications running. For example, if you are running your services from the US East (Northern Virginia) Region and your backup servers are running in Europe (Ireland) Region, you have better protection against a catastrophic event in Virginia. Even if a storm takes out the data centers in Virginia, you can still run your operations from Ireland, even if the services are slower due to physical distance. Businesses with mission-critical services can even distribute their services in more than two regions to increase redundancy. Of course, the cost can be a prohibitive factor. Also, if your business has restrictions about where data can reside, you might be unable to use multi-region as a risk mitigation strategy.

Run Test Cases and Monitor Applications

Being prepared for disaster recovery will help you during outages. You should run test cases and check the efficiency of your disaster recovery (DR) and business continuity plans (BCP). A great way to improve your cloud resiliency is to look into chaos engineering. Netflix pioneered the concept of chaos engineering by developing Chaos Monkey. The tool randomly disables cloud production instances to ensure the services can withstand the disaster. Chaos Monkey is available as an open-source tool. But there are other tools like Chaos Mesh, Gremlin, and Litmus. Also, monitor your applications to alert you at the first sign of any trouble. Automated monitoring can help you stay ahead of any disaster scenarios. To learn about automating your monitoring tasks, check out Automated Application Monitoring – Solution Brief.

Set Up Communication Channels with Customers

Even with mitigation strategies, you might be unable to keep your services going during an outage. If your services go down, you need to inform your users. So, set up channels to give status updates during an outage situation. Honest communication will help you earn your customers’ trust.

Use Vendor-Agnostic Tech Partners

Vendor-agnostic tech partners can decrease risk exposures. They can help you implement multi-vendor cloud solutions. A vendor-agnostic consultant will not try to sell you a particular cloud product or service. So, you have a better chance of getting robust solutions that can handle cloud outages. Vendor-agnostic solutions will ensure the tools you use are aligned with your strategic requirements and risk tolerance.

In a recent webinar Stay Out of Outages – The BCP Element Now One Talks About we discussed multi-vendor strategies to combat cloud outages.

Downtime Insurance

Cloud outages have real financial consequences. If your applications and services are offline, it can lead to missed revenues, lost clients, and reputation damage. As an industry-standard, cloud service-level agreements (SLAs) provide remedy through service credits. But the service credits are not enough to cover the losses you incur. Moreover, cloud providers can define downtimes as degradation if some connectivity is available during an incident. It further restricts your ability to recuperate any loss. Considering the situation, some insurance companies have started providing downtime insurance for cloud users. Downtime insurance can help you mitigate outage risks. Before buying downtime insurance, you should run a cost-benefit analysis to determine if the insurance cost is worth it for your business.

In Conclusion: Be Prepared for Cloud Outages

In today’s business environment, it is impractical to avoid the cloud. But cloud outages are inevitable. Your best bet is to mitigate your risks through multi-vendor cloud strategies and vendor-agnostic tech partners.

At GlobalDots, our senior-level DevOps engineers and experts have experience working with multiple vendors. We take a holistic approach to evaluating and mitigating risks for multi-cloud environments. To learn more about our risk mitigation services, contact GlobalDots today.

Latest Articles

Mobile Cloud Computing Explained

Introduction Mobile Cloud Computing (MCC) is an architectural approach that combines the processing power of mobile devices like smartphones or tablets with cloud-based resources. Remotely rather than locally, as a result of computational augmentations, MCC’s mobile devices may augment resources from various cloud-based accounts. The combination creates a new kind of mobile computing that provides […]

Shalom Carmel Chief Information Officer at GlobalDots
29th December, 2021
Case Study: Creating a Scalable Security & Compliance Infrastructure

An innovator in its field, Euclidea democratizes wealth management with its user-friendly platform and modular service. To support its expansion from Italy to the rest of Europe while preserving a lean workforce, Euclidea approached GlobalDots to implement time-and-work saving innovation, making the company scalable, secure, and compliant at ease. In this case study, you will […]

Francesco Altomare Technical Sales Lead for Southern Europe, GlobalDots
22nd September, 2021
Case Study: GlobalDots cuts 60% off client’s IT expense and revamps undocumented infra.

You would think that an IT consultancy firm will boast the finest IT infrastructure. So did we, until Italian IT consultant and service provider Thinkopen S.p.a contacted us. The complexity, waste, and lack of visibility or documentation to Thinkopen’s infrastructure reached a point of undermining end-user service.  Read how our engineering team mapped this black […]

Francesco Altomare Technical Sales Lead for Southern Europe, GlobalDots
9th August, 2021

Unlock Your Cloud Potential

Schedule a call with our experts. Discover new technology and get recommendations to improve your performance.

    GlobalDots' industry expertise proactively addressed structural inefficiencies that would have otherwise hindered our success. Their laser focus is why I would recommend them as a partner to other companies

    Marco Kaiser
    Marco Kaiser

    CTO

    Legal Services

    GlobalDots has helped us to scale up our innovative capabilities, and in significantly improving our service provided to our clients

    Antonio Ostuni
    Antonio Ostuni

    CIO

    IT Services

    It's common for 3rd parties to work with a limited number of vendors - GlobalDots and its multi-vendor approach is different. Thanks to GlobalDots vendors umbrella, the hybrid-cloud migration was exceedingly smooth

    Motti Shpirer
    Motti Shpirer

    VP of Infrastructure & Technology

    Advertising Services