As engineers and architects, we automatically build resilience into platforms as far as possible. But what about the unknown failures? What about the unknown behavior of your platform? The philosopher, Socrates, once said "You don’t know what you don’t know". What if I could tell you there is a way to turn these unknowns into knowns – a way to understand how your platform will behave to specific failure events…
Gone are the days of having your manager/CTO walk into the server room and pull out a power plug of one of the servers to simulate "chaos" (true story – I worked with a colleague who did this to his IT team). No support engineer wants to be woken up at 2 am in the morning due to some unknown behavior of the platform they support, and have to determine that it was not designed to withstand these events.
In this two-part blog series, we will first look at Chaos Engineering and then explore how it is useful for testing the resilience of distributed systems. Following this, we will walk through some scenarios that should be tested with Kong Gateway to ensure that your platform is correctly configured to deal with failures. The follow-up blog post will build on this, with a tutorial implementing the above-mentioned scenarios as experiments. We will get hands-on using a Chaos Engineering tool and put your [Kong Gateway](https://konghq.com/products/api-gateway-platform)Kong Gateway deployment to the test.
With the move to distributed systems, the confidence around high availability and robustness becomes a complicated beast to tame. Yet another layer of complexity is added with the move from bare metal infrastructure to cloud services. Furthermore, with the added dimension of [microservices](https://konghq.com/blog/learning-center/what-are-microservices)microservices, you may get a massively complex ecosystem (with exponential possibilities for failure points) depending on the number of services in your system.
Enter Chaos Engineering: the practice of injecting controlled failures into your system to improve its overall response. The way the system behaves in response to these failures provides us with the favorable opportunity of monitoring and improving them.
Like most things in life, practicing for events that are out of our control, is a great way to build muscle memory for when it actually happens. A great example of this is a fire drill. If you have ever worked in an office building you will have experienced a fire drill where the entire building vacates onto the street via the emergency exit stairwell. By practicing this time and time again, in the unlikely event of a fire in the building, everyone should know what to do, where to meet and how to ensure everyone gets out of the building safely. Any failures in the process should have been identified in the drills that were performed on a weekly basis.
Similarly, injecting failures into your platform in a controlled manner makes total sense, as it allows for the resolution of any unknown behavior due to failure. It also lets your teams ensure their documentation, system access, playbooks or scripts are correct and builds a reflexive response to an actual critical (Priority 1) event.
According to a [State of Chaos Engineering Report](https://www.gremlin.com/state-of-chaos-engineering/2021/)State of Chaos Engineering Report, teams who consistently run Chaos Engineering experiments have higher levels of availability than those who have never performed an experiment, or those who only run ad-hoc experiments. [1] Whilst this doesn't necessarily guarantee high levels of availability, it clearly is best practice.
Important key findings expressed in the report are that:
[1]
**2008**
Netflix suffered a massive three-day outage that impacted its DVD shipping business at the time. It took them three days to resolve the problem which, after a lot of forensic analysis, concluded that it was a hardware failure. This pushed them to rethink their architecture and they migrated to a distributed cloud architecture.
**2010**
When Netflix moved to the cloud it prompted them to design for failure, as hosts can be terminated or fail at any time. With this in mind, Netflix needed a tool to help them test these failures in a controlled way and allow them to identify weaknesses in their architecture. So, they built a tool called [*Chaos Monkey*](https://netflix.github.io/chaosmonkey/)*Chaos Monkey*. This tool was used to pull down instances and services intentionally. It gave Netflix the opportunity to see how their system would respond to this failure and enabled them to design levels of automation that would resolve these failures.
Due to the success of this, Netflix open-sourced [*Chaos Monkey*](https://netflix.github.io/chaosmonkey/)*Chaos Monkey* and created a new job role specifically for Chaos Engineering.
**2020**
Chaos Engineering went mainstream, with it making headlines on [Bloomberg](https://www.bloomberg.com/news/articles/2020-12-16/the-pentagon-should-learn-from-netflix-and-amazon)Bloomberg. AWS also released their own Chaos Engineering tool, [AWS Fault Injection Simulator](https://aws.amazon.com/fis/)AWS Fault Injection Simulator [2]
**2022**
In the last 5 years, Google searches on the topic of Chaos Engineering have exploded, increasing 24 fold. [1]
As mentioned before, Chaos Engineering is the practice of injecting failures into a system and observing how the system responds.
The principles of this practice are simple:
[2]
How does this apply to Kong I hear you say? There are a number of scenarios that can be tested with your Kong Gateway deployment. Here are two examples:
In a hybrid deployment, the [control plane and data planes](https://konghq.com/blog/learning-center/control-plane-vs-data-plane)control plane and data planes are separated, and the configuration is transferred via a WebSocket. The separation of these components improves the resilience of the platform. This deployment mode is designed to ensure that your data plane is isolated from the control plane and more resilient to failure. If the connection between the control plane and data plane goes down, it can be for a number of reasons:
This experiment will test the failure of the database connected to the control plane.
Bring down the control plane's database which will cause the control plane to fail. This will break the connection between the data plane and the control plane.
With the database down, the control plane will fail as it needs a database connection to function, but the data plane will be unaffected. To test this the following must be true:
When designing your cloud architecture it's always important to ensure that you are designing for failure; not only failure of your application or platform, but also for outages of the data center your platform is running on. Cloud providers give you the ability to handle this by offering different availability zones (AZs) in your regions. Each AZ is a different physical data center. This means you should have the ability to design your system for the unlikely event of an entire AZ going down.
The following experiment will test for exactly that failure, which has happened in the past on several occasions:
To test out this hypothesis, a worker node in a random AZ will be pulled down, which could simulate the data center losing power, or even the physical server having some kind of hardware failure.
Cluster autoscaling will be disabled. This will ensure that a new worker is not created in the AZ, thus simulating an AZ outage.
This test will ensure the following is correctly configured:
As long as Kong Gateway and your cluster have been configured correctly, this outage should have no effect on Kong deployment. There will be a data plane and control plane in another AZ which will be unaffected by this failure. To test this the following must be true:
Hopefully, this paints a clear picture of the importance of testing your distributed ecosystem in a controlled manner. It gives you the ability to understand how your platform will respond to failures that you are controlling, rather than having to try to dissect an issue after an outage has lost your organization’s revenue.
You saw two test scenarios in a Kong Gateway hybrid deployment. However, this approach can and should be applied to any platform.
In part two of this series, we’ll go one step further to:
Planning for unknown failures and how your platform behaves in these situations is the best way to improve the resilience of your API platform. This will reduce the number of critical events and improve confidence in the platform’s usage.
**For more content from this author, check out **[**this interview**](https://youtu.be/r8e9ib0powM)**this interview**** on Kongcast.**
Managed Redis cache is a turnkey "Shared State" add-on for Kong Dedicated Cloud Gateways. It is designed to combine the performance of an in-memory data store with the simplicity of a SaaS product. When you spin up a Dedicated Cloud Gateway in Kong
[](https://konghq.com/blog/product-releases/multicloud-cloud-gateways-managed-redis-cache)
Imagine 47 million requests hitting your platform last month. Can you prove who made each one—and invoice with confidence? If that question tightens your stomach, you're not alone. Metered billing for APIs promises fair, transparent pricing that s
[](https://konghq.com/blog/enterprise/guide-to-metered-billing-for-apis)
The pace of the industry today is pressuring software developers to build, test, and release software more frequently than ever. To achieve this pace, teams have built two core processes into their workflow: Continuous Integration and Continuous Dep
[](https://konghq.com/blog/engineering/what-is-ci-cd)
Stop me if you’ve heard this one before, but there’s a lot of data out there — and the amount is only growing. Estimates typically show persistent data growth roughly at a 20% annual compounded rate. Capturing, storing, analyzing, and actioning data
[](https://konghq.com/blog/engineering/decentralized-apis)
In this blog post, we’ll demonstrate how easy it is to use Gateway API HTTPRoutes to route traffic to workloads deployed in different namespaces in a single Kubernetes cluster — a process that’s easier than ever. Previously, we only had Ingress API
[](https://konghq.com/blog/engineering/sending-traffic-across-namespaces-with-gateway-api)
We're pleased to announce the launch of Standard Webhooks! Kong has been part of the Technical Committee of this standard with other great companies like Svix (the initiator of the project), Ngrok, Zapier, Twillio, Lob, Mux, and Supabase. This was
[](https://konghq.com/blog/engineering/announcing-standard-webhooks)
The Kong Gateway CI was failing intermittently (about once every 100 runs) on the ARM64 platform with a strange error: “attempt to perform arithmetic on local 'i' (a function value)”. The variable i in the context is an integer but at runtime, it w
[](https://konghq.com/blog/engineering/troubleshooting-an-intermittent-failure-on-arm64)