Chaos Monkey Resilience Testing
Create controlled failures to build systems that degrade gracefully
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 98%
Chaos Monkey resilience testing deliberately introduces controlled component failures into a live or realistic system. The team first defines the indispensable customer outcome, such as finding and playing a video, then randomly terminates processes or disables dependencies while monitoring behavior. Secondary capabilities may temporarily disappear or lose quality, but the core service should remain usable. Each experiment exposes assumptions, tight coupling, and missing fallbacks that conventional happy-path testing overlooks. Engineers repair those weaknesses and repeat the process with broader failure scenarios. By making small failures routine and observable, the organization develops systems that degrade gracefully rather than collapsing catastrophically when an unexpected production fault occurs.
Origin
Netflix introduced Chaos Monkey in 2011 as part of its effort to become a world-class internet company. The software killed random infrastructure processes so engineers would continually encounter and correct failure modes.
Core principles
- 01Continuously test assumptions about system reliability
- 02Treat component failure as normal rather than exceptional
- 03Preserve the user's primary outcome during degradation
- 04Expose weaknesses while engineers are available to respond
- 05Design graceful degradation before catastrophic failure occurs
How to run it
- 1
Define the Critical User Outcome
Choose the single customer action the system must preserve during partial failure. Use that outcome to distinguish essential dependencies from optional features.
Pro tip Frame the outcome in user language, such as “search for a title and watch it.”
Watch out Trying to preserve every feature equally can obscure the system's true priorities.
- 2
Establish Observability and Guardrails
Instrument the relevant services, define stop conditions, and ensure responders can reverse an experiment. Begin during hours when the responsible engineers are available.
Pro tip Track both technical health and whether users can complete the critical action.
Watch out Never inject failures without reliable monitoring and a tested recovery path.
- 3
Inject a Controlled Failure
Terminate a process, remove an instance, or disable a noncritical dependency within a bounded scope. Allow the system's redundancy and fallback behavior to respond naturally.
Pro tip Start with one low-blast-radius dependency before adding randomness or concurrency.
Watch out An experiment that can exceed its declared scope is an uncontrolled outage, not chaos testing.
- 4
Evaluate Graceful Degradation
Confirm that the primary outcome remains available even if recommendations, profiles, or resolution degrade. Record hidden dependencies and unexpected propagation paths.
Pro tip Evaluate the customer experience directly instead of relying only on server metrics.
Watch out A technically healthy dashboard can conceal a broken user journey.
- 5
Remove the Weakness and Repeat
Fix brittle coupling, add fallback behavior, and rerun the scenario. Gradually broaden the range and frequency of failures as the system becomes more resilient.
Pro tip Convert every discovered weakness into a regression test or recurring experiment.
Watch out Running experiments without closing findings creates risk without improving reliability.
In the wild
Netflix designed its experience so that a failed recommendation or profile service could reduce personalization while preserving the essential ability to search for and play a title. Chaos Monkey repeatedly tested whether the infrastructure honored that priority.
→ Customers could continue watching even while secondary capabilities degraded.
An ecommerce team disables its recommendation service during staffed hours. Product suggestions disappear, but search, carts, payments, and order confirmation continue operating. Engineers discover and remove an unnecessary checkout dependency on the recommendation database.
→ Checkout becomes resilient to a previously dangerous secondary-service outage.
Common mistakes
Injecting Chaos Without Guardrails
Unbounded experiments can create genuine outages. Define scope, stop conditions, monitoring, and rollback procedures before introducing failure.
Protecting Metrics Instead of User Outcomes
Healthy infrastructure metrics do not prove that customers can complete the essential task. Test the critical journey directly.
Ignoring Discovered Weaknesses
Chaos testing only creates value when findings produce fallbacks, decoupling, or regression coverage. Repeatedly exposing the same weakness is not progress.
Is it for you?
Best for
It is best for mature distributed systems whose teams can monitor experiments and rapidly address exposed weaknesses.
Not ideal for
It is not ideal for unstable early prototypes lacking basic observability, rollback controls, or incident response capacity.
From the transcript
“what better way to uh prevent failure than to always be failing and be able to construct systems that are extremely resilient and sort of…”
“it would just start pinging around all the different, you know, internals of your system and just kill random processes at will.”
“you could always do the number one thing that people want to watch on Netflix, which is search for a thing and then watch it.”
From the episode
Netflix (Part 2)