Chaos engineering principles

Reference Card

• Origin/Prior Art: Netflix Chaos Monkey (2011).
• Related Practices: Resilience engineering, Game days, Disaster recovery testing.
• Key Concepts: Steady state, Hypothesis, Blast radius, Abort switch.
• Primary Failure Modes: Testing in production without safety nets, Assuming the system is resilient without proof.

Introduction

In complex distributed systems, failure is a statistical guarantee. Hard drives fail, network cables are severed, and memory limits are breached. The traditional approach was to attempt to architect a system that never failed. Chaos engineering asserts that because failure is inevitable, the system must be designed to survive it gracefully, and that resilience must be constantly, actively tested.

Chaos engineering is the practice of intentionally introducing controlled failures into a system to identify weaknesses before they manifest as uncontrollable, user-facing outages at 3:00 AM.

Mental Model

Think of a vaccine. A vaccine introduces a small, controlled amount of a virus into the human body. This intentional “attack” trains the immune system to recognize and defeat the virus. When the real, uncontrolled virus eventually arrives, the body is already prepared to handle it, preventing a catastrophic illness.

Chaos engineering is a vaccine for distributed systems. You intentionally terminate a database node during business hours (the vaccine) to ensure your automated failover scripts (the immune system) actually work.

Operational Pattern

A chaos experiment follows a strict scientific method:

1. Define the Steady State: What does “normal” look like? (e.g., “The API returns 99% of requests in under 200ms”).
2. Form a Hypothesis: “If we terminate one of the three Redis cache nodes, the load balancer will route around it, and the steady state will remain unchanged.”
3. Run the Experiment: Introduce the chaos (terminate the node).
4. Verify the Results: Did the steady state remain unchanged?
   • If yes, the hypothesis is confirmed; the system is resilient.
   • If no (e.g., latency spiked to 5000ms), you have discovered a vulnerability. Stop the experiment immediately, write a postmortem, and fix the system.

Failure Modes

Uncontrolled Blast Radius

The most dangerous mistake is running a chaos experiment that accidentally takes down the entire production environment. Experiments must always start with the smallest possible blast radius. Do not unplug an entire data center on your first try. Terminate a single container. If the system survives, terminate a whole virtual machine. Only scale the experiment up as confidence grows.

Missing the Abort Switch

If you inject latency into a network route to see how the system handles a slow dependency, and the entire platform instantly crashes, you must have an immediate, automated “abort switch” to halt the experiment. If you have to manually SSH into a router to remove the latency injection while the site is down, the chaos experiment has caused a genuine incident.

Chaos Without Observability

If you run a chaos experiment but do not have the metrics, logs, and traces to observe exactly how the system reacts, the experiment is useless. You are just breaking things for fun. Chaos engineering is only valuable if you can measure the precise impact of the failure.

Security

Chaos engineering can be applied to security (Security Chaos Engineering). Instead of testing reliability, you test security controls. Hypothesis: “If a container starts scanning the internal network on port 22, the IDS will detect it and the container will be isolated within 60 seconds.” By intentionally running a safe network scan from a test container, you mathematically prove your security tooling works, rather than just trusting the vendor’s brochure.

Operational Guidance

Start with “Game Days” in staging. Gather the engineering team in a room (or a Zoom call), announce the failure you are about to inject into the staging environment, inject it, and watch the dashboards together. This builds team confidence and uncovers obvious flaws before you ever attempt chaos engineering in production.

Diagnostics

When an actual outage occurs, cross-reference the failure with your chaos engineering backlog. If the outage was caused by a database node dying, and you had previously run a chaos experiment that proved the database could survive a node dying, your experiment’s hypothesis was flawed, or the system drifted since the experiment. Re-evaluate your testing methodology.

Related topics

• Incident command system (ICS)
• Circuit breaker pattern

Sources & further reading

• Principles of Chaos Engineering - Chaos Community