Circuit breaker pattern

Reference Card

• Origin/Prior Art: Release It! (Michael Nygard, 2007).
• Related Practices: Retries, Timeouts, Graceful degradation.
• Key Concepts: Closed, Open, Half-Open states, Failure thresholds.
• Primary Failure Modes: Thread pool exhaustion, Cascading failures.

Introduction

In distributed systems, services constantly make synchronous network calls to other services. When a downstream service becomes slow or unresponsive, the calling service often waits for a timeout (e.g., 30 seconds) before failing.

If traffic is high, the calling service will quickly exhaust its own thread pool or connection pool waiting for the broken downstream service. This causes the calling service to crash, which then causes the service calling it to crash, creating a catastrophic cascading failure across the entire architecture.

The Circuit Breaker pattern prevents this by monitoring the error rate of outbound calls. If the error rate exceeds a threshold, the breaker “trips,” immediately rejecting all further outbound calls without even attempting them, protecting the caller’s resources.

Mental Model

Think of the electrical circuit breaker in your house. If you plug too many appliances into a single outlet, the wires will overheat and start a fire (a cascading failure). To prevent this, the electrical circuit breaker monitors the current. If the current spikes too high, the breaker trips (opens the circuit). The power is cut instantly. The appliances stop working, but the house does not burn down. You must manually reset the breaker to restore power.

A software circuit breaker does the exact same thing for network calls, but it resets itself automatically.

Operational Pattern

The software Circuit Breaker operates in three distinct states:

1. Closed: The normal state. Network calls pass through. The breaker counts successes and failures. If the failure rate (e.g., 50% over the last 10 seconds) exceeds the configured threshold, the breaker trips into the Open state.
2. Open: The broken state. All network calls instantly fail and return an error (or a fallback value) to the user. The breaker does not attempt to contact the downstream service, saving time and threads. A timer is started.
3. Half-Open: The testing state. After the timer expires (e.g., 60 seconds), the breaker allows a single test request to pass through to the downstream service.
   • If the request succeeds, the downstream service is healthy again. The breaker transitions back to Closed.
   • If the request fails, the downstream service is still broken. The breaker immediately transitions back to Open and restarts the timer.

Failure Modes

Missing Fallbacks

Tripping a circuit breaker stops the bleeding, but it still means the user’s request fails. The best circuit breakers implement graceful degradation via fallbacks. If the breaker to the Recommendation Engine trips, the fallback logic should return a hardcoded list of “All-Time Best Sellers” instead of throwing a 500 error. The user still gets a page, even if it is slightly degraded.

Thrashing

If the threshold to trip the breaker is too sensitive (e.g., tripping after 2 errors), the breaker will constantly flip between Closed and Open during normal, minor network blips. This causes erratic user experiences. The failure threshold must be calculated as a percentage of traffic over a sliding time window (e.g., 50% failure rate over a minimum of 100 requests).

Timeouts vs Breakers

A circuit breaker is useless if the underlying HTTP client does not have strict, short timeouts. If the HTTP client waits 60 seconds before classifying a request as a failure, the caller’s thread pool will still be exhausted before the circuit breaker has time to register enough failures to trip. Circuit breakers must be paired with aggressive (e.g., 1-second) network timeouts.

Security

Circuit breakers can inadvertently assist Denial of Service (DoS) attacks. If an attacker knows that sending a specific, malformed payload causes a downstream database to timeout, they can intentionally send that payload repeatedly. This will trip the circuit breaker, taking the downstream service offline for all legitimate users as well. Circuit breakers must monitor errors globally but should ideally isolate state per client or tenant (the Bulkhead pattern) to prevent noisy neighbor attacks.

Operational Guidance

Always emit metrics when a circuit breaker changes state. An alert must fire whenever a critical circuit breaker enters the Open state. While the breaker is protecting the caller, the underlying fact remains that a downstream dependency is completely broken and requires human investigation.

Diagnostics

To diagnose a circuit breaker issue, review the application’s metric dashboards. Look for the metric tracking the breaker state (e.g., circuit_breaker_state{name="payment_gateway"}). If it is flapping between Open and Half-Open continuously, it indicates that the downstream service (payment_gateway) is severely degraded but not completely dead, or that it cannot handle the sudden rush of traffic when the breaker attempts to close.

Related topics

• Retry with exponential backoff and jitter
• Bulkhead pattern

Sources & further reading

• Martin Fowler: Circuit Breaker - Martin Fowler