Retry with exponential backoff and jitter

A network resilience strategy that spaces out repeated requests to a failing service to prevent self-inflicted Denial of Service attacks.

What it is

In distributed systems, transient network failures are common. A packet drops, a router reboots, or a load balancer momentarily drops a connection. The simplest way to handle a transient failure is to simply try the request again.

However, if a downstream service is struggling under heavy load, and 10,000 clients all instantly retry their failed requests at the exact same millisecond, the downstream service will be utterly crushed. Exponential backoff and jitter are mathematical algorithms applied to retry logic to smooth out this traffic and give the struggling service time to recover.

Why it matters

Without backoff, automated retries create a “Thundering Herd” problem.

If a central database goes offline for three seconds, every API server trying to write to it receives a connection error. If they all retry immediately in a tight loop, the moment the database comes back online, it is instantly hit with hundreds of thousands of queued connections. This massive spike crashes the database again immediately, turning a three-second outage into a three-hour cascading failure.

How to implement

1. Exponential Backoff

Instead of retrying immediately, the client waits a certain amount of time. Crucially, if the retry fails again, the wait time increases exponentially.

Attempt 1 fails: Wait 1 second.
Attempt 2 fails: Wait 2 seconds.
Attempt 3 fails: Wait 4 seconds.
Attempt 4 fails: Wait 8 seconds.

This exponentially reduces the load on the failing service over time.

2. Adding Jitter

Exponential backoff alone is not enough. If 1,000 clients all fail at the exact same moment (e.g., when the server restarts), they will all wait exactly 1 second, and then all hit the server simultaneously again. Then they will all wait 2 seconds, and hit it simultaneously again. The load graph will look like massive, destructive spikes separated by silence.

Jitter introduces a random element to the wait time. Instead of waiting exactly 4 seconds, a client waits a random amount of time between 0 and 4 seconds. This spreads the 1,000 retries evenly across the 4-second window, smoothing the load curve and allowing the server to process the backlog efficiently.

// Example implementation of Full Jitter
function getWaitTime(attempt, baseDelay) {
  // Calculate exponential backoff
  const maxDelay = baseDelay * Math.pow(2, attempt);
  // Apply random jitter
  return Math.random() * maxDelay;
}

Common mistakes

Infinite Retries

If a client is configured to retry infinitely, a permanently dead service will cause the client to queue up an infinite number of background retry threads. Eventually, the client runs out of memory and crashes. Retries must always have a hard limit (e.g., maximum 3 attempts) before permanently failing and returning an error to the user.

Retrying Non-Idempotent Actions

It is safe to retry an HTTP GET request (reading data). It is incredibly dangerous to blindly retry an HTTP POST request (like charging a credit card). If the initial network call actually reached the server, processed the charge, but the network connection died before the server could send the 200 OK response back, the client assumes it failed. If the client retries, the user’s card is charged twice. Retries must only be used on idempotent operations (actions that can be performed multiple times without changing the result beyond the initial application).

Retrying Client Errors

A client should never retry an HTTP 400 Bad Request or an HTTP 401 Unauthorized. Those are deterministic errors; retrying a malformed payload a second later will not magically fix it. Clients should only retry transient server errors (HTTP 500, 502, 503, 504) or network connection timeouts.

Verification

To verify backoff logic in application code, inspect the telemetry traces of a failed request. If the trace shows a network call failing, you should see subsequent spans representing the retries. Calculate the timestamps between those spans.

Span 1 (Failed): 12:00:00.000
Span 2 (Failed): 12:00:01.234 (Wait ~1s)
Span 3 (Failed): 12:00:03.541 (Wait ~2s)
Span 4 (Failed): 12:00:07.112 (Wait ~4s)

If the gaps are identical (e.g., exactly 1.000s every time), jitter has not been implemented, and the system is vulnerable to thundering herds.

Sources & further reading

Exponential Backoff And Jitter - Amazon Web Services (Marc Brooker)