On this page
Idempotency keys in distributed systems
The architectural mechanism for ensuring that retrying a failed network request never results in a duplicated action, such as charging a customer's credit card twice.
Reference Card
- Origin/Prior Art: Mathematical idempotency ($f(f(x)) = f(x)$), Stripe API design.
- Related Practices: Exactly-once delivery, Distributed transactions.
- Key Concepts: Two-Generals Problem, Network timeouts, De-duplication.
- Primary Failure Modes: Double billing due to naive client retries, Storing idempotency state in memory instead of a persistent database.
Introduction
In distributed systems, networks are inherently unreliable. When a client sends a request (e.g., “Charge this credit card $50”) and the network connection drops before the server can reply, the client is blind. It does not know if the server never received the request, or if the server successfully processed the charge but the response was lost in transit.
If the client assumes the request failed and retries, it risks charging the customer twice. If it assumes the request succeeded and does not retry, it risks silently dropping the order. This is a manifestation of the Two-Generals Problem.
Idempotency keys solve this by mathematically guaranteeing that no matter how many times a specific request is transmitted, the underlying action is executed exactly once.
Mental Model
Think of paying a bill at a restaurant. If you hand the waiter your credit card (the request) and they walk away, and 20 minutes pass without them returning (the timeout), you don’t just hand a second credit card to another waiter (a naive retry). You find the first waiter and ask about the specific transaction you already initiated. An idempotency key is a unique serial number attached to that specific payment attempt. If the server sees the serial number for the first time, it processes the payment. If the server sees the exact same serial number again, it knows it is a retry. Instead of charging the card again, it simply hands back the receipt from the original transaction.
Operational Pattern
An idempotency implementation requires cooperation between the client and the server.
- Client Generation: Before making a mutating request (POST, PUT, PATCH, DELETE), the client generates a globally unique identifier (usually a V4 UUID) called the
Idempotency-Key. - Transmission: The client includes this key in the HTTP header (e.g.,
Idempotency-Key: 8b4c-9f...). - Server Verification: The server intercepts the request and checks a fast, highly-available datastore (like Redis or a Postgres table with a unique constraint).
- If the key does not exist: The server marks the key as “in progress,” executes the business logic (charging the card), saves the final HTTP response payload alongside the key in the datastore, and returns the response to the client.
- If the key does exist: The server completely skips the business logic. It retrieves the saved HTTP response payload from the datastore and returns it to the client immediately.
Failure Modes
The Concurrent Race Condition
If a client has a bug and fires two identical requests with the same idempotency key at the exact same millisecond, the server might check the datastore for both requests before either has been saved. Both requests appear new, and the credit card is charged twice. An effective idempotency layer must rely on strict database constraints (like an SQL UNIQUE index) or atomic caching operations (like Redis SETNX) to guarantee that only the first request acquires the lock.
Key Expiration
Idempotency state cannot be stored forever, or the database will grow infinitely. Keys are usually configured with a Time-To-Live (TTL) of 24 to 72 hours. If a client retries a request after the key has expired and been deleted from the database, the server will treat it as a brand new request and execute it again. Clients must be programmed to abandon retries before the server’s idempotency TTL expires.
Security
Idempotency keys prevent accidental duplication, but they also protect against malicious replay attacks. If an attacker intercepts a legitimate request to transfer money and attempts to replay it 1,000 times against the server, the idempotency layer acts as a structural defense, blocking 999 of the malicious replays without ever invoking the core banking application logic.
Operational Guidance
Never use business logic identifiers (like a user_id or order_id) as an idempotency key. If an order fails due to a validation error, the user must be allowed to fix the error and submit the order again. If the order_id is the idempotency key, the server will just return the original validation error forever.
The idempotency key must be unique to the specific attempt, not the business entity. Clients should generate a new UUID every time the user clicks “Submit,” but reuse that exact same UUID for automatic background retries triggered by network timeouts.