CodeOrigin All articles
Distributed Systems

The Second Request Is the Real Test: Idempotency as an API Contract

CodeOrigin
The Second Request Is the Real Test: Idempotency as an API Contract

Retries are usually framed as a network concern: a timeout occurs, a connection drops, or a load balancer loses a response. The harder problem begins after that moment. The client does not know whether the server performed the operation. It sends the request again. If the endpoint was designed around the assumption that each request arrives once, the second request becomes a second payment, a second order, or a second webhook delivery. The defect is not in the network. It is in the contract.

Idempotency is the property that repeating the same logical request produces the same observable result as the first successful execution. It does not mean that every request is harmless. It means the endpoint defines what “same request” means, records what happened, and enforces that definition under concurrency and failure. From first principles, that is a design problem that starts at the request boundary and ends in durable storage.

Retry Is an Ambiguity Problem

A client retries because it lacks knowledge. It sent a request, waited, and received no definitive answer. The server may have done nothing, may have completed the work, or may still be working. The safe response is not to guess. The API must provide a way for the client to ask again with the same identity and receive the original outcome rather than a duplicate effect.

This is why “exactly once” is often the wrong phrase. Networks cannot promise it. Business operations can approximate it through idempotent handling: at least one delivery, with repeated deliveries collapsing into one recorded outcome. The guarantee belongs to the operation, not to the transport.

Request Identity Must Be Deliberate

The foundation is a stable idempotency key supplied by the caller for unsafe operations such as creating a payment or placing an order. The key should be treated as an opaque value with explicit length and character limits. It should be scoped to a principal and an endpoint, so one customer cannot collide with another and one operation cannot be replayed against a different resource.

A key alone is not enough. The server should bind it to a fingerprint of the request: the method, path, authenticated principal, and a canonical representation of the body. If the same key arrives with a different payload, the correct response is a conflict, not silent reuse of the earlier result. This prevents a client bug from turning into a surprising state transition.

It is useful to separate the idempotency key from a tracing request ID. The request ID identifies a single HTTP exchange for observability. The idempotency key identifies a logical operation across retries. Mixing the two weakens both.

Storage Is the Contract

An idempotent endpoint needs durable memory. The server must record that a key has been seen, what request it was bound to, what state the operation reached, and what response should be replayed. That record must survive process restarts and be visible to every instance that might receive a retry.

The record should have an explicit state machine. A new key moves to in-progress. Successful completion stores the status code and response body, or enough information to reconstruct them. A retryable failure may return to a state that permits another attempt under the same key. A permanent failure should remain recorded so that a retry does not accidentally bypass the decision.

The difficult boundary is the side effect itself. When the operation is a database write, the idempotency record and the business change can often share a transaction. When the side effect is an external payment authorization, the endpoint must use the provider’s idempotency mechanism or an outbox pattern. Marking an operation complete before the downstream effect is acknowledged creates a false record. Performing the effect before recording the key creates a window for duplication. The design must acknowledge that window and close it with reconciliation, not optimism.

Concurrency Demands a Single-Flight Rule

Two retries can arrive at the same time. A check-then-insert sequence is not sufficient; two workers can both observe that the key is absent. A unique constraint on the key is the simplest enforcement point. The first insert wins. The second request must then wait, poll the record, or receive a deliberate in-progress response with guidance to retry later.

The same rule applies downstream. If the endpoint calls a payment processor or inventory service, the retry should carry a stable downstream idempotency key. Otherwise the API may deduplicate at its edge while duplicating the effect one layer deeper.

For long-running operations, a lease or ownership token can prevent two workers from executing the same key. The lease must expire so that a crashed worker does not block progress forever, but expiration must be paired with recovery logic that checks the operation’s actual state before resuming.

Expiration Is a Policy Decision

Idempotency records cannot be kept forever at unbounded cost, but expiration is not merely a cleanup task. If a key is removed while a client or downstream system can still retry it, the endpoint loses its ability to recognize the duplicate. Retention must exceed the maximum retry window the API permits, the client’s own retry horizon, and any reconciliation period required by the business domain.

Different endpoints deserve different policies. A payment authorization may need a longer window than a profile update because chargebacks, settlement, and support investigations can surface late. Webhook receivers need their own duplicate suppression because delivery systems routinely retry after apparent failures. The policy should be documented and monitored, not left to a background job’s default setting.

Failure Modes Worth Designing For

The most important failure is the crash after the side effect but before the idempotency record is finalized. The second is the completed record whose response never reached the client. Others include key reuse with a different body, simultaneous duplicate requests, storage outages during key lookup, non-idempotent downstream calls, and partial failures inside multi-step operations. Each deserves a defined response and an audit trail.

A Practical Checklist

The second request is the real test of an API. A system that handles the first request correctly but duplicates the second is not reliable; it is merely fortunate. Idempotency turns retry behavior from an accidental property of the network into an explicit contract that callers can rely on.

All Articles

Related Articles

Failure as an Interface: A Taxonomy of Errors Callers Can Act On

Cursors That Keep Their Word: Stable Pagination Under Concurrent Writes

Cursors That Keep Their Word: Stable Pagination Under Concurrent Writes

Cron Is Not a Scheduler: Durable Background Jobs From Guarantees Upward

Cron Is Not a Scheduler: Durable Background Jobs From Guarantees Upward