CodeOrigin All articles
Distributed Systems

The Stampede You Wrote: Containing Retry Amplification with a Shared Budget

CodeOrigin
The Stampede You Wrote: Containing Retry Amplification with a Shared Budget

Photo: A.Savin, FAL, via Wikimedia Commons

Every retry is a small wager. The client observes a failure and decides, implicitly, that the failure was incidental: a dropped packet, a brief queue, a momentary timeout. That decision is a causal claim about a system the client cannot see. When the claim is true, retries repair transient faults cheaply. When it is false, retries are additional load aimed at a service already struggling to answer the first request. The difference between those two outcomes is not a matter of good intentions. It is a matter of accounting.

What a Retry Actually Promises

A retry promises that the failure is not evidence of a persistent condition. It is conditional on three assumptions: that the operation is safe to repeat, that the failure is independent of the attempt, and that the client has time to try again. Each assumption can fail. Non-idempotent writes can duplicate side effects. Overload failures are correlated with the retry itself. And a retry issued without a deadline is not a retry at all; it is an unbounded commitment.

The first-principles move is to treat retries as a load-management decision rather than an error-handling convenience. Error handling asks, "Did this attempt fail?" Load management asks, "Should this caller be permitted to add another request to a dependency that is already signaling distress?" Those are different questions, and the second one requires a budget.

The Arithmetic of Amplification

Retry amplification is the ratio of total attempts to logical operations. If a client retries three times, it can turn one intended request into four. If three layers in a call path each retry three times, the worst case is twenty-seven attempts at the deepest dependency for a single user action. No individual layer made an unreasonable choice; the product of reasonable choices is a stampede.

Fixed backoff makes the problem worse in a specific way. When many clients fail at the same moment, they wait the same interval and return at the same moment. The resulting synchronized wave is often more damaging than the original fault because it arrives with full force precisely when the dependency has had no time to recover. Unbounded attempts convert a thirty-second disruption into a long tail of retries that outlives the incident.

Deadlines Before Backoff

Backoff without a deadline is just a slower unbounded loop. The governing constraint should be the total time the caller is willing to spend on the logical operation. That deadline must be propagated to the dependency, not merely enforced locally. A client that gives up after two seconds while the server continues working for thirty has not canceled the work; it has only stopped waiting for it.

Once a deadline exists, the retry decision becomes tractable. Compute the next backoff interval, add an estimate of the attempt duration, and retry only if that sum fits inside the remaining budget. Otherwise, fail fast and let the caller decide. A deadline also makes retries comparable across layers: each layer should consume from the same end-to-end budget rather than inventing its own.

Jitter Is Not Decoration

Jitter decorrelates clients so that a shared failure does not produce a shared return time. Full jitter, in which the sleep is drawn uniformly from zero to the exponential cap, is usually preferable to equal jitter because it spreads arrivals across the entire window rather than clustering them near the top. The server's own signal matters as well: when a response includes a Retry-After header, the client should treat it as the dependency's explicit vote on timing, subject to the caller's deadline.

A Budget Shared per Dependency

A per-dependency retry budget caps the additional load a client is willing to generate. The common pattern is a token bucket in which successful initial requests replenish tokens and retries consume them. A ratio of ten percent means that, across a fleet, retries may add at most one attempt for every ten initial requests. When the budget is exhausted, the client fails fast or routes to a fallback rather than adding pressure.

type RetryBudget = {
  tokens: number;
  capacity: number;
  refillPerSuccess: number;
  lastRefill: number;
};

function allowRetry(budget: RetryBudget, now: number): boolean {
  refill(budget, now);
  if (budget.tokens < 1) return false;
  budget.tokens -= 1;
  return true;
}

function recordSuccess(budget: RetryBudget, now: number): void {
  refill(budget, now);
  budget.tokens = Math.min(
    budget.capacity,
    budget.tokens + budget.refillPerSuccess
  );
}

The budget belongs to the dependency, not to the individual request. One hot endpoint should not be able to consume the entire allowance for a service that other callers also need. Keep one bucket per dependency and, where the traffic mix warrants it, one per endpoint class.

Measuring Retry Amplification

Instrumentation should make amplification visible before it becomes an incident. Track attempts divided by logical operations per dependency, and record the retry success rate: the share of retries that eventually succeed. A low success rate combined with a high attempt count is the signature of retries that are pure load. Also compare client-side attempt rates with server-side request rates; a widening gap indicates that retries are being generated faster than the dependency can absorb them.

Alert on sustained budget exhaustion rather than on individual failures. A budget that is empty for five minutes is a stronger signal than a spike in 503 responses, because it reflects the client's own judgment that the dependency is not recovering. Log the attempt number, the remaining deadline, and the budget state as structured fields so that post-incident analysis can reconstruct why each retry was permitted.

Notes for HTTP and Background-Job Clients

For HTTP clients, retry only idempotent methods or requests carrying an idempotency key, honor Retry-After, and cap attempts independently of the budget. For background-job clients, apply the budget at the scheduler or worker-pool level, and use lease or visibility timeouts so that a retry does not run concurrently with an attempt that is merely slow. In both cases, distinguish validation errors from transient ones; a retry cannot fix a malformed request and should not consume budget.

The Discipline of the Second Attempt

The retry is a useful mechanism precisely because most failures are transient. But the same mechanism, applied without accounting, is an amplifier. A deadline bounds the commitment, jitter spreads the return, and a shared per-dependency budget bounds the total additional load. Together they turn retries from an optimistic reflex into a controlled expenditure, which is the difference between surviving an outage and writing the next one.

All Articles

Related Articles

Three Verdicts for a Failed Event: Building a Quarantine That Explains Itself

Three Verdicts for a Failed Event: Building a Quarantine That Explains Itself

The Buffer Is a Borrowed Promise: Backpressure as an API Contract

The Buffer Is a Borrowed Promise: Backpressure as an API Contract

The Second Request Is the Real Test: Idempotency as an API Contract

The Second Request Is the Real Test: Idempotency as an API Contract