CodeOrigin All articles
Distributed Systems

Three Verdicts for a Failed Event: Building a Quarantine That Explains Itself

CodeOrigin
Three Verdicts for a Failed Event: Building a Quarantine That Explains Itself

Photo: Intel Free Press, CC BY 2.0, via Wikimedia Commons

Retries are a wager. Each one bets that an event failed for a reason that will disappear. Most of the time, that wager is sound: a network path recovers, a dependency returns, a rate limit resets. But a poison message is what happens when the wager is repeated past the point of evidence. The event is not waiting for conditions to improve. It is malformed, unauthorized, or semantically impossible, and no amount of patience will make it valid. A quarantine is not a trash can. It is a verdict with a paper trail.

The first principle is simple: a failed event needs a decision, not a counter. A retry count records effort. A quarantine record explains outcome. Once that distinction is clear, the implementation follows: classify failures by meaning, preserve redacted evidence, expose controlled replay, and define the threshold at which a human must look.

A Failed Event Needs a Verdict, Not a Counter

Most retry systems collapse every error into one loop. They increment an attempt count, wait with backoff, and eventually move the event somewhere labeled dead. That label is doing too much work. It hides at least three different situations: an event that might succeed later, an event that can never succeed, and an event whose failure nobody has understood yet.

A better model uses three verdicts. retry means the failure is plausibly transient and the event remains eligible for automated attempts. reject means the event is permanently invalid and must not be retried without a corrected replacement. review means the system cannot prove either case, so the event moves to a human queue instead of circulating indefinitely. These verdicts are not error severities. They are statements about future possibility.

Classify at the Boundary Where Meaning Exists

Classification should happen where the system still knows what the event was supposed to mean. A schema validation failure is not a timeout. An expired credential is not a temporary network fault. A business invariant violation is not a queue backlog. The classification function should read the failure kind, not the stack trace alone.

A small TypeScript shape is enough to make the contract concrete:

type Verdict = 'retry' | 'reject' | 'review';

function classify(error: Failure): Verdict {
  if (error.kind === 'auth' || error.kind === 'schema') return 'reject';
  if (error.kind === 'timeout' || error.kind === 'rate_limit') return 'retry';
  return 'review';
}

Production rules will be richer, but the principle holds: permanent failures should be rejected early, transient failures should keep their retry eligibility, and ambiguous failures should go where uncertainty is visible. The worst outcome is not a rejection. It is a permanent failure disguised as a temporary one, consuming retry budget and polluting incident signals.

Store Evidence, Not Payload

A quarantine record must answer four questions: what failed, why it was classified that way, what was attempted, and who can act on it. That does not require storing the raw event forever. In fact, raw storage is often the wrong default. Events can contain credentials, personal data, payment details, or internal identifiers that should not sit in a debug bucket for months.

Redact before persistence. Keep the envelope metadata, the failure class, a stable error hash, the classification rationale, attempt timestamps, and a redacted representation of the payload. Store the original only when policy allows it, under stricter access controls and a shorter retention period. The quarantine record should be immutable enough to serve as evidence and annotated enough to support investigation. Separate the event from the annotations so that reviewers can add notes without rewriting history.

Replay Is a Controlled Operation

Replay tooling is where well-meaning quarantine systems often become accidental retry engines. A replay button that can select ten thousand events and fire them at production is not a recovery tool. It is an incident generator.

Guardrails should be explicit. Replay starts with a dry run that reports how many events would be sent, to which target, and under which schema version. Single-event replay should be the default. Batch replay should have a cap, a rate limit, and an approval step. Every replay should create a new attempt identifier while preserving the original record, and idempotency keys should prevent duplicate side effects. Permanent rejects should not be replayed without a corrected replacement event. Even then, the replay should target a controlled environment first when the failure suggests a contract change rather than a transient fault.

Set the Human Threshold Explicitly

A quarantine without an escalation rule becomes a silent graveyard. The fix is to define, in advance, which conditions require a person. Unknown classifications, events older than a defined age, spikes in reject volume, and failures on high-value event types are reasonable triggers. A permanent schema rejection may need an owner to fix the producer, but it does not need an on-call page at 2 a.m. An unexplained auth failure on a payment event probably does.

The threshold should be tied to actionability. If a human can correct the event, repair the producer, or make a policy decision, escalation is justified. If the only possible action is to acknowledge that the event is invalid, the rejection path is enough. The goal is not to eliminate automation. It is to stop automation from pretending that every failure has a future.

The Quarantine Is a Product Surface

Treat quarantine as a small internal product. It needs a dashboard, an owner, retention rules, and a review workflow. Track reject rate by producer, review backlog age, replay success rate, and the number of events that exhausted retries without a verdict. Those metrics reveal whether the classification rules are learning or merely hiding uncertainty.

The first principle remains unchanged: an event that cannot succeed should not circulate, and an event whose failure is not understood should not disappear. A quarantine earns its name when it makes that distinction visible, reversible where appropriate, and accountable in every other case.

All Articles

Related Articles

The Buffer Is a Borrowed Promise: Backpressure as an API Contract

The Buffer Is a Borrowed Promise: Backpressure as an API Contract

The Second Request Is the Real Test: Idempotency as an API Contract

The Second Request Is the Real Test: Idempotency as an API Contract

Failure as an Interface: A Taxonomy of Errors Callers Can Act On