CodeOrigin All articles
Distributed Systems

The Buffer Is a Borrowed Promise: Backpressure as an API Contract

CodeOrigin
The Buffer Is a Borrowed Promise: Backpressure as an API Contract

Queues are usually added with good intentions. A buffer between a producer and a consumer absorbs bursts, isolates slow dependencies, and turns a hard failure into a delay. That is the theory. In practice, a queue is a promise about future capacity, and every promise has a creditor. If the buffer is unbounded, the creditor is the caller, and the bill arrives as latency, memory pressure, and retry storms.

The Arithmetic Behind the Buffer

At steady state, a service has a maximum sustainable throughput. Call it mu. Demand arrives at lambda. If lambda stays below mu, a small buffer smooths jitter. If lambda exceeds mu, backlog grows at roughly lambda minus mu. No queue design changes that arithmetic; it only decides where the excess is stored. An unbounded queue stores it in memory and in caller patience. A bounded queue forces a decision at the boundary: wait, reject, or slow the producer.

Little's law makes the cost legible. The number of items in the system equals arrival rate multiplied by time in system. When a queue grows, either throughput rises or latency rises. If throughput cannot rise, latency must. Callers experience that as timeouts, retries, and duplicated work. Retries then add more arrivals, which is the feedback loop that turns a slowdown into an outage.

This is why backpressure is not a library choice. It is a statement about what the service will do when demand exceeds capacity. The queue is merely the place where that statement becomes visible.

The Contract Before the Code

An API that accepts work has to say what it will do when it cannot keep up. That statement is the contract. For a US-based team running a checkout or account service, the contract should answer five questions:

Without those answers, clients assume infinite patience and infinite capacity. They retry aggressively, open more connections, and amplify the original problem. With those answers, the service can fail in a way that callers can act on.

A Bounded Gate in Node.js

The smallest useful backpressure mechanism is a gate with two limits: active work and waiting work. The gate admits a request only if capacity exists. Otherwise it either queues the request for a short, bounded interval or rejects it explicitly. The following Node.js implementation is deliberately small, but it preserves the essential property: no caller waits forever.

class BoundedGate {
  constructor({ limit, maxWaiting }) {
    this.limit = limit;
    this.maxWaiting = maxWaiting;
    this.active = 0;
    this.waiters = [];
  }

  acquire(waitMs = 1500) {
    if (this.active < this.limit) {
      this.active += 1;
      return Promise.resolve(this.release.bind(this));
    }

    if (this.waiters.length >= this.maxWaiting) {
      return Promise.resolve(null);
    }

    return new Promise((resolve) => {
      const waiter = { resolve, timer: null };
      waiter.timer = setTimeout(() => {
        const index = this.waiters.indexOf(waiter);
        if (index !== -1) this.waiters.splice(index, 1);
        resolve(null);
      }, waitMs);
      this.waiters.push(waiter);
    });
  }

  release() {
    this.active -= 1;
    const next = this.waiters.shift();
    if (next) {
      clearTimeout(next.timer);
      this.active += 1;
      next.resolve(this.release.bind(this));
    }
  }
}

Wiring it into an HTTP handler keeps the contract close to the boundary:

const gate = new BoundedGate({ limit: 64, maxWaiting: 128 });

const server = http.createServer(async (req, res) => {
  const lease = await gate.acquire(1500);

  if (!lease) {
    res.writeHead(429, {
      'content-type': 'application/json',
      'retry-after': '2',
      'x-capacity-limit': '64'
    });
    return res.end(JSON.stringify({
      error: 'capacity_exceeded',
      retryAfterSeconds: 2
    }));
  }

  try {
    await handle(req, res);
  } finally {
    lease();
  }
});

The gate does not make the service faster. It makes the service honest. Active work is capped at 64. At most 128 callers may wait briefly. Everyone else receives a rejection that says exactly what happened and when to try again.

Signals That Callers Can Act On

A rejection is only useful if it carries information. HTTP 429 is appropriate when a specific caller or tenant has exceeded an allowance. HTTP 503 is appropriate when the service as a whole cannot accept work. In both cases, Retry-After gives the caller a concrete delay rather than an invitation to hammer the endpoint.

A response body should be machine-readable. An error code such as capacity_exceeded is more useful than a prose sentence. Headers such as x-capacity-limit and x-queue-depth turn the queue from a private implementation detail into an observable part of the contract. Callers can then back off, batch work, or route to a fallback instead of guessing.

Idempotency belongs here as well. If a caller retries a rejected request, the service must know whether that retry is safe. A payment or order-creation endpoint that rejects under load but accepts a duplicate retry has replaced a capacity problem with a correctness problem.

Shedding Is a Decision About Whom to Disappoint

Load shedding is often treated as an error path. It is better understood as a policy decision. When capacity is scarce, the service must choose which work matters. A retail API on a high-traffic sales day might protect checkout and authentication while delaying catalog refreshes or recommendation rebuilds. That choice should be documented, tested, and visible in metrics.

Admission control belongs at the edge, but it must be paired with limits inside the service. A gateway that admits 10,000 requests per second into a service that can handle 500 has not solved the problem; it has moved it. The gate must sit near the work, where the real concurrency limit is known.

Operating the Truth

The contract is only credible if it is measured. Track active work, waiting work, queue wait time, rejection rate, and retry rate. Alert on the time callers spend waiting, not only on CPU. A service can have idle CPU and a saturated queue if the bottleneck is a downstream dependency. Load tests should verify that the rejection path works under pressure, not just that the happy path handles a burst.

Backpressure is not an admission of weakness. It is the operational form of a simple first principle: capacity is finite, and demand must be told so. The working code is the gate that says it clearly. The product is the promise that the gate keeps.

All Articles

Related Articles

The Second Request Is the Real Test: Idempotency as an API Contract

The Second Request Is the Real Test: Idempotency as an API Contract

Failure as an Interface: A Taxonomy of Errors Callers Can Act On

Cursors That Keep Their Word: Stable Pagination Under Concurrent Writes

Cursors That Keep Their Word: Stable Pagination Under Concurrent Writes