Before the Abuse Arrives: A Rate-Limiter Ladder From Naive Counters to Token Buckets

An API without a rate limiter is an invitation to unexpected invoices. The first line of defense is not a sophisticated gateway product or a machine-learning anomaly detector. It is a small, explicit decision about how many requests a caller may make in a period of time, and what the service does when that number is exceeded.

The useful way to learn rate limiting is as a ladder. Each rung solves the previous rung's most visible failure, and each rung adds a cost. Start with the naive counter. Then fix its boundary burst with a fixed window. Then trade memory for accuracy with a sliding log. Then smooth the result with a token bucket. Only after that does distributed rate limiting deserve attention.

The Naive Counter and Its Boundary Problem

A first implementation often looks like a map keyed by IP address or API key. Each entry stores a count and a reset time.

const counts = new Map<string, { count: number; resetAt: number }>();

function allow(key: string, limit = 100, windowMs = 60_000) {
  const now = Date.now();
  const bucket = counts.get(key);
  if (!bucket || now >= bucket.resetAt) {
    counts.set(key, { count: 1, resetAt: now + windowMs });
    return true;
  }
  if (bucket.count >= limit) return false;
  bucket.count += 1;
  return true;
}

This is cheap and easy to explain. It is also easy to defeat. If the limit is 100 requests per minute, a caller can send 100 requests at 12:00:59 and another 100 at 12:01:00. The counter resets between the two bursts, so the service accepts 200 requests in roughly one second. That is not a theoretical edge case. It is the normal behavior of a fixed window at its boundary.

Fixed Window: Cheap and Predictable

The naive counter is a fixed-window limiter once the reset logic is made explicit. It remains a reasonable choice for coarse quotas: daily API budgets, per-account upload caps, or internal service limits where a brief boundary burst is acceptable. The operational appeal is strong. Memory is O(1) per key, the decision is O(1), and the behavior is easy to document.

The weakness is equally clear. Fixed windows permit bursts at boundaries, and the maximum burst can approach twice the configured limit if traffic is timed deliberately. For public endpoints that protect expensive downstream systems, that is often too much slack.

Sliding Log: Accuracy at a Memory Cost

A sliding log stores the timestamps of recent requests and counts only those inside the trailing window. On each request, prune old timestamps, check the remaining length, and append the current time.

const hits = new Map<string, number[]>();

function allowSliding(key: string, limit = 100, windowMs = 60_000) {
  const now = Date.now();
  const cutoff = now - windowMs;
  const log = (hits.get(key) ?? []).filter((t) => t > cutoff);
  if (log.length >= limit) {
    hits.set(key, log);
    return false;
  }
  log.push(now);
  hits.set(key, log);
  return true;
}

The sliding log removes the boundary burst because the window moves with time instead of resetting on a clock boundary. The cost is memory and CPU proportional to the number of requests in the window. At 100 requests per minute per key, that is manageable. At 10,000 per minute, the array pruning becomes the dominant cost. A ring buffer helps, but it does not change the fundamental trade-off.

Use a sliding log when accuracy matters and the limit is modest: login attempts, password resets, expensive report generation, or low-volume partner APIs.

Token Bucket: Smoothing Bursts Without Losing Capacity

A token bucket gives each key a capacity and a refill rate. Requests consume tokens. If no token is available, the request is rejected. The bucket refills continuously, so a caller can burst up to capacity and then settle into the sustained rate.

type Bucket = { tokens: number; updatedAt: number };
const buckets = new Map<string, Bucket>();

function allowBucket(key: string, capacity = 100, refillPerMs = 100 / 60_000) {
  const now = Date.now();
  const b = buckets.get(key) ?? { tokens: capacity, updatedAt: now };
  const refill = (now - b.updatedAt) * refillPerMs;
  b.tokens = Math.min(capacity, b.tokens + refill);
  b.updatedAt = now;
  if (b.tokens < 1) {
    buckets.set(key, b);
    return false;
  }
  b.tokens -= 1;
  buckets.set(key, b);
  return true;
}

This is the workhorse for public APIs. It allows short bursts, which real clients need after reconnects or batch jobs, while enforcing a steady average. The configuration is two numbers: capacity and refill rate. Those numbers should come from downstream capacity and business policy, not from habit.

A Small Benchmark: What You Pay for Accuracy

The following figures come from a representative single-process Node.js 22 microbenchmark on a 2026-era laptop. They measure limiter decisions only, not network or database latency. Treat them as relative guidance, not service-level promises.

Strategy	Time complexity	Representative throughput	Memory per active key	Best fit
Fixed window	O(1)	About 2.0M decisions/sec	Small	Daily quotas, coarse limits
Sliding log	O(n)	About 180k decisions/sec at 100 entries	Proportional to limit	Low-volume, high-value endpoints
Token bucket	O(1)	About 1.7M decisions/sec	Small	Public APIs, bursty clients

The sliding log's cost scales with the configured limit. At 1,000 requests per minute, pruning arrays dominates. At 100, the accuracy may be worth it. The benchmark should be rerun in the target runtime because memory allocation and garbage collection change the shape of the curve.

Failure Modes and Operational Choices

A rate limiter is a policy enforcement point, so its failure behavior matters. Public read endpoints often fail open with logging when the limiter store is unavailable, because rejecting all traffic is worse than a temporary overshoot. Authentication and payment endpoints usually fail closed. That decision should be made per route, not inherited from a global default.

Key selection is another policy decision. IP addresses are easy but collide behind NAT and mobile carriers. User IDs are more precise but require authentication first. API keys are useful for partner quotas. A composite key can separate anonymous traffic from authenticated traffic without letting one group consume the other's budget.

Response headers deserve the same care. A Retry-After header tells well-behaved clients when to try again. Limit and remaining headers make the quota legible. Retries need idempotency keys, or a limiter can turn a transient failure into duplicate work.

Why Distributed Rate Limiting Is Harder

A single-process limiter is a data structure. A distributed limiter is a coordination problem. Put the counter in Redis and every decision adds a network hop. Check and increment must be atomic, usually through a Lua script or a conditional write. Clocks drift between nodes. Multiple regions create independent buckets unless a global store is consulted, and that store becomes a latency and availability dependency.

A common compromise is a local token bucket per node plus periodic synchronization with a shared budget. This reduces latency and accepts bounded overshoot. The important question is not whether overshoot exists. It is how much overshoot the downstream system can tolerate. For a US service with users on both coasts and infrastructure in several regions, that answer should be written down before the first abuse incident.

A Practical Ladder for 2026

Start with fixed windows for coarse quotas and internal limits. Move to a sliding log when boundary bursts threaten a sensitive endpoint and the limit is low enough to afford the memory. Use token buckets for public APIs that need burst tolerance and steady enforcement. Add distributed coordination only when a single process cannot enforce the policy, and define the acceptable failure mode before selecting the store.

The ladder is not a maturity model. It is a set of trade-offs. The right rung is the simplest one that matches the abuse model, the downstream capacity, and the cost of being wrong.