Cron Is Not a Scheduler: Durable Background Jobs From Guarantees Upward

Every team has one. A crontab on a single instance that runs a nightly reconciliation script. It worked for two years. Then the instance restarted mid-run, the script threw halfway through a batch of invoices, and nobody noticed for nine days because cron does not send a message when it fails — it simply does not do the thing, silently, forever.

The instinct is to reach for a queue product. The better first move is to write down the guarantees the work actually needs, because those guarantees determine which product is adequate and which is expensive overkill. A timer and a job system are different tools that happen to share a schedule.

What Cron Does Not Provide

Cron answers exactly one question: when should this process start? It says nothing about whether the previous run finished, whether this run succeeded, or what to do if it did not. Four consequences follow, and all four are familiar:

• Overlapping runs. A slow night collides with the next scheduled start, and two processes operate on the same rows.
• No record. Failure leaves no artifact beyond a log line in a file that rotates.
• No retry. A transient network error is indistinguishable from a permanent business-logic failure.
• No pacing. A backlog cannot be inspected, drained gradually, or paused during an incident.

Start From Guarantees

Before choosing a tool, state what the system must promise. For most business jobs, the list looks like this:

• Durable intent. A job that has been accepted is recorded in durable storage before any worker touches it.
• At-least-once delivery. Exactly-once execution is a distributed-systems fantasy for anything with external side effects; design for at-least-once and make handlers safe to repeat.
• Idempotency. Repeating a job produces the same end state as running it once.
• Retry with backoff. Transient failures are retried on a growing delay with jitter, up to a limit.
• Visibility. Anyone can answer: how many jobs are pending, how old is the oldest, what is failing, and why.
• A terminal state. Jobs that exhaust retries land somewhere a human will look.

Notice that none of these name a product. They are properties. A system that has all six is a job system regardless of what it is built on.

A Minimal Durable Queue

A relational table is an adequate and frequently underrated queue for workloads in the thousands-to-millions-per-day range. The schema below carries every guarantee on the list:

CREATE TABLE jobs (
  id           bigserial PRIMARY KEY,
  type         text        NOT NULL,
  payload      jsonb       NOT NULL,
  status       text        NOT NULL DEFAULT 'pending',
  run_at       timestamptz NOT NULL DEFAULT now(),
  attempts     int         NOT NULL DEFAULT 0,
  max_attempts int         NOT NULL DEFAULT 8,
  locked_until timestamptz,
  locked_by    text,
  last_error   text,
  created_at   timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX jobs_ready_idx ON jobs (run_at) WHERE status = 'pending';

The critical piece is the claim query. FOR UPDATE SKIP LOCKED lets many workers pull work concurrently without blocking each other or handing the same row to two of them:

UPDATE jobs SET status = 'running',
                locked_until = now() + interval '5 minutes',
                locked_by = $1,
                attempts = attempts + 1
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'pending' AND run_at <= now()
  ORDER BY run_at
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
RETURNING *;

The locked_until column is the lease. If a worker dies mid-job, the row becomes claimable again when the lease expires — no separate reaper needed for correctness, only for hygiene.

The Worker Loop

The worker is deliberately boring. Dispatch on type, complete on success, schedule a retry on failure:

async function tick(workerId) {
  const job = await claim(workerId);
  if (!job) return false;
  try {
    await handlers[job.type](job.payload);
    await complete(job.id);
  } catch (err) {
    await reschedule(job, err);
  }
  return true;
}

Retry policy belongs in one place, not scattered across handlers:

const delayMs = Math.min(2000 * 2   job.attempts, 3600000);
const jittered = delayMs * (0.5 + Math.random());

The exponent gives you a bounded escalation; the jitter prevents a thousand workers from retrying in lockstep after a shared dependency recovers.

Idempotency Is a Design Constraint, Not a Library

At-least-once delivery is the honest guarantee, which means every handler must tolerate being invoked twice. Three patterns cover nearly every case: an upsert keyed on a natural identifier from the payload; a uniqueness constraint on an external reference so a duplicate insert fails loudly and harmlessly; and a conditional update that only transitions state forward, such as WHERE status = 'open'.

For anything with an external side effect — charging a card, sending an email — pass an idempotency key to the provider and store it alongside the job. The provider's deduplication window is part of your contract, so document its length.

Where Observability Belongs

A queue without metrics is a cron job with extra steps. Four signals carry most of the diagnostic weight: pending depth, age of the oldest pending job, a histogram of attempts per job type, and failure rate grouped by type. Alert on the age of the oldest pending job rather than on depth — a deep queue that is draining is healthy, while a shallow queue whose oldest item is four hours old is not.

Ship a structured log line per attempt with job id, type, attempt number, duration, and error class. That single line makes the difference between an incident that takes ten minutes and one that takes a weekend.

Failure Drills Worth Running

Guarantees are claims until they are tested. Four drills, run quarterly, will find more defects than any review:

1. Kill a worker mid-job. Confirm the lease expires and the job is reclaimed rather than stranded in running.
2. Deliver the same job twice. Confirm the handler produces one side effect, not two.
3. Poison a payload. Confirm retries exhaust, the job lands in a dead-letter state, and an alert fires.
4. Stop the queue for an hour. Confirm the backlog drains at a rate you can predict and nothing downstream times out.

Each drill should be scripted and repeatable, not an improvisation during an outage.

The Path From Here

Once the table, the claim query, and the worker loop exist, swapping the storage layer for a managed broker is a contained change behind the same interface. That is the argument for starting from guarantees: the properties stay constant while the plumbing becomes replaceable. A schedule is a wish. A job that can be retried, observed, and accounted for is a commitment — and the difference is what keeps the nightly reconciliation from failing quietly for nine days.