Background Jobs

Overview

Reliable async task execution: enqueue work, process it in workers decoupled from the request cycle, and survive crashes/retries without corrupting state. This file covers job queues, retries/backoff, DLQs, scheduling, worker pools, and delivery guarantees.

For pub/sub, event sourcing, CQRS, sagas, and streaming brokers (Kafka/Pulsar), see loom-event-driven — don't reimplement those here.

The two invariants everything hangs off

1. At-least-once is the default. Design every handler to be idempotent. Redis-backed queues (Sidekiq, BullMQ, Celery+Redis), SQS standard, and any queue that retries on crash will deliver a job more than once. "Exactly-once delivery" does not exist over an unreliable network; the achievable goal is exactly-once effect = at-least-once delivery + idempotent handler.
2. Ack after success, never before. The job must stay owned by the worker until the side effect is durably committed. Ack-then-process = at-most-once = silent data loss on crash. Process-then-ack = at-least-once = duplicates you dedup away. Always choose the latter.

Idempotency: the non-negotiable pattern

Derive a stable key from the job's business identity (not a random UUID per enqueue), and gate the side effect on it.