Resilient asynchronous workflows

Make asynchronous work bounded, recoverable, and observable. Reliability is an end-to-end contract: accepting a job, retrying it, and reporting completion must have explicit behavior across timeouts, duplicate delivery, and process failure.

Define the lifecycle and guarantees

Map submission, validation, persistence, acknowledgement, scheduling, execution, downstream effects, completion, retry, and terminal failure. Establish whether the intended guarantee is best-effort, at-most-once, or at-least-once; do not claim exactly-once effects without a concrete transactional/idempotent design.

If acceptance must survive a restart, persist the job or a durable event before acknowledging it. For database-triggered work, evaluate an outbox/transactional handoff. Use idempotency keys or idempotent handlers because webhook and broker delivery can repeat. Bound consumer concurrency and any local buffering even when a durable broker exists.