batch-job-patterns
Installation
SKILL.md
Batch Job Patterns
Overview
Batch jobs fail in subtle ways: two instances start simultaneously and corrupt shared state, a crash at item 50,000 restarts the job from item 1, a dead worker holds a lock forever, or an unbounded batch runs out of memory. Use this guide when designing, implementing, or reviewing batch processing systems.
When to use: Designing scheduled jobs, ETL pipelines, bulk data migrations, report generation, queue-draining workers, or any process that operates on a bounded or streaming set of records.
Quick Reference
| Pattern | Core Idea | Primary Red Flag |
|---|---|---|
| Distributed Locking | Only one instance runs at a time via SETNX / advisory lock | Multiple instances starting the same job simultaneously |
| Idempotent Checkpoint/Resume | Track cursor position so a crash restarts mid-batch, not from scratch | Job restarts from item 1 on every failure |
| Heartbeat / Dead Job Detection | Worker renews a lease; expired lease means worker is dead | Lock held forever by a crashed worker |
| Job Scheduling | Cron, interval, event-triggered, or priority queue dispatch | Drift, missed runs, or runaway overlapping executions |
| Graceful Shutdown | SIGTERM drains in-flight items before exit | Partial item writes or corrupted state on deploy/restart |
| Retry and DLQ | Per-item retry with skip-or-fail policy; unprocessable items route to DLQ | Silent discard of failed items, or one bad item halts the entire batch |
| Batch Size Optimization | Memory-bounded chunks with throughput tuning | OOM crashes or 1-item-at-a-time throughput bottlenecks |