Checkpoint & Resume Processing

Exactly-once processing semantics with distributed coordination for file-based data pipelines.

When to Use This Skill

• Processing large file batches across multiple workers
• Need to resume after worker crashes
• Preventing duplicate processing of the same file
• Tracking processing progress and statistics

Core Concepts

The pattern provides:

• Atomic file claiming (only one worker processes each file)
• Status tracking (pending, processing, completed, failed)
• Automatic retry with configurable limits
• In-memory fallback for development/testing