Broadway Data Pipelines

RULES — Follow these with no exceptions

1. Use Broadway.Message.failed/2 for errors — never raise in handle_message/3
2. Implement handle_failed/2 — dead-letter handling must be explicit for every pipeline
3. Use batchers for database inserts — don't insert one-by-one; batch size of 100 is a good default
4. Configure :status in start_link — set :max_restarts, :max_seconds for production resilience
5. Test with Broadway.Test.push_message/2 — verify each message type including failures
6. Wire telemetry — attach handlers to Broadway's telemetry events for observability

End-to-End Setup Workflow

Follow these steps in order when building a new Broadway pipeline: