DE Production Readiness

When to Use

Load this skill when the user needs to:

• Make a data pipeline idempotent and safe to retry or re-run
• Implement exponential backoff, retry logic, or error classification in Python or Airflow
• Set up SLA monitoring, alerting (Slack, PagerDuty), or Prometheus metrics for a pipeline
• Define or enforce data freshness SLAs and automate freshness checks
• Add structured logging, OpenTelemetry tracing, or observability metrics to a pipeline
• Build or audit data reconciliation: row counts, checksums, duplicate detection
• Implement graceful degradation — circuit breakers, stale-data fallback, skipping bad partitions
• Perform a blue/green table swap, schema migration, or define a rollback procedure

1. Idempotency Patterns

An idempotent pipeline produces the same final state whether it runs once or ten times for the same logical batch. This is the single most important property of a reliable pipeline because it makes retries, backfills, and incident recovery safe.