Data Pipelines

Orchestration, transformation, and validation patterns for production data pipelines.

When to Activate

• Writing Airflow DAGs, operators, sensors, or XComs
• Building dbt models, sources, tests, or macros
• Designing incremental vs full-load strategies
• Implementing idempotent pipeline runs
• Validating data quality with dbt tests or Great Expectations
• Orchestrating multi-step ELT/ETL workflows
• Debugging failed runs, backfills, or data freshness issues

ETL vs ELT Decision