data-engineering-orchestration
Installation
SKILL.md
Pipeline Orchestration
Workflow orchestration tools for data pipelines: Prefect, Dagster, and dbt. These tools handle scheduling, dependency resolution, retries, monitoring, and state management for production data pipelines.
Quick Comparison
| Tool | Paradigm | Best For | Learning Curve |
|---|---|---|---|
| Prefect | Flow-based | Pythonic workflows, quick prototypes, cloud-first | Moderate |
| Dagster | Asset-based | Data asset lineage, reproducibility, type checking | Steeper |
| dbt | SQL transformations | Analytics engineering, ELT, data warehouses | Low (SQL-focused) |
| FlowerPower | Hamilton DAGs | Lightweight batch ETL, configuration-driven pipelines | Low-Moderate |
When to Use Which?
-
Prefect: You want Python code flexibility, Prefect Cloud UI, and quick setup. Good for general-purpose data pipelines, ETL, and API integrations.
-
Dagster: You care about data asset observability, type safety, and reproducibility. Good for complex data platforms with clear asset dependencies.