Data Workflow Orchestration

Purpose

Design robust workflow orchestration for data pipelines. Select the right orchestrator (Airflow, Prefect, Dagster), design DAGs with proper task typing, configure executors for scale, implement CI/CD for pipeline code, and set up observability with alerting.

Agent Protocol

Trigger

Exact user phrases: "Airflow", "Prefect", "Dagster", "Luigi", "Argo Workflows", "DAG", "scheduler", "task", "executor", "sensor", "operator", "pipeline orchestration", "data pipeline scheduling", "workflow CI/CD", "pipeline alerting", "task retry".

Input Context

Before activating, verify:

• Orchestrator preference (Airflow, Prefect, Dagster, Argo)
• Deployment environment (K8s, VMs, hybrid)
• Task types (Python, SQL, Spark, dbt, custom)
• Scale (DAGs count, task count per DAG, execution frequency)
• Infrastructure (database, message broker, logging, secrets)
• Team size and expertise (Python vs YAML vs UI)