Apache Airflow AI Skill Guide

Overview & Engine Architecture

Airflow schedules DAGs of tasks executed by workers; the scheduler parses DAG files, the metadata DB stores run state, and executors (Local/Celery/Kubernetes) run task instances. Agents write idempotent tasks, set explicit retries/timeouts, avoid top-level heavy I/O in DAG files, and pass data via XCom sparingly (or external storage).

DAG file -> scheduler -> executor/workers
                |
           metadata DB (runs, XCom)
                |
           task logs / sensors

When to use this skill

• Time-based or data-aware batch pipelines
• Orchestrating dbt, Spark, warehouse SQL, ML batch jobs
• Backfills with clear logical dates