Mage AI Pipelines

When to Use

Activate this skill when the task involves:

• Authoring data pipelines in Mage AI using block-based composition
• Configuring io_config.yaml connection profiles for PostgreSQL, S3, BigQuery
• Mixing SQL transformer blocks with Python loaders/exporters in the same pipeline
• Setting up schedule, event, or API triggers and backfills
• Building streaming pipelines (Kafka → transform → sink)
• Integrating dbt projects as blocks inside a Mage pipeline
• Running Spark workloads from Mage with executor_type: spark
• Deploying Mage with Docker Compose or Kubernetes
• Writing block-level callbacks (Slack, audit table) and unit-testing blocks in CI

Mage vs Airflow vs Prefect — when to choose Mage: