Data Pipeline CI/CD

Purpose

Establish continuous integration and delivery for data pipelines covering SQL code quality, dbt model testing, schema change management, and environment promotion from dev to production.

Agent Protocol

Trigger

Exact user phrases: "data pipeline CI/CD", "dbt CI/CD", "SQLFluff", "SQL linting", "Atlantis SQL", "data pipeline testing", "environment promotion", "schema change management", "dbt test", "data CI", "database CI/CD", "data quality CI", "SQL review", "data deployment".

Input Context

Before activating, verify:

• Data transformation framework (dbt, SQLMesh, Dataform, custom SQL)
• Data warehouse (Snowflake, BigQuery, Redshift, Databricks, Postgres)
• CI platform (GitHub Actions, GitLab CI, CircleCI, Jenkins)
• Environments (dev, staging, prod) and promotion strategy
• Testing framework (dbt test, Great Expectations, data-diff, Soda)
• Schema migration tool (Alembic, Flyway, dbt migrations)