Data Engineering

Scope: Data pipeline design, transformation (dbt), and orchestration (Dagster). For analytics and dashboards built on top of these pipelines, see analytics-workflow. For analytical query patterns, see duckdb-patterns and postgres-patterns.

When to Activate

• Building an ELT pipeline (Extract → Load → Transform)
• Writing dbt models or tests
• Setting up Dagster assets, jobs, or sensors
• Designing incremental data processing (avoid full table scans)
• Adding data quality checks / anomaly detection
• Organizing a data warehouse with staging → intermediate → mart layers

Modern Data Stack Overview