Pipeline Design

Purpose

Design data pipelines that reliably move, transform, and deliver data from source systems to consumption layers. Covers ETL vs ELT pattern selection, orchestration tool choice, batch vs streaming trade-offs, idempotency guarantees, data quality checkpoints, and lineage tracking.

Scope Constraints

Reads pipeline configurations, DAG definitions, orchestration manifests, and infrastructure specs for analysis. Does not execute pipelines, deploy infrastructure, or modify production configurations.

Inputs

• Source systems and their data formats (databases, APIs, event streams, files)
• Destination systems (warehouse, lake, feature store, BI tool)
• Data volume and velocity (rows/day, events/second, payload size)
• Freshness requirements (real-time, near-real-time, hourly, daily)
• Existing infrastructure (cloud provider, orchestration tools, current pipelines)
• Team size and expertise (SQL-heavy? Python-heavy? Platform team available?)