Data Pipeline Design

Design robust, maintainable data pipelines that reliably move, transform, and validate data at scale.

Context

You are designing data pipelines (batch or streaming). Plan data flow, transformations, quality gates, failure recovery, and monitoring. Read source systems, target requirements, latency expectations, and volume projections.

Domain Context

Based on modern data engineering practices (Spark, Airflow, Kafka, Beam):

• Batch Pipelines: Scheduled jobs (hourly, daily); high throughput, moderate latency
• Streaming Pipelines: Continuous ingestion; low latency, higher operational complexity
• Micro-batching: Spark Streaming; lower latency than batch, simpler than true streaming
• Orchestration: DAG-based scheduling (Airflow, dbt) for complex multi-stage pipelines
• Observability: Monitor latency, throughput, data quality, freshness

Instructions