Data Engineering Principles

Guidelines for building reliable, scalable data pipelines and platforms.

When to Invoke

• Designing data pipelines (ETL/ELT)
• Evaluating batch vs stream processing
• Data quality and governance requirements
• Data warehouse/lake architecture decisions

Pipeline Architecture

Design Principles

1. Idempotent pipelines — re-running produces same result. Use upserts, not inserts.
2. Schema evolution — handle new fields without breaking consumers.
3. Exactly-once processing — deduplication at ingestion, idempotency keys.
4. Incremental processing — process only new/changed data, not full reloads.