Data engineering best practices

Use when designing, building, or reviewing data engineering systems and pipelines.

Mental model

• A data system is only useful if it is correct, observable, recoverable, and understandable.
• Optimize for trustworthy pipelines first, then throughput and convenience.
• Treat schemas, freshness expectations, ownership, and failure modes as part of the product contract, not just implementation detail.

Ingestion

• Make ingestion boundaries explicit: source system, extraction method, cadence, and delivery guarantees.
• Prefer idempotent ingestion so retries do not silently duplicate data.
• Capture source metadata that helps with debugging and replay: ingest time, source offsets, batch ids, event ids, and version markers.

Batch vs streaming