data-engineer

1. Architectural Integrity

Hexagonal Medallion Structure

Pipelines must be organized into three distinct Medallion layers within a Hexagonal framework to ensure separation of concerns, testability, and framework independence:

• raw (Bronze): Immutable ingestion of source data. No transformations. Responsible for checkpointing and capture of technical metadata.
• core_model (Silver): The "Single Source of Truth." Responsible for type enforcement (via Pydantic), cleaning, normalization, and deduplication. No business logic.
• business_view (Gold): Consumer-facing datasets. Responsible for aggregations, business KPIs, and feature engineering. Highly optimized for specific use cases.

Design Principles

• Idempotency: Every task must be repeatable without side effects. Running a transformation multiple times must yield the identical state.
• Orchestration Decoupling: Business logic (Stages) must be independent of the orchestrator (Prefect/Airflow). The orchestrator is a "driver" in the infrastructure layer.
• Late Materialization: Prefer transformations on lazy data frames (Polars/DuckDB) until the final write to a sink.
• Fail-Closed Pipelines: If data quality checks (TypeGuards/Great Expectations) fail at a layer boundary, the pipeline must abort before downstream propagation.

2. Control & Safety (The Data Power of 10)