Market Data Engineering

objective

Execute market data engineering work with reproducible research, explicit controls, and deployable outputs.

workflow

1. define source contracts, schema versions, and freshness objectives.
2. ingest data with replay support and deterministic normalization.
3. validate keys, timestamps, and point-in-time join behavior.
4. monitor quality metrics continuously and quarantine degraded feeds.
5. publish only when lineage, ownership, and quality thresholds are satisfied.

required diagnostics

• freshness, completeness, null-rate, and duplicate-rate trends.
• schema drift and breaking-change frequency across sources.
• point-in-time join integrity for features and labels.
• backfill and replay consistency versus canonical snapshots.
• spread resilience under liquidity withdrawal