assuring-data-pipelines
Installation
SKILL.md
Assuring Data Pipelines
Data quality validation and observability for production data pipelines. Ensure data correctness with validation frameworks and gain visibility into pipeline performance with tracing and metrics.
Two Pillars of Pipeline Assurance
| Concern | Tools | Purpose |
|---|---|---|
| Data Quality | Great Expectations, Pandera | Validate schema, distributions, business rules |
| Observability | OpenTelemetry, Prometheus | Trace execution, monitor health, alert on issues |
Why Both Matter
- Quality without observability: You know data is wrong, but can't trace where it broke
- Observability without quality: You see pipeline latency, but miss silent data corruption
- Together: Complete feedback loop from detection → diagnosis → remediation