data-engineering-observability
Installation
SKILL.md
Pipeline Observability
Tracing and metrics for data pipelines using OpenTelemetry and Prometheus. Instrument code for visibility into performance, errors, and data lineage.
Quick Reference
| Tool | Purpose | What it Measures |
|---|---|---|
| OpenTelemetry | Distributed tracing | Pipeline stages, latency, dependencies |
| Prometheus | Metrics | Throughput, error rates, resource utilization |
| Grafana | Visualization | Dashboards combining traces + metrics |
Why Observable?
- Debugging: Trace failed records through pipeline stages
- Performance: Identify bottlenecks, optimize slow transformations
- Reliability: Set alerts on error rates, SLA breaches
- Cost: Track resource usage, optimize expensive operations
- Compliance: Audit trail of data transformations