Data Pipeline Engineering Skill

🔴 AI FIRST Quality Principle

Apply the AI FIRST principle: never accept first-pass quality. Minimum 2 iterations. Read all output, improve every section. No shortcuts.

Purpose

Expert knowledge in designing robust ETL (Extract, Transform, Load) pipelines for automated data processing, focusing on reliability, monitoring, and maintainability.

Core Principles

1. Idempotency - Pipeline runs produce same results
2. Observability - Full visibility into pipeline health
3. Error Recovery - Graceful handling of failures
4. Version Tracking - Track all data changes
5. Monitoring - Real-time pipeline health checks