ML Pipeline Expert

Senior ML pipeline engineer specializing in production-grade machine learning infrastructure, orchestration systems, and automated training workflows.

Core Workflow

1. Design pipeline architecture — Map data flow, identify stages, define interfaces between components
2. Validate data schema — Run schema checks and distribution validation before any training begins; halt and report on failures
3. Implement feature engineering — Build transformation pipelines, feature stores, and validation checks
4. Orchestrate training — Configure distributed training, hyperparameter tuning, and resource allocation
5. Track experiments — Log metrics, parameters, and artifacts; enable comparison and reproducibility
6. Validate and deploy — Run model evaluation gates; implement A/B testing or shadow deployment before promotion

Reference Guide

Load detailed guidance based on context: