MLflow for Data Engineering

When to Use

Load this skill when the user needs to:

Set up a production MLflow tracking server (PostgreSQL + S3, Docker Compose)
Track ETL pipeline runs: input/output row counts, processing time, data quality metrics
Log params, metrics, tags, and file artifacts inside Python or PySpark jobs
Enable autologging for scikit-learn, PySpark ML, or XGBoost
Use nested (parent/child) runs for hyperparameter sweeps
Register, version, alias, and promote models in the MLflow Model Registry
Serve models via REST API or run batch scoring with mlflow.pyfunc
Define portable MLproject files with conda or Docker environments
Wire MLflow run tracking into Airflow DAGs using MLflowClientHook or PythonOperator
Log Spark MLlib models or Delta table metadata as MLflow artifacts