AutoML and Hyperparameter Optimization

This skill covers designing sound hyperparameter searches and using AutoML tooling (Optuna, Ray Tune, PyCaret, time-series AutoML libraries) without bypassing problem framing, validation design, or explainability.

Workflow for Running a Hyperparameter Search

1. Define the target metric and baseline first — Pick the metric before selecting tooling, and train a simple baseline (linear model, random forest, or naive time-series forecast) with a fixed, minimal search.
2. Design the validation scheme — Use nested cross-validation or a final untouched test split for any model-selection claim; use time-aware splits (never shuffled) for time-series problems.
3. Fit preprocessing inside the fold — Fit scalers, encoders, and imputers only on the training portion of each fold to prevent leakage.
4. Define a structured search space — Use log-scale ranges for learning rates, regularization strength, and tree counts; keep ranges domain-informed rather than arbitrarily broad.
5. Choose the right tool — Optuna or Ray Tune for custom training loops with pruning and distributed trials; PyCaret for a quick low-code comparison on a straightforward tabular problem; a time-series-specific library (AutoTS, Merlion, PyAF) when seasonality and horizon handling need first-class support.
6. Run with resource limits and pruning — Set a trial or time budget and use early stopping/pruning so bad trials don't consume the full budget.
7. Track every run — Log datasets, splits, metric definitions, random seeds, library versions, and the search space itself to MLflow, Weights & Biases, TensorBoard, or an equivalent tracker.
8. Report against the baseline — Compare the selected model to the baseline and at least one non-AutoML alternative before calling it production-ready.

Experiment Design