ML Failure Audit

Purpose

Audit ML failures from supplied artifacts without assuming the headline explanation is true. Use this skill when the user provides a repo, logs, W&B/MLflow/TensorBoard exports, CI artifacts, config files, or reports and asks for a diagnosis, go/no-go decision, or structured output.

Core Workflow

1. Locate evidence

   • Find the repo root, log files, metric exports, configs, test definitions, golden values, and any requested output schema.
   • Treat raw logs, raw telemetry, configs, and source code as higher-trust than reports, PR text, summaries, or generated JSONs.

2. Classify the failure

   • Separate model/convergence signals from correctness, data, config, runtime, infra, and metric-policy signals.
   • Do not label a failure as model/convergence regression when only a performance, timeout, logging, or tolerance gate failed and correctness/loss checks passed.