Autoresearch — Autonomous ML Experimentation

Run an autonomous experiment loop that modifies training code, runs experiments with a fixed time budget, measures results, and keeps/discards changes based on a target metric. Inspired by Karpathy's autoresearch pattern.

Core Loop

FOREVER:
  1. Analyze current state (git log, results.tsv, best val_bpb)
  2. Generate hypothesis (architecture change, hyperparameter tweak, optimizer mod)
  3. Modify train.py with the change
  4. git commit -m "experiment: <description>"
  5. Run: bash autoloop.sh run (5-min training, captures all metrics)
  6. Read results: bash autoloop.sh results
  7. If improved → bash autoloop.sh keep "<description>"
  8. If worse/equal → bash autoloop.sh discard "<description>"
  9. If crashed → bash autoloop.sh crash "<description>", diagnose, try fix or move on
  10. GOTO 1 — never stop, never ask permission