Autoresearch Setup

Overview

An autoresearch harness lets an LLM iterate autonomously on an ML problem under a fixed time budget. It is a two-file split:

• harness.py — FIXED. Owns data loading, the train/val split, the fixed fast-iteration subset, and the metric. The LLM never edits it.
• train.py — hackable. Owns the model, optimizer, scheduler, loss, and training loop. The LLM rewrites it freely, subject only to the harness's I/O contract (build_spec → ExperimentSpec, eval_forward, train_fn).

Core invariant: harness.py owns data + the metric and is never edited; train.py owns everything else under the contract. This is what makes results across variants and sessions comparable. Violating the letter of the immutable-harness rule is violating the spirit of comparable, honest results.

Workflow