Design Resumable Model Evaluation

Make interruption cheap and evidence comparable.

Freeze an evaluation revision

Create a unique revision for every combination of corpus membership, model, prompt/instructions, schema, decoder settings, validator, run count, and thresholds. Hash this configuration into the state file. Never seed a new revision from incompatible results.

Write the case manifest before inference. Select cases from stable metadata, not previous outcomes.

Persist atomically

Use an append-only result stream or atomically replace a small state file after every case/run. Store: