grader-replay
Historical Eval Replay for Grader Changes
Golden calibration (gd dev <guide> --test-grader) only tests three locked solution patches and zero-passrate.patch (which is empty on almost every guide, i.e., the untouched starting app). Historical eval runs contain diverse, imperfect agent implementations that expose broken assumptions, parser edge cases, and false positives.
Because replay applies today's graders against yesterday's saved outputs, before/after mismatches are expected — treat them as signals to triage, not an automatic pass/fail gate.
1. Run Calibration First
Run the repo's golden + zero-passrate calibration on each modified guide:
gd dev <guide> --test-grader
Never run gd dev without --test-grader during validation, or write mode may overwrite manual grader.ts edits.
2. Pull Jobs (if Missing or Stale) & Replay
scripts/replay.ts checks results/suites/ and ~/guidance-evals/ and picks the newest nightly-* job per configured agent using suite_config.json.