leaderboard
nurb leaderboard
Two generated surfaces, one editorial layer. evals/REPORT.md (audit tables) and site/benchmarks.html (the page behind nurb.dev/benchmarks) both regenerate mechanically from evals/submissions/; the verdict sentences and subscription labels live in evals/src/nurb_evals/site.py and are written by a person. Submission PRs are pure additions and merge freely; nothing reaches the public page until this skill runs. That gap is deliberate: it is where the sanity check and the verdicts happen, so publishing is an editorial act, not a side effect of merging.
Step 1: Find what is new
The runs added since the page was last regenerated:
git log -1 --format=%H -- site/benchmarks.html
git diff --stat <that-commit>..HEAD -- evals/submissions/
Each new directory under evals/submissions/ is one run: <harness>-<model>-<effort>-<hex>/ holding results.jsonl plus per-trial transcripts and part sources. If nothing is new, say so and stop.
Step 2: Sanity-check every new run
Work from evals/ (its own uv project; uv sync --locked first if the venv is stale). For each new run directory: