nurb leaderboard

Two generated surfaces, one editorial layer, two repos. The benchmark (tasks, scorer, submissions, REPORT.md) lives in Shpigford/nurb-benchmarks; the public page (site/benchmarks.html, behind nurb.dev/benchmarks) lives here. Both surfaces regenerate mechanically from that repo's submissions/; the verdict sentences and subscription labels live in its src/nurb_evals/site.py and are written by a person. Submission PRs are pure additions and merge freely; nothing reaches the public page until this skill runs. That gap is deliberate: it is where the sanity check and the verdicts happen, so publishing is an editorial act, not a side effect of merging.

Step 0: Get the benchmarks checkout

Clone or update Shpigford/nurb-benchmarks somewhere outside this repo (for example ../nurb-benchmarks, or a temp directory), on its main. Everything below that touches submissions, verdicts, or REPORT.md happens in that checkout; only the final site/benchmarks.html lands here.

Step 1: Find what is new

The runs added since the page was last regenerated. In this repo, git log -1 --format=%ci -- site/benchmarks.html dates the last publish; in the benchmarks checkout, list the submission directories whose merge commits landed after it (git log --since=<date> --diff-filter=A --name-only -- submissions/ | grep results.jsonl).

Each directory under submissions/ is one run: <harness>-<model>-<effort>-<hex>/ holding results.jsonl plus per-trial gzipped transcripts and part sources. If nothing is new, say so and stop.

Step 2: Sanity-check every new run

Work from the benchmarks checkout (its own uv project; uv sync --locked first if the venv is stale). For each new run directory: