eval-integrity
eval-integrity — Benchmark Credibility Audit
User-invokable auditor for LLM evaluation and benchmark repos. It answers one question: if you published this benchmark's numbers, would they survive an adversarial reviewer?
Most benchmark repos have a runner and a leaderboard but lack the integrity scaffolding that makes a published score mean what it appears to mean. This skill checks for that scaffolding, names what is missing, and rates whether each gap invalidates published numbers or is nice-to-have.
The checks below were extracted from hardening a real agent benchmark (cot-bench, a private podcast-evals repo) for an external grant review. Where a check needs a concrete example, cot-bench file names appear as illustrations — they are hints for what good looks like, not paths the auditor expects to find in your repo.
This is the static-analysis cousin of code-review: same parallel-subagent orchestration, but the subagents audit eval methodology instead of code correctness.
Invocation: deliberately model-invocable — read-only audit; it reports and offers fixes, never edits the benchmark.