evals
evals
One exam, one seat. A reviewer miss is the expensive kind, and a diff with a known defect is a
cheap, honest exam of exactly that. Every other seat (brainstorm, plan, execution lanes) is
judged on trial runs: the worklog's Run:/Seats: lines, compared by checkup.
Run it before giving a new model the strict-reviewer seat, and before keeping a change to
workflow/references/phase-4-review.md (same model, old and new instructions). The human
invokes it.
The exam set
One central set for every repo, in the private repo j-skills-evals, cloned at the fixed path
~/.agents/j-skills-evals, so the cases are in ~/.agents/j-skills-evals/strict-reviewer/. The path is fixed
because installed skills are copies, not git checkouts, so no lookup from the skill folder works.
Missing → stop and give the human the clone command:
gh repo clone johnviklund/j-skills-evals ~/.agents/j-skills-evals. The set is private
because cases copy code from private repos; never copy a case into a public repo.