skill-evals
A skill makes two empirical claims, and the author's intuition cannot settle either. The description claims I am the right tool for this class of situation — an activation claim. The body claims loading me makes the work better — an impact claim. An eval measures whether each claim holds against evidence the author has not seen yet. The discipline is to commit to what a passing result looks like before running anything, then run an agent under controlled conditions and report the gap between claim and reality.
The eval is driven by an orchestrator that never performs the work it is measuring. It delegates every generation, every eval run, and every LLM judgement to a separate subagent with isolated context — because an agent that has already seen the skill, the labels, and the result the author is hoping for cannot produce an uncontaminated scenario, run, or verdict. Isolation is the mechanism that keeps each step honest. And the final word on whether the agent did a good job stays with the human author: the machinery narrows the question down to the cases worth looking at, it does not answer it.
Both eval modes ride one spine. Only what is held fixed and what is judged differ between them.
The spine
Every eval, of either mode, is the same four moves. Skip one and the result stops meaning anything.
- Commit ground truth first. Have an isolated subagent generate the candidate scenarios — corpus prompts (activation) or tasks (impact) — so they are shaped by the skill's purpose rather than by its exact wording or the result the author wants. Then the orchestrator commits their labels (activation) or success criteria (impact) — with the author approving them before any eval run — derived from the skill's intent, never from outputs already seen. A criterion authored after the output exists describes the output, not the goal, and the eval can no longer fail. Committing is literal: the approved suite is written to disk as a hash-keyed version (
v1,v2…) and frozen. A re-run reuses the existing frozen suite — that reuse is what makes the eval repeatable — and mints the next version only when the suite's content genuinely changes. Seeing the outputs can legitimately reveal that a label was wrong or a criterion ill-formed; that is a real change to the test set, so encode it by minting a new version, never by loosening the standard in place to fit what was just observed. The first is learning; the second is drift wearing its costume. Regenerating the suite from scratch on every run is the bug, not the feature: it swaps the test set out from under the trend, so no two runs are comparable. - Run each scenario in its own isolated subagent. Hold everything fixed except the one variable under test — the prompt (activation) or the presence of the skill (impact). One subagent per run, its context limited to that scenario and carrying nothing about the eval's purpose, the other runs, or the outcome anyone expects. Shared or carried-over context bleeds the answer from one run into the next; isolation is what makes "controlled" more than a label.
- Judge blind as a first pass, then validate with the author. A judge subagent scores each result against the committed standard without being told which run the author hopes will win — blindness stops it from scoring the hope instead of the work. But an LLM judge shares the blind spots of the agents it judges and is swayed by the same plausible-but-wrong output, so its verdict is triage, not truth: it narrows the set the author has to inspect. The author makes the final call on whether the agent did a good job. Where scoring is mechanical — comparing a fire/no-fire decision against a label — no judge subagent is needed, and the author instead reviews the misfires that scoring surfaces.
- Append the run; read the gap from the history. A single number ("80%") tells the author nothing they can act on, and a single run tells them nothing about whether the skill is moving. Each run appends a record — precision and recall, or the judge's per-task verdicts — to a persisted log, and a frozen HTML report (copied into the output directory once, never regenerated) reads the whole history back. It plots each metric's trend segmented by suite version — so scores measured against different test sets never join into one misleading line — and surfaces the latest run's specific misfires: this prompt that should have fired and didn't, this part of the output the skill failed to improve. For the activation eval, it shows precision and recall alongside the two named lists: the should-fire prompts it missed and the should-stay-silent prompts it fired on. For the impact eval, it pairs each task with the judge's first-pass verdict and the committed criteria each delta served or missed, marks any run the author has not yet validated, and leaves the author's final call as the last step. Because the report fetches the run and suite files, view it over a local server (see Persisted state and run history), not by opening the file directly. The history is the deliverable; the grade is not.
The point of the eval is to find where the skill falls short, not to confirm it works. An eval that cannot produce a failing result is theatre. Treat a clean pass with suspicion and check the corpus and criteria were demanding enough to bite.