Eval Harness

Every prompt tweak in a long-running agent looks like an improvement in the moment. The only way to know is a graded run against fixed inputs. Loopkit already ships .claude/agents/verifier.md — that is your grader. Do not rebuild it.

The three-stage loop

inputs.jsonl  →  runner  →  outputs.jsonl  →  verifier (per row)  →  verdicts.jsonl  →  diff vs baseline

Each stage writes to disk. No stage holds the whole run in context.

Stage 1 — inputs.jsonl

One JSON object per row: {"id": "case-01", "input": "...", "expected": "..."}.

• 20-100 cases is enough for a signal. More is nice, not required.
• Include known-hard cases, edge cases, and a couple of trivial ones as sanity anchors.
• Freeze the file. Rev the eval with a suffix (inputs-v2.jsonl) when you change it. Never edit in place — you lose the baseline.