eval-harness-first

Installation
SKILL.md

Eval Harness First

The Phase 0 gate for the whole plugin: finetuning-method-selection and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout.

Input: production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. Output format: the eval/ directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on.

The Gate

Installs
438
Repository
wshobson/agents
GitHub Stars
38.5K
First Seen
Jul 14, 2026
eval-harness-first — wshobson/agents