SkillHone Evaluation

Run evaluations and diagnose results to inform improvement decisions.

Evaluator Harness Model

The evaluator is not just a scorer. It is a harness that runs the skill against private tasks and leaves a structured evidence trail:

• Input: a skill checkout plus an eval repo containing datasets, verifier logic, task contract, and optional compiler/audit helpers.
• Execution: each eval item runs in an isolated solver workdir. The solver may create files, call scripts, and write the required artifact.
• Verification: the eval repo verifier scores the produced answer and may call task-local validators, compilers, parsers, renderers, or audit helpers.
• Output JSON: aggregate score plus redacted per-item trace summaries.
• Workdir evidence: trajectory.jsonl, produced artifacts, and stderr-like signals that explain failures the score cannot explain.