Skill Eval Methodology

Use skill-creator for skill mechanics. Use this discipline to make eval claims supportable and eval records auditable.

Reference

Read @references/artifact-examples.md when defining an artifact layout, adapting an existing harness, or auditing completeness. Treat its records as a logical model, not as required filenames or a universal JSON schema.

Read @references/trigger-evals.md before a trigger eval (does the description load the skill when it should, and only then): what run_eval.py really counts, a staged protocol that spends in steps, exact commands, measured pitfalls and per-call costs, and a worked example.

1. Define The Evaluation Contract

• State the user behavior or failure being measured and the claim the eval should support.
• Classify the suite:
  • Capability: measure what the system can do and preserve headroom for improvement.
  • Regression: verify that known behavior still works; a near-perfect result is the intended steady state.
• Identify the system under test: model and version, agent harness or scaffold, eval harness, skill version or fingerprint, tools, prompts, and environment.
• Define task success before running. Keep task, trial, grader, assertion, and suite metrics distinct; assertion pass rate is not automatically task success.
• Require a controlled baseline only for comparative claims such as skill lift. Do not require one for a standalone regression check.