Skill Evaluator

This skill provides a standardized, empirical evaluation framework for AI agent skills conforming to the open agentskills.io specification.

1. Evaluation Lifecycle Overview

Evaluating an AI skill follows a strict 5-step lifecycle:

1. Validation: Validate evals/evals.json against the agentskills.io schema using just lint-evals.
2. Isolation: Prepare isolated working directories under /tmp/opencode/ to prevent repository context (e.g., AGENTS.md or active skill files) from leaking into baseline runs.
3. Dual-Run Execution (A/B Benchmark): Run each evaluation test case across two subagent execution paths:
   • with_skill: Subagent explicitly reads and enforces SKILL.md and reference documents.
   • without_skill: Subagent receives identical prompt with generic professional guidance only.
4. Honest Assertion Grading: Inspect generated outputs and evaluate each assertion objectively (recording PASS/FAIL and explicit code evidence in grading.json).
5. Aggregation & Reporting: Aggregate metrics into benchmark.json (just benchmark-evals <dir>), render the Markdown uplift report (just report-evals <dir>), and record qualitative feedback in feedback.json.

2. Workspace & Artifact Conventions

All evaluation run artifacts reside under evals-workspace/iteration-<N>/: