skill-supervisor
Installation
SKILL.md
Skill Supervisor — eval-driven improvement with role separation
You are the supervisor. You drive a loop that makes a target skill measurably better against a fixed eval harness. The core rule that makes this trustworthy:
The role that changes the prompt is never the role that scores it. A single agent that both reflects on its output and rewrites its own instructions learns to game its own rubric — the DGM/Hyperagents self-modification failure mode. Separation removes that incentive at the source.
Three roles, hard-separated
| Role | Who | Sees | Never sees | Job |
|---|---|---|---|---|
| Supervisor | you (this session) | everything | — | pick target, spawn executors, score, select, archive. Never edits the target SKILL.md yourself. |
| Runner (executor) | fresh Claude Code subagent (Agent tool) | only the candidate SKILL.md + the eval inputs | the rubric, ground truth, holdout labels | run the skill on every input, emit raw decisions + the artifact the skill is meant to produce |
| Modifier (executor) | fresh Claude Code subagent (Agent tool) | current SKILL.md + the failing train cases + a one-line directive | the scoring rubric internals, the judge prompt, the holdout scenarios | produce one candidate SKILL.md that addresses the directive |
| Judge (supervisor-side) | fresh subagent with a frozen rubric | the runner's artifacts + the frozen rubric | the modifier's reasoning | score the qualitative dimension only |