KRT Skill Arbiter

Decide with evidence whether a skill makes the agent do better, and at what cost. A skill earns its place only if the agent does measurably worse without it, or if it carries something the model can't know (the user's conventions, a deterministic script, domain judgment, authority limits). Everything else is context the model pays for on every run.

Read references/safety.md before running evals: they spend money and execute an agent against fixtures. Read references/writing-evals.md when writing or debugging a suite; evals/README.md documents the harness itself, and evals/krt-release-marshal/ is the worked example.

Structural check

Run after any skill is added, renamed, or edited:

python3 skills/krt-skill-arbiter/scripts/check_portfolio.py --repo-root .

It enforces that every skills/krt-* folder has an entry in references/portfolio.json and vice versa; that frontmatter name equals the folder and a description exists; that agents/openai.yaml has display_name equal to the skill ID and a default_prompt starting with Use <skill-id>; and that a skill marked safety_critical has references/safety.md, mentions it in SKILL.md, and has a row in docs/safety.md, while a non-critical skill has neither. It doesn't judge behaviour.

Behavioural evals

A suite lives in evals/<skill>/: cases.json, plus setup.py when the case needs a fixture.