Skill Battle

Default behavior

State one narrow claim the target skill should improve.
Choose representative tasks that measure that claim, not generic model quality.
Run identical tasks with and without the skill. Use multiple trials for stochastic tasks.
Prefer deterministic graders; use blind LLM judging only where necessary.
Track regressions, cost/time, token usage, and user burden alongside quality.
Keep full prompts, outputs, traces, and local artifacts local by default.
Use a train/held-out split if the skill or rubric is tuned during the experiment.
Never publish demo/synthetic numbers as benchmark results.
For multi-turn skills, simulate the same hidden user state in both conditions rather than flattening the workflow into one prompt.
Use the repository's sibling skill-eval-loop when available. It already enforces blind paired runs, skill-load checks, deterministic verification, and strict-majority adoption.

Run locally first. Do not imply a cloud submission happened unless the configured Edge endpoint returned a canonical result identifier.