model-gap
Model gap
A skill tested on one model (the reference) often passes fewer tasks on another (the target). Find where and why, fix it with text every model reads, and prove the target gains while no model drops.
This runs skill-diet's eval: tasks and scoring per eval.md, the adapter per adapter.md, commands as node ../skill-diet/scripts/eval.mjs …. Edits follow skill-diet's Editing a skill.
Set up
- One adapter, one host for every model, so only the model differs. Pass
--modelto everyrunandscore. - Tasks per eval.md, each labeled with the case it covers. Split them blind and at random: about a third, at least one per case, go to a held-back folder whose
adapterandsk/link to the tuning folder's. Held-back tasks get no runs until the end. - Versions:
base(the skill as it is),none, and one folder per candidate fix. - Models: the reference has the best
basepass rate (on a tie, the one that reads the skill most); every other model with a gap is a target. Add at least one more model as a harm check. - Paid quota: before each step, give the user sessions × cost per run and stay within the budget they set. Two tasks first measure the cost.
1. Measure the gap
base and none on every model, tuning tasks, 3 runs. score <dir> --models <all> --versions none,base puts the models side by side; --model <m> instead gives one model's warnings. A gap is a task the target fails more often than the reference with a cause in its logs; 2/3 against 3/3 alone is noise. If the target does better with none (MISS or WORSE here), look for lines to cut first. No gap on any target → stop and report.