finetuning
Installation
SKILL.md
Finetuning
Priors, not rules. Only firm guardrails: held-out eval you never train on, no leakage, trust evo's recorded numbers over the run's self-report. Override anything else against the gate.
Pick the technique by reward shape
Decide on the reward first, technique second. Choosing the comfortable technique over the matching one is the most common failure.
| Reward shape | Technique |
|---|---|
| Verifiable (exact match, unit tests, parser-decidable) | RL (GRPO / RLOO / PPO) — reward includes format, so the model learns to emit verifier-acceptable shape |
| Preference pairs (chosen vs rejected) | DPO / KTO / ORPO — cheaper than full RL, no rollouts |
| Demonstrations only (curated traces, chat data) | SFT — install format/tone/capability the base lacks |
| Have a scorer + want SFT stability | RFT — sample, filter by reward, SFT on survivors |
"SFT-then-RL" is not a law. For a competent base model on a verifiable benchmark, RL-from-base often beats SFT-then-RL end-to-end.