Pick a model with evidence, not vibes

A "GPT-killer at a tenth of the price" launches roughly every other week, and the user's realistic options are (a) ignore it and maybe overpay forever, or (b) burn a day benchmarking. There's a third option: a one-afternoon bake-off on a free tier, sized so it provably fits the cap before it starts. GitHub Models (as of 2026-06) exposes 45+ models, frontier ones included, behind an existing GitHub login, capped around 50 requests/day on top models: too tight for production, exactly enough for a careful comparison.

Steps

1. Collect the prompts that actually matter: 3-7 real examples of the user's task, with a known-good answer or a concrete scoring rule each. Not synthetic puzzles; the actual tickets, extractions, or drafts this model will face.
2. Pick 2-4 candidate models. Include the incumbent (the model currently doing the job) so the bake-off can conclude "keep what you have"; that is a valid and cheap outcome.
3. State select_by before running: exact-match rate, win rate on pairwise comparison, latency-under-quality-bar; anything, as long as it's written down first. A criterion chosen after seeing outputs is a rationalization.
4. Check the arithmetic: prompts x models must fit the free daily cap, or the bake-off dies at the rate limit halfway through with a biased partial result. Run the proof below.
5. Run, score, keep the winner; then move the workload to whichever provider hosts the winner properly. The free tier was for deciding, not serving.

Prove it