llm-evaluation
LLM Evaluation
LLM development without evals is debugging in the dark. You change a prompt, the output looks different, and you have no principled way to say whether different is better. Eval discipline is what separates LLM features that improve over time from ones that drift, regress, and lose user trust.
The honest version of this skill: most teams know they should have evals and don't. The barrier is not knowledge but starting cost — the first eval suite feels like a lot of work for one team. This skill covers the fastest path to a useful eval system and the patterns that scale once you have one.
What evals are for
Evals serve several purposes that share infrastructure but have different design implications:
- Regression detection. Catch quality drops when you change a prompt, model, or pipeline component.
- Improvement measurement. Quantify whether a change is better, by how much, and on which subsets.
- Capability mapping. Find where your system fails — which query types, which input shapes, which user segments.
- Selection. Choose between models, prompts, or pipeline configurations.
- Monitoring. Track quality drift in production over time.
The same eval set can serve multiple purposes, but the design varies. Regression detection wants stable, representative cases. Capability mapping wants edge cases and known-hard inputs. Monitoring wants production-like distributions, refreshed regularly. Resist the temptation to make one eval set do everything; instead, layer them.