Prompt Evaluation

Disclaimer. Tooling references (promptfoo, OpenAI Evals, Inspect, Braintrust) are illustrative — these tools' own docs are authoritative for their APIs and capabilities. The patterns below transfer regardless of harness.

A prompt without an eval set is an opinion. A prompt with an eval set is a system.

The job of this skill is to set up the smallest harness that catches regressions when you change the prompt, the model, the schema, or the few-shot block — without becoming a maintenance burden.

When you need an eval set

• The prompt runs in production, even at low volume.
• You'll change the prompt at least once after writing it (you will).
• A model upgrade (Sonnet 4.6 → 4.7) might silently change behavior.
• You're picking between two prompts and "it looks better" isn't a defensible answer.

If a prompt is a one-off you'll discard tomorrow, skip the harness and write fewer than five fixtures. For everything else, build the eval set before iterating.

Anatomy of a fixture