Evaluate Your DSPy Program

Guide the user through measuring AI quality with DSPy's Evaluate class. The pattern: pick a metric, prepare a devset, run the evaluator, interpret results, then feed the same metric into an optimizer.

Step 1 — Gather context

Before recommending a metric or writing evaluation code, clarify:

1. What fields does your program output? (answer, response, custom fields — metric field names must match exactly; SemanticF1 requires response, not answer)
2. Do you have labeled gold answers, or do you need an LM judge to assess open-ended quality?
3. What does "correct" mean for your task? Exact string match, semantic overlap, factual groundedness, safety, or a weighted combination?
4. Is this metric feeding into an optimizer? If yes, you may want trace-aware logic — stricter requirements during optimization than during bare evaluation.

What is dspy.Evaluate

dspy.Evaluate runs your program on every devset example, scores each with a metric, and reports the aggregate score. It handles threading and progress display. Returns an EvaluationResult; access .score for the percentage (0-100) and .results for per-example (example, prediction, score) tuples.

Built-in metrics