Evaluation Harness

Pain Signals — You Need This Pattern When:

• "Does this prompt work?" is answered by trying a few examples manually
• A prompt change ships with no way to know if quality improved or regressed
• Production quality is unknown — you learn about failures from user complaints
• The same prompt produces different quality across different input types
• There is no shared definition of "correct" or "good enough" for the AI output
• You cannot compare two prompt versions or model versions quantitatively

Core Principle

LLMs are non-deterministic. You cannot assert exact outputs. Instead, you assert statistical properties over a representative dataset. The evaluation harness is TDD for AI: define what "good" looks like first, then iterate until you get there.