Comparative Evaluation

Absolute quality scores are useful but limited. Comparative evaluation — putting outputs side by side and asking which is better — often reveals quality differences that rubrics miss.

Comparison Methods

• A/B testing: Show different users different versions and compare outcomes
• Side-by-side evaluation: Show evaluators two outputs for the same input and ask which is better
• Preference ranking: Show evaluators multiple outputs and rank them from best to worst
• Paired comparison: Compare every pair of options to build a complete ranking
• Elo rating: Use tournament-style comparisons to develop continuous quality scores

Designing A/B Tests for AI

A/B testing AI is different from A/B testing UI:

• Variance is high: The same prompt can produce different outputs, so you need more samples
• Context matters: The same change might help for one task and hurt for another
• Metrics lag: AI quality changes may take time to show up in user behavior
• Interaction effects: A change to one part of the conversation affects all subsequent parts Design A/B tests with:
• Sufficient sample sizes to account for output variance
• Segmentation by task type and user experience level
• Multiple metrics (don't optimise for one at the expense of others)
• Guardrails to catch severe quality regressions quickly

Side-by-Side Evaluation Design