Measure and Improve Your AI

Guide the user through measuring how well their AI works, then systematically improving it. This is a loop: define "good" -> measure -> improve -> verify.

The Workflow

1. Define what "good" means — write a metric
2. Measure current quality — run an evaluation
3. Improve — choose an optimizer, run it
4. Verify — re-evaluate to confirm improvement
5. Iterate or ship

Step 1: Understand the problem

Ask the user:

1. What does your AI get wrong? (wrong answers, wrong format, inconsistent, too slow?)
2. Do you have labeled examples? (how many? what format?)
3. How do you know when an answer is good? (exact match, partial credit, human judgment?)
4. Have you tried optimization before? (if yes, what and what happened?)