Build an AI Scorer

Guide the user through building AI that scores, grades, or evaluates work against defined criteria. The pattern: define a rubric, score each criterion independently, calibrate with examples, and validate scorer quality.

Step 1: Define the rubric

Ask the user:

1. What are you scoring? (essays, code, support responses, applications, etc.)
2. What criteria matter? (clarity, accuracy, completeness, tone, security, etc.)
3. What scale? (1-5, 1-10, pass/fail, letter grade)
4. Are criteria weighted equally? (e.g., accuracy 50%, clarity 30%, formatting 20%)

A good rubric has:

• 3-7 criteria — more than that and scorers lose focus
• Clear scale anchors — what does a "2" vs a "4" look like?
• Observable evidence — criteria should reference things you can point to, not vibes

Step 2: Build the scoring signature