Agentic Evaluation Framework

Category: Engineering Domain: AI Engineering

Overview

Design and run trustworthy evaluations for LLM and agent outputs: pick the right grading method (programmatic check, LLM-as-judge, or human review), write a scoring rubric that judges can apply consistently, rank competing variants by pairwise comparison, and watch for the biases that quietly corrupt judge scores — position bias, verbosity bias, and self-preference. The goal is an eval that you can trust enough to ship on: calibrated against human labels, cheap enough to run on every change, and tracked alongside cost and latency so you never trade quality away by accident. This skill is model- and vendor-agnostic: it reasons about the evaluation method, not any one provider's API, and its scripts aggregate scores you have already collected — they never call a model.

Clarify First

Before designing or running an evaluation, confirm these inputs. If any is unknown or vague, ASK — do not assume:

• What "good" means — the dimensions you care about (accuracy, helpfulness, safety, format, tool-use) and their relative weight (defines the rubric criteria and weights)
• Grading method — can a deterministic check decide it, do you need an LLM judge, or must a human review it? (selects programmatic vs rubric_scorer.py absolute scoring vs pairwise_ranking.py comparison vs human-in-the-loop)
• Ground truth & budget — do you have human-labeled examples to calibrate the judge against, and what cost/latency per eval run is acceptable? (sets calibration plan and the quality/cost/latency budget)

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions.