design-ai-product-evaluation
Installation
SKILL.md
Design AI Product Evaluation
Contract
Evaluate the product behavior users experience, not a model in isolation. No single score captures usefulness, safety, reliability, latency, and cost. Automated graders require validation against qualified human judgment.
Workflow
- Define the user decision or job, workflow context, AI responsibility, failure consequences, and release decision the evaluation must support.
- Decompose quality into task-specific dimensions such as correctness, completeness, grounding, instruction adherence, calibration, consistency, safety, style, latency, and cost.
- Build a representative evaluation set from real or carefully governed synthetic tasks, including common, difficult, adversarial, multilingual, accessibility, and high-impact slices.
- Define reference answers or rubrics only where they are valid. Preserve tasks with multiple acceptable outputs and specify pairwise, scalar, categorical, or task-success judgments.
- Choose human reviewers, deterministic checks, model graders, simulations, and online measures for each dimension. Measure grader agreement, bias, and drift.
- Establish baselines, uncertainty, minimum acceptable thresholds, regression budgets, hard safety gates, and tradeoff rules across quality, latency, and cost.
- Run blinded comparisons when possible. Investigate failures by slice and mechanism rather than averaging them away.
- Connect offline results to staged online behavior, monitoring, incident review, dataset refresh, and versioned release decisions.