design-ai-product-evaluation

Installation
SKILL.md

Design AI Product Evaluation

Contract

Evaluate the product behavior users experience, not a model in isolation. No single score captures usefulness, safety, reliability, latency, and cost. Automated graders require validation against qualified human judgment.

Workflow

  1. Define the user decision or job, workflow context, AI responsibility, failure consequences, and release decision the evaluation must support.
  2. Decompose quality into task-specific dimensions such as correctness, completeness, grounding, instruction adherence, calibration, consistency, safety, style, latency, and cost.
  3. Build a representative evaluation set from real or carefully governed synthetic tasks, including common, difficult, adversarial, multilingual, accessibility, and high-impact slices.
  4. Define reference answers or rubrics only where they are valid. Preserve tasks with multiple acceptable outputs and specify pairwise, scalar, categorical, or task-success judgments.
  5. Choose human reviewers, deterministic checks, model graders, simulations, and online measures for each dimension. Measure grader agreement, bias, and drift.
  6. Establish baselines, uncertainty, minimum acceptable thresholds, regression budgets, hard safety gates, and tradeoff rules across quality, latency, and cost.
  7. Run blinded comparisons when possible. Investigate failures by slice and mechanism rather than averaging them away.
  8. Connect offline results to staged online behavior, monitoring, incident review, dataset refresh, and versioned release decisions.

BuildBetter Acceleration

Installs
1
GitHub Stars
4
First Seen
Oct 27, 2026