Build and validate graders

Read the operating agreement and calibration. Use managed grading for Confident AI, DeepEval integration only for that local track, or other graders.

For bundled local tools and exact commands, read toolkit setup. Run the workspace installer for this skill; it preserves existing customized files. For the managed route, use concrete service operations after confirming the connected schema.

Establish the claim

Start from an observed failure or explicit product requirement. Reuse reviewed taxonomy IDs, definitions, and boundary examples. If notes need consolidation, use eval-error-analysis when installed; taxonomy approval does not replace independent calibration labels. Name the check, its unit (output, turn, conversation, trace, or end state), necessary evidence, and clear pass/fail meaning. A vague “quality” score is not enough.

Use code when it can directly test the outcome. Do not substitute keyword matching for a semantic requirement without measuring that proxy against human judgment. For agents, inspect fixture state, tests, files, or expected API effects; use transcript checks for process constraints.

For subjective checks prefer one failure mode per judge and concrete Pass/Fail definitions. Use continuous measures for genuine tradeoffs and blind pairwise comparison for comparative preferences; document thresholds and tie treatment before scoring. Keep cost/latency separate from quality.

Implement within the chosen track

Managed: configure supported hosted criteria, retain definitions, and verify example verdicts. No DeepEval setup is required.