Validate one evaluator

Help the user decide what the evidence supports for their intended use. Teach through one concrete comparison at a time. The human defines acceptable behavior; the agent organizes evidence and measures agreement.

Read the relevant reference

• Read method.md before defining labels, choosing splits, tuning, or interpreting a result.
• Read kitaru-contracts.md before discovering operations, opening review, exporting records, running evaluations, or resuming jobs. It defines identity, provider semantics, and the local reporting contract.
• Read visual-guidance.md when presenting a criterion, disagreement, split, or result. Ordinary prose and tables are a complete fallback.

Use scripts/summarize_validation.py for count reports, following the input contract in kitaru-contracts.md. The helper reads local analysis records; it does not obtain labels, contact Kitaru, or run the evaluator.

Start from the user's current evidence

Resolve the criterion, intended use, evaluator/version and selected result name, session population, existing investigation, and any saved analysis or job IDs. Infer these from a supplied handoff where possible. Ask only for a missing decision that changes the next action; do not make the user repeat setup or permission.

Re-read saved review and execution state before creating replacements. Prefer discovered MCP operations and use supported CLI operations where needed. Set KITARU_ACTIVE_SKILL=kitaru-validate-evaluator on Kitaru CLI commands and SDK scripts. Explain the content transfer, remote writes, and paid execution before obtaining any still-missing authorization. Keep credentials and raw traces out of committed files.

Preserve the chosen provider. TypeSafe/Jev is optional; use its released integration when selected and verify the installed contract. Another evaluator can enter this workflow directly. No sibling skill is required.