Evaluator Plugin

The Plugin CLI entrypoint is uv run nemo evaluator.

Purpose

Use this skill to choose an evaluation interface and metric, validate a minimal example, submit a NeMo Platform evaluation job, and retrieve its results.

Inputs

Establish these inputs before building an evaluation:

• Evaluation interface: dataset-driven vs. task-driven agentic evaluation
• Execution interface: standalone SDK evaluation or a durable NeMo Platform job.
• Pass/fail dataset examples: the smallest representative pass and failure cases.
• Metrics: the behaviors to score and the template fields they consume.
• Target: no target for offline scoring, or the model, agent, runner, or precomputed trials that produce outputs.