Read the eval-author core skill for the shared standard and boundaries. Direct invocations enter its shared prerequisite flow, consuming an applicable saved result on return. A fresh start uses the core's onboarding renderer for orientation, then ends the turn before repository reads, prerequisite checks, or Ethos work. The core's narrow pure-renderer command exception applies; rendering never supplies readiness. Continue through shared onboarding after the actual reply. Complete and save the core's state-only prerequisite pass before any target Ethos inspection, then reconcile artifacts and include the completed findings in the final results response. A statement that no evals exist settles the starting point; no discovery is needed to prove absence. Reuse confirmed Ethos and prerequisite decisions.

When the user asks to expand a starter suite created here, resume at scope planning for the affected behaviors. Reuse the saved scope in journey state, corpus, Ethos, setup, and accepted cases; recheck prerequisites only where the change affects them. Do not restart onboarding or require an audit merely to add agreed examples. Requested coverage measurement still belongs to eval-author-audit.

No evals is a normal starting state. Do not require a previous run or manufacture a coverage report to enter eval-author-task-create. The initial deliverable is a working suite at the agreed breadth that the user understands and can rerun. A minimal pilot is acceptable when explicitly chosen; comprehensive coverage, repeated agent success, and trace-driven optimization are not prerequisites. Include difficult cases when relevant to the agreed behavior and purpose. A working suite and strong evaluation quality are separate claims.

1. Reuse the confirmed Ethos

Consume the path, confirmed digest, validation result, and unresolved intent from the core's Ethos confirmation handoff after the user has confirmed the document and its current intent in the shared flow. A new shared entry does not skip that review because saved state was previously complete. After the handoff, reopen only changed or unresolved content; reuse the current confirmation without repeating the interview or writing another Ethos. When shared onboarding finishes, show the core's completion pause before source selection or evaluation work. Continue the stages below only after the user replies.

2. Reuse prerequisites and verify when needed

Preserve the selected NeMo Gym or Harbor setup and recorded prerequisite evidence and answers. Shared onboarding requires an available supported runtime before Ethos and case planning. Before scaffolding or another runtime-dependent operation, reuse applicable checks from shared prerequisites or use the shared runtime locator as described in Runtime prerequisite checks. Record actual checks separately from user confirmation and reuse unchanged evidence. If the selected runtime becomes unavailable, preserve completed work and return to the shared prerequisite blocker before continuing the full flow. Other setup needs pause the operations that depend on them.

3. Understand the evaluation starting point

Reuse the user's stated absence of evals or discovery's confirmed starting point, along with available agent and documentation findings. Do not run discovery or ask whether evals exist again solely to complete this milestone. Explain the outcome concretely: a starter eval set of customer scenarios with criteria for scoring responses, which the user can rerun after changes to detect improvement or regression.

Read Corpus and coverage planning before trace search or inspection. Present already-known source candidates and confirm the intended source and corpus, or reuse the user's explicit selection. Support proceeding without traces. Knowing there are no evals does not settle the trace choice. Carry the selected paths, bounds, exclusions, and inspected counts into the plan; repository examples must not become the full corpus by default. Save this selection and current progress in .eval-author/.working/journey-state.json as soon as established, even before the case plan is complete.

Use “starter eval set” for the collection and “eval case” for each scenario. Introduce a case as a request with criteria for scoring the response. Reserve “sanity checks” for validation of the cases themselves; use Harbor's technical term “task” when discussing its files or CLI.

4. Define the evaluation scope

Read the agent entry point, tool definitions, and usage docs to understand how the actual agent runs. Follow Corpus and coverage planning to propose breadth per behavior from Ethos and the selected evidence. Explain a minimal pilot or broader coverage in terms of distinct examples, meaningful variations and difficulty, positive and negative scenarios, and remaining gaps. Include the request, initial fixture, expected observable outcome, a meaningful failure example, and the Ethos requirement each case tests. Ask only for intent or invocation details not already established. Do not make the user supply Harbor YAML or choose a framework.

When agent or setup evidence points to software or state outside the agent process, use Execution dependencies to establish the selected cases' runtime and result-collection requirements. Record the agent's provider fit and dependency inventory using Steps 1 and 2 of the environment sub-flow; files and embedded databases the agent works on count as dependencies. The kit is built and proven at stage 5, before any test. Raise a poor fit for the provider selected at stage 2 in the scope checkpoint; if the user changes provider, verify that runtime as stage 2 describes before preparing cases.

Save the selected journey’s scopes entry in .eval-author/.working/journey-state.json with the Ethos path and requirement references, selected corpus (or no-trace choice), per-behavior scope and example counts, outcomes and grading, priorities, exclusions, task/attempt mapping, cost implications, agent invocation, prerequisites, and unresolved questions. Separate proposed choices from the user's agreement and record subsequent feedback. These are agreed planning decisions, not a coverage report or runnable evals.

Use reproducible inputs and variations supported by Ethos and the agreed scope, including semantically justified difficult negatives where relevant. Grade an observable result rather than a specific tool call or exact prose. Simple assertions are acceptable if their limits are clear. Do not replace subjective quality with brittle string matching or mock away the behavior under test. If the requested case cannot be tested with available resources, retain the original requirement as blocked. Explain the limitation; select a supported alternative only with the user, and record it as a scope change rather than completion of the original case.

If the selected runtime becomes unavailable at this stage, preserve the agreed plan and return to the shared runtime blocker before continuing.

Present the plan using the shared scope review. Settle unresolved scope choices and incorporate changes before preparing cases. When expanding, record retained and added examples and the new unique totals; the pilot's size does not determine the expansion's scope.

5. Prepare cases and grading

For each new case, use private construction declarations to record grading authority, authored or copied inputs, and observable checks. Validate and bind the private plan before recording native controls. Keep simple cases simple; preserve correct inaction and nonbinary rewards where the agreed requirements call for them. A healthy agent failure remains a valid baseline.

After each native scaffold is created, record it as Draft using the shared Task status procedure. Keep that status while completing and validating the task.

Apply Task validation and execution evidence for requirement-to-check review, declared verifier controls, and revision-bound run receipts. Include a realistic incorrect result and applicable valid-alternative and side-effect cases. These are local task controls, not extra model repeats.

Read the reference for each provider in the selected setup for the runtime-specific work in stages 5–9:

Native NeMo Gym first evals: native scaffolding, component configuration, verifier fixtures, and NeMo Gym execution.
Harbor first evals: native task files, environment and agent setup, NOP/Oracle controls, and Harbor execution.

Keep the shared scope, stage check-ins, explanations, suite README, and result reconciliation here. A combined setup uses the relevant parts of both references; reading a reference does not change the selected provider.

Build and prove the kit before the tests. Before writing any case's verifier, complete Steps 3 to 7 of eval-author-environment: choose how each dependency is provided, generate the kit's seeded starting data, make its end state readable, set its isolation, and run its smoke task, recording each job in environment-plan.md. Write each case's own records on top of that data, and derive its expected values with an independent reference query over the generated data rather than by hand. Hand-typed fixtures of a few rows are not starting data.

Create the agreed examples and map their stable IDs to native tasks or dataset rows. Keep source provenance and expected-label rationale in shared scope or verifier-only data, outside the agent's starting environment. Start with one representative case and check its available parts before reusing the pattern. If its live environment or agent connection is pending, continue independent case and grading work for the other selected cases.

Before running, document each case's reward format, metric names and ranges, and expected control results. Derive these criteria from the intended outcome. Add a task README with the Ethos requirement, fixtures, verifier, and run commands, following the shared Task documentation requirements for Harbor tasks. Also create the suite's review entry point following Review and rerun guidance. Present its case inventory and link the guide at this milestone, with input, expected outcome, grading, and available source evidence; mark new cases as not run until executed. Keep it current as the environment, agent connection, and run configuration are completed and fresh control or agent results become available.

Apply the core's Explain the eval pieces as they become relevant guidance to the files being created. For each verifier, explain its actual assertion and a concrete incorrect example it must reject. Fix demonstrated false acceptance before claiming that check is validated; retain untested distinctions as explicit limitations. Introduce the provider's reference solution or known-correct fixture when creating it. Keep this teaching part of building the starter cases, without a separate tutorial or exhaustive methodology exercise.

Before the checkpoint question, introduce Harbor's case viewer in the reply and walk the user to one created case's input and grading files, following Show the user how to review. Include how to launch and open it; a link to the suite guide alone is insufficient.

6. Prepare the execution environment

Hand off the kit proven at stage 5 with Step 8 of eval-author-environment, and use the selected provider reference to confirm each task's environment builds on that kit; rerun the smoke task if the kit changed after it passed. Keep solutions and verifier-only data outside the agent's initial environment. This milestone is complete only when the kit's environment-plan.md, seeded data generator, and smoke results exist under .eval-author/.working/environments/<agent-slug>/; the tasks' own controls never count as environment proof, and without the smoke task's own jobs the plan's status stays unproven. Verify the selected backend and required application access before declaring this milestone complete; identifying their requirements is only partial progress. Record unavailable access and which cases or checks it blocks, naming credential variables without recording secret values.

7. Connect the agent

Use the selected provider reference to establish a supported integration for the actual agent from its entry point, installed interfaces, registry, and CLI help. Do not substitute another agent or rewrite the application. If integration is missing, identify the specific connection required. Verifier-control validation remains available, but performance of the user's agent remains unmeasured.

Record the selected cases, agent and model settings, agreed attempts, reset procedure, and output paths using the provider's configuration format. Propose one attempt for an initial baseline unless the purpose needs repeats; record that choice and its cost during scope planning. Repeats do not add examples. Do not include unrelated drafts merely because they share a parent directory. Resolve paths from the repository root and reference credential environment variables rather than embedding secrets. Configure request delivery, required conversation state, and collection of outputs and actions through supported interfaces. Record connection checks under Validate the evals; configuration alone does not prove that the connection works.

8. Validate the evals

Finalize each task's files and README run instructions before preparing revision-bound evidence for these checks.

Follow Execution recovery for control and target-agent runs, compatibility repairs, and all result recaps. Confirm the current environment kit's proof (Harbor's smoke task or NeMo Gym's persistence and reset cases) passed first; a proof failure is an environment defect to fix, not a task or agent result. That proof is recorded in the kit's plan, not with the task evidence recorder below.

Use the selected provider's native validation and controls, retaining fresh artifacts on reruns. Require checks to complete without exceptions and meet the case's predeclared reward criteria, including reward shape. When inaction is correct, also exercise an explicitly incorrect response or action: a successful no-action control alone does not show that incorrect behavior is rejected.

Run the native commands through the shared evidence recorder, along with the other declared controls. For each case, inspect trial results and rewards. These are basic wiring and verifier sanity checks, not evidence of broad coverage or a robust benchmark. Keep the declared controls proportional to the task; do not require repeated proof or additional model runs as an onboarding gate. Investigate obvious unconditional rewards or leaked answers. Fix broken cases and rerun affected checks; do not weaken the intended assertion to force a pass. Report blocked cases separately and deliver the working subset without silently dropping planned cases or claiming the whole suite passed. Keep unfinished configuration or connection validation visible separately from successful verifier controls.

9. Evaluate the agent

Confirm that the resolved case set matches the selected starter cases. Explain the task count, distinct examples, attempts, execution-cost assumptions, and where rewards and errors will appear. Check that packaging still executes the agreed examples. Once validated, show the cases, agent, model, and provider-native command with the agreed attempt settings, then run with established execution and spend authorization using the selected reference.

Inspect results for every selected case; retain outputs, actions, rewards, and exceptions as fresh run evidence. A completed evaluation can have low scores. Recording this baseline with the evidence recorder is optional.

10. Review results and explain reruns

Confirm the retained controls, configuration validation, actual-agent baseline, and completed result review describe the final task revision, including its README. Then record Accepted through Task status. Link the retained evidence; low agent scores do not prevent acceptance of a functioning evaluation. Keep incomplete tasks Draft. For accepted tasks, follow Keep task paths stable. Acceptance updates status and suite membership without copying or moving task files. Verify the native config resolves exactly the accepted set and report links use those same paths. Keep unfinished tasks visible and exclude them from runnable suite configs. Include the accepted task count and location in the handoff.

Update .eval-author/.working/journey-state.json with artifact references, completion state, and blockers. Keep exact commands in the suite README and control results, per-task agent rewards, and exceptions in native evidence. Show how to rerun the suite, inspect one result, and add or modify a task. Maintain the suite README's run instructions and results links, link the finalized task READMEs, and refresh .eval-author/report.html and .eval-author/README.md following the shared report contract. Preserve the full-suite command and concrete commands for individual coverage items, following Review and rerun guidance.

Report refresh must not change the evaluation result or workflow state. Follow the closing-sentence template in Report generation when linking the report and suite README. Do not link the selected journey’s scopes entry in .eval-author/.working/journey-state.json in the run handoff unless asked.

Continue saving agreed scope and current progress in .eval-author/.working/journey-state.json through the existing workflow, including when HTML generation fails. Refresh the case inventory with links to control and actual-agent results and available execution traces, keeping source evidence and missing traces distinct. Save any needed subset configs and verify their selected cases and retained agent settings. Label rerun commands as task sanity checks or actual agent evaluation.

When an actual-agent attempt exists, explain in the results reply how to open and inspect its saved evidence, using the shared review walkthrough. For Harbor, give the run viewer's launch command and browser URL or a verified live link, and identify the saved job and case. For NeMo Gym, link the readable evidence for the selected dataset rows and identify the saved attempt. State what can be reviewed when the adapter emits no ATIF; captured actions and replies do not imply a native trajectory exists. If only controls ran or execution was blocked before any actual-agent run, say the actual agent has not run and offer case review plus any clearly labeled control evidence.

Reconcile agreed versus delivered examples per behavior using Expand and reconcile. Separate generated, validated, and executed counts and name omissions, changed fixtures, and remaining gaps. Repeated attempts never increase example totals. Render the final three-item checklist, checking every completed step and leaving blocked or partial steps open with their next action.

The working-suite deliverable is reached when the selected starter tasks execute and record rewards through a validated config, even if the agent scores poorly or checks are basic. A functioning case that the agent fails provides a useful baseline; do not weaken it to obtain success.

Name each created eval case and explain the customer behavior it measures. Translate the recorded controls into their meaning. For NeMo Gym, explain the actual positive and negative fixtures and observed rewards. For a Harbor binary action case, doing nothing failed and the prepared reference solution passed, so the grader distinguishes those examples. Do not call NOP an empty answer unless that is what it tested. For graded metrics or cases where inaction is correct, explain the actual acceptance criteria and observed controls. These checks do not show how the user's agent performs or prove that all correct answers receive credit.

State whether the actual agent ran and explain the observed scores when it did. If integration is missing, describe the concrete connection needed at the verified entry point; use “Harbor adapter” only if that helps the user act. If execution is blocked by a credential, name the required environment variable to configure in the execution environment. Describe the next run as scoring actual agent responses, distinguishing it from configuration or connection setup checks.

Explain practical limitations, such as narrow fixtures, a wording assertion rejecting a correct paraphrase, or an assertion checking only part of an outcome. Link the eval cases, results, and established Ethos requirements; identify relevant intent left outside the starter set. Keep detailed scores, interpreter details, and full rerun commands in the saved report unless useful in the explanation; include the viewer instructions in the reply as described above.

Traces and improvement are optional follow-up work. Explain that traces reveal steps and tool calls that can expose failure patterns, missing coverage, and weak checks. Link actual trace artifacts when emitted; otherwise identify the adapter or instrumentation needed to produce them. For requested coverage accounting, hand the established Ethos and actual ATIF to eval-author-audit, then use eval-author-task-create for measured actionable gaps.

Prerequisites

Shared onboarding requires the selected NeMo Gym or Harbor runtime before Ethos. Establish Ethos before case design; no prior traces or coverage reports are required. Scaffolding and validation use that selected installation. Execution needs the chosen backend, agent connection, and any configured provider access. Real-agent and paid-judge runs require authorization for execution and spend.

Limitations

A starter suite establishes a baseline for the selected cases, not comprehensive coverage. NeMo Gym verifier controls or Harbor NOP and Oracle validate task wiring and selected verifier behavior; they do not measure the real agent. If runtime access is lost, retain existing plans and drafts while the flow is blocked.

Troubleshooting

Missing Ethos: resume the shared Ethos confirmation adapter before case design.
Selected runtime unavailable: retain the case plan and identify the NeMo Gym setup requirement or follow discovery's Harbor setup guidance. Install only when authorized, following the user's selected setup.
Oracle fails or NOP unexpectedly passes: inspect task state, reference solution, and verifier against the intended outcome; repair and rerun the controls.
Agent connection or credentials missing: record the verified entry point or required variable name and the blocked run; never request secret values in chat.

eval-author-first-eval

Eval Author: first eval

ATIF as a starting source

Purpose