Run a reproducible evaluation

Read the operating agreement, execution requirements, and portable artifacts. For hosted runs read managed guidance.

For bundled local tools and exact commands, read toolkit setup. Run the workspace installer for this skill; it preserves existing customized files. For the managed route, use concrete service operations after confirming the connected schema.

Prepare

Reuse the real application entrypoint and existing runner. Check reviewed cases, criteria, versions, split, fixtures, and intended claim. If those are missing, finish the corresponding discovery/curation/calibration stage first.

Choose case count, repetitions, app/judge settings, concurrency, timeout, and retry policy. Confirm paid scope before calling services; use existing authorized limits. A measured pilot supports duration and spend estimates. Include app, judge, and billed retry usage. No budget change is implicit authorization to continue.

Pilot

Run one authorized case, then inspect the actual saved output, trace, grade, model identity, timing, and available usage. Verify unknown fields are not silently zero. Check that graders consume the actual app result and that requested settings take effect. Test local error classification without spending a full pass.

Execute