Eval

uv run eval evaluates a model in one or more environments and exits after one epoch per source. It reuses the orchestrator's eval pipeline: one env server per source, a concurrency band, every episode through the monitors, and a trace stream that --resume continues from. Online evals of training runs are the training skill.

Start an eval

The user launches runs; hand over the command unless told otherwise. Always cap smokes (-n, -r).

uv run eval gsm8k -n 32 -r 4                                       # Prime Inference
uv run eval gsm8k -n 32 -r 4 -c 8 --env.agent.harness.id bash      # pin concurrency; set a field of the env block
uv run eval gsm8k -n 32 -r 4 -m Qwen/Qwen3-4B --client.base_url http://localhost:8000/v1   # a local `uv run inference` server
uv run eval @ eval.toml --run.name my-eval                          # multi-source TOML
uv run eval @ eval.toml --run.name my-eval --resume                 # continue an interrupted run
uv run eval @ eval.toml --dry-run                                   # resolve and write the config, exit
uv run eval @ eval.toml --monitors.prime                            # stream each source's epoch to the platform