running-evals
Running Evals — the single in-session entrypoint
Running the full eval picture by hand takes several easy-to-forget commands, and the AI/trajectory lane cannot be a pure CLI: a standalone process has no in-session Agent and cannot spend subscription tokens, so only an in-session driver can produce the transcripts the grader reads. This skill is that driver. Bare /t3:running-evals runs the full default suite.
test vs eval — two sides of one coin
| term | what it is | command | determinism |
|---|---|---|---|
| test | unit / integration code test | t3 teatree run tests |
deterministic, free |
| eval | AI / trajectory eval (agent behaviour) | t3 eval run |
non-deterministic (model) |
The CLI mirror is noun-first: deterministic tests run under the overlay's t3 teatree run tests (a future top-level "t3 test run" form is owned by the separate CLI-simplification audit), AI evals are t3 eval run. There is intentionally no "t3 run evals" group — t3 eval run is canonical. When in doubt: "eval" grades what the agent did on a prompt; "test" asserts what a function returns.
Cost split (never silently meter)
Two buckets under one umbrella. The free deterministic lanes are tests — they assert code/config behaviour with fixed I/O, no live model, every commit. The metered lanes are genuine evals — they judge a live model's behaviour on a cadence. The t3 eval … command surface is shared; the split is about what each lane is.