smoke-harness
Smoke the harness
Unit tests fake the agent CLIs. The smoke evals in tests/*-smoke.eval.yaml run the real ones, so they are the evidence that a harness change works end to end. They call paid models, which is why CI skips them and why you run only the backends your change reaches.
Steps
-
Plan. From the repo root, with the dev environment active:
python .agents/skills/smoke-harness/scripts/smoke.py --dry-runWith no arguments the script picks the backends this branch touched (a backend's own harness file reaches that backend; shared
caliper/code reaches all of them) and skips any whose CLI caliper cannot find (its*_CLI_PATHoverride, install locations, thenPATH). Name backends to override:smoke.py codex pi, or pin a model withclaude-code:claude-haiku-4-5-20251001. Done when the plan lists a run for every backend your change reaches. A backend skipped for a missing CLI is unverified: say so in your report and in the PR, never as a pass. -
Run the same command without
--dry-run. Each spec runs once (--k 1) and the script checks its saved results JSON: one attempt per task, every attemptpass, a transcript saved, no hook failures, the requested backend recorded, user customizations off. Done when the script exits. Exit 0 means every check passed; 1 lists each failing check under its spec; 2 means nothing could run.