smoke-harness

Installation
SKILL.md

Smoke the harness

Unit tests fake the agent CLIs. The smoke evals in tests/*-smoke.eval.yaml run the real ones, so they are the evidence that a harness change works end to end. They call paid models, which is why CI skips them and why you run only the backends your change reaches.

Steps

  1. Plan. From the repo root, with the dev environment active:

    python .agents/skills/smoke-harness/scripts/smoke.py --dry-run
    

    With no arguments the script picks the backends this branch touched (a backend's own harness file reaches that backend; shared caliper/ code reaches all of them) and skips any whose CLI caliper cannot find (its *_CLI_PATH override, install locations, then PATH). Name backends to override: smoke.py codex pi, or pin a model with claude-code:claude-haiku-4-5-20251001. Done when the plan lists a run for every backend your change reaches. A backend skipped for a missing CLI is unverified: say so in your report and in the PR, never as a pass.

  2. Run the same command without --dry-run. Each spec runs once (--k 1) and the script checks its saved results JSON: one attempt per task, every attempt pass, a transcript saved, no hook failures, the requested backend recorded, user customizations off. Done when the script exits. Exit 0 means every check passed; 1 lists each failing check under its spec; 2 means nothing could run.

Installs
16
GitHub Stars
235
First Seen
Oct 3, 2026