fail-closed-eval-gate
Fail-closed eval gate
Cost: every claude plugin eval run and every judge grader is a real model call billed to your plan
or API key (ANTHROPIC_API_KEY in CI). Requires Claude Code v2.1.269 or later. The gate script
itself is free, offline, zero-dependency Node 18+.
The gap this closes
claude plugin eval exits 0 when every case's with-plugin score meets --threshold. It also
measures the with-minus-without delta, and the docs say: "The with-minus-without delta is reported
but never changes the exit code". A plugin that scores 1.0 where the bare model also scores 1.0
passes while adding nothing; one that makes answers worse than no plugin passes too, if its
absolute score clears the threshold.
scripts/delta-gate.mjs reads the run's JSON result and fails unless the plugin beat the baseline,
the run finished, and every number it judged is comparable.