testsmash

Installation
SKILL.md

Testsmash

Audit what the suite can catch, not what it executes. A test is worth the real bugs it alone catches, minus what it costs to run, read, and keep green. Keep only tests that would catch a real bug; everything else is cost. Agents are rewarded for a green suite and a coverage number, so they write tests that mirror the implementation: they pass today, pass after the feature is deleted, and bury the few tests that matter. Coverage counts execution; only an independent oracle counts as checking.

Tests have a second job: showing behavior to the humans and agents who must understand a change. A well-chosen example, or a readable trace committed beside the code, turns a logic diff into a behavior diff a reviewer can read. Credit that, and scale scrutiny to what a defect in that code would cost.

Audit only. Do not edit, commit, or push. Running the suite and read-only commands is allowed. A mutation probe (break one line, run the tests that claim it) is allowed only in a throwaway worktree or copy, discarded afterwards.

You are the coordinator. Continue until every test has a verdict and the final audit is validated.

The Four Questions

Ask them of every test, at every tier:

  1. Which real bug does this catch? Name it as a bug a competent engineer or agent would plausibly write here: an off-by-one at this boundary, a missed variant of this enum, a reordered write, a dropped error, a wrong unit, a past regression (cite it). An arbitrary mutation nobody would write ("negate the condition", "return early", "invert the empty-list branch") is not a real bug. No real bug means the test catches nothing.
  2. Is it the only test that catches it? Search the suite, at every tier, for another surviving test that fails on the same bug. If one exists, this test's marginal value is zero: duplicate, or merge its distinct inputs into the survivor.
  3. Where does the expected value come from? A spec, a hand-derived value, a reference implementation, or an invariant is an oracle. The code under test, a helper it shares, a stub, or an unread snapshot is a mirror.
  4. What breaks it that is not a defect? A test that fails on a behavior-preserving refactor taxes every change and trains people to re-bless.
Installs
1
GitHub Stars
6
First Seen
6 days ago