with-testing-principles

A test earns its place only if it can fail when the code is wrong. Most tests that pass review fail this bar silently: they execute the code, raise coverage, and assert nothing that would break if the behaviour regressed. The job is not to produce tests that pass — it is to produce tests that would fail for the right reason. This discipline holds identically whether writing new tests or judging existing ones; auditing is just applying the same pillars to code already on the page.

The pillars below are independent. Each defends against a distinct failure, and no pillar covers for another — a test can satisfy one and still be worthless under the rest. Hold every test against all of them. See references/examples.md for a bad/good pseudocode pair illustrating each pillar.

Falsifiability

A test that cannot fail is not a test. Before trusting any assertion, establish that it bites: that some realistic defect in the code under test would turn it red.

• At write time, prove falsifiability through the test, never by editing the code under test. Either write the test before the implementation exists so red comes for free, or flip the expected value in the assertion, confirm the test fails, then restore it. Flipping a literal in the test is trivially reversible and touches nothing else; mutating source code to prove a point risks leaving the codebase broken and is never necessary here — if it ever happens, restore it before moving on.
• At audit time, the code already exists, so falsifiability is checked by mutation: imagine a plausible bug — an off-by-one, a flipped comparison, a dropped branch, a swapped operand — and ask whether any test would catch it. A mutation that survives every test exposes a gap, not a passing suite. Where a mutation-testing tool is available, this reasoning can be made empirical; where it is not, reason through the mutations by hand.

Independent derivation

Expected values come from the specification, the requirements, or first-principles reasoning about what the code should do — never from observing what the code currently returns. Reading the output off a running implementation and pasting it into the assertion produces a test that passes by construction and blesses whatever the code does, bugs included. Such a test is fully falsifiable — break the code and it goes red — yet it locks in incorrect behaviour, because the expectation was copied from the defect.

This is the trap to watch most closely when tests are written against code that already exists: the path of least resistance is to run it, capture the output, and assert equality. Derive the expectation independently instead. When the independently-derived value disagrees with the code's actual output, that disagreement is a finding — surface it as a possible bug rather than quietly adopting the code's answer.

When the requirement cannot be established from the available context — no specification, no documented contract, and the correct behaviour is not derivable from first principles — stop and ask the user for it. Do not infer the expectation from the code's current output as a fallback; that is the bug-blessing trap wearing a different hat, and a test built on a guessed requirement is worse than no test, because it looks authoritative. State precisely what is undetermined and what is needed to proceed.