Investigating honestly, and writing it up

An investigation that could only have produced the answer you wanted is not evidence. This skill is about designing work that can come out against you, and reporting it so that a skeptical reader can check.

Design

State the question so it has a wrong answer. "Does structure improve retrieval?" cannot fail. "Does structure-aware retrieval raise exception retention above the 83% ceiling that flat retrieval imposes on every model tier?" can.

Pre-register when the result will be used to decide something. Write hypotheses, effect-size thresholds, and the analysis plan before collecting data, and freeze them. After the freeze, a protocol change is a logged deviation and a deviated run cannot satisfy a gate. This converts the weakness of grading your own homework into a commitment — and publishing the protocol before the run is the single most credibility-earning move available to a solo researcher.

Write kill conditions, and mean them. Every investigation should name what result would make it stop. The best outcome this project has produced was a FID cancelled by its own kill condition — the bake-off showed structure, not model tier, was the lever, so the learned router was never built. A document written so it can be cancelled is worth more than one written to be justified.

Isolate one variable through one pipeline. Every arm should run the same path with only the thing under test swapped, so a difference is attributable. If arms differ in prompt and representation, the result means nothing.

Prefer deterministic verification over model judgement. Symbolic checkers — normalised quantity comparison, verbatim span matching, AST checks — remove judge bias, judge cost, and judge variance, and make every number reproducible from recorded responses. The crude-looking check is often exactly right: where the correct answer is near-extractive, identity-up-to-whitespace is fidelity. When the task stops being extractive (code generation), say so and replace the metric rather than stretching it.

Note the class of defect a model judge structurally cannot catch: an answer that omits a required qualifier is still entailed by its sources, so a grounding judge passes it. Detecting omission needs to know which qualifier was required — which is the knowledge the judge lacks.

Record responses content-addressed, so re-scoring is free. Key each call by a hash of its inputs. The paid run happens once; every later analysis is deterministic and costs nothing. This is what makes a result re-checkable rather than a story about a run.