flaky-test-diagnosis
Flaky Test Diagnosis
Use this skill to turn a non-deterministic test into a deterministic one by finding why it sometimes fails, not by retrying it until it passes. The recurring failure mode it prevents is "fix" by sleep increase, retry annotation, or disabling — all of which hide the defect (which is sometimes in the code, not the test) and erode trust in the suite.
The governing rule: a flaky test is a bug report — either the test races/leaks, or the code does. Find which, then remove the non-determinism at its source.
WORKFLOW SKILL. INVOKES: build/run tooling to reproduce (repeat runs, shuffled order, parallel, sanitizers), read-only inspection of the test and code under test. Apply the deterministic-wait patterns of the project's test framework to express the fix, and hand a confirmed data race or lifetime bug to a concurrency/object-lifetime review rather than masking it. FOR SINGLE OPERATIONS: classify one observed flake against the source table and name the fix.
Scope
- Use this skill for: reproducing and root-causing an intermittently failing test and making it deterministic, or escalating to a real code defect when the test correctly caught a race/leak.
- Out of scope: authoring a new test from scratch (a test-framework authoring concern), judging whether a consistently-passing test is a good test (a separate test-quality-judgement concern), and fixing the underlying concurrency/lifetime bug itself once identified (a concurrency / object-lifetime review concern). Name the boundary when a finding depends on it; route the work accordingly.
First: reproduce deterministically
Do not trust a single green run. Increase the failure rate until you can observe it: