Systematic Debugging

Diagnose software failures through observable state and controlled checks. Keep a suspected cause, an effective mitigation, and a verified fix distinct.

When to activate

• Crash, timeout, flaky test, exploit instability, silent corruption, race, or environment-specific failure.
• Repeated fixes have not explained or resolved the issue.
• The observed failure is far from the likely source: heap corruption, async timing, protocol state, ABI mismatch, or bad fixture isolation.

Workflow

1. Frame and preserve: record expected versus observed behavior, affected scope, command/input, build, configuration, seed, and exact failure evidence as applicable. In an active incident, prioritize authorized containment or recovery; preserve useful evidence without delaying urgent mitigation. Follow Google SRE's triage guidance.
2. Reproduce or compare: seek the smallest case retaining the same failure signal. If reproduction is unavailable or intermittent, compare failing and successful traces and relevant environmental differences; continue diagnosis with bounded claims. Do not reset state or discard the only failing artifact to get a clean run.
3. Localize: classify the failure and inspect the actual execution path. Trace data, control, ownership, and configuration from the symptom toward the first violated contract; identify the next unknown boundary rather than assuming the crashing line is the cause.
4. Test a prediction: state what the suspected mechanism predicts and choose a check that distinguishes it from plausible alternatives. Instrument that boundary with a log, breakpoint, sanitizer, packet/syscall trace, or watchpoint. Before reading a probe or forced change as evidence, confirm it actually took effect — the log line executed, the rebuilt artifact is the one running, the mutation differs from a pristine copy; a stale build, wrong binary, cached artifact, or edited-but-unused file turns a no-op into a false negative. Iterate tracing and experiments as results arrive; use hypothesis-driven when competing explanations need a structured investigation.
5. Correct the supported cause: make the smallest complete change that repairs the violated contract, including affected consumers. Keep a necessary workaround explicit when the underlying cause cannot yet be corrected. Preserve diagnostic evidence, then remove temporary probes or retain only useful, bounded diagnostics.
6. Verify: add or update a regression test/reproducer where practical, check the original failure and relevant neighboring behavior, then run applicable broader checks. Use testing-reliability when the test itself is suspect and verification-before-completion for completion gates. For intermittent failures, report tested schedules, seeds, or workload and remaining uncertainty; a single successful run does not establish elimination.