Investigating Anomalous Results

Overview

Random tweaks waste time and manufacture false findings. Quietly dropping the inconvenient data point, nudging the cutoff, or re-running until it "works" doesn't fix the problem — it fabricates a result.

Core principle: ALWAYS find the root cause before adjusting anything. An adjustment made before you understand the cause is, at best, noise and, at worst, fraud.

Violating the letter of this process is violating the spirit of it.

The first question for any anomaly: is this a code bug, a data issue, or a real finding? You cannot answer by guessing, and you must not "fix" it until you know — because a real finding is not something to fix.

The Iron Law

NO ADJUSTMENTS WITHOUT ROOT-CAUSE INVESTIGATION FIRST

No dropping data, changing a test, transforming a variable, re-running with new parameters, or "cleaning" until Phase 1 is complete. If you haven't found the cause, you cannot justify the adjustment.