foundations-statistical-inference
Statistical Inference Foundations
Turn observations into appropriately bounded claims. A numerical estimate is incomplete without its population, estimand, sampling mechanism, independent unit, uncertainty, and assumptions.
Triggers: "is this A/B test uplift significant or noise", "can we stop the experiment early / we peeked daily", "how many users or eval items do we need", "error bars on eval or benchmark scores", "is model B really better than A", "CI for pass rate graded by an LLM judge", "we tested 20 metrics", sample-ratio mismatch, CUPED, ratio metrics, confidence or credible intervals, calibration, conformal prediction.
Use measurement theory for whether the instrument measures the intended construct, causal inference for identification of intervention effects, and decision theory for choosing actions given uncertainty. This skill owns inference contracts; applied evaluation workflows remain with ai-evals. Do not create causal claims from statistical significance.