Information Theory Foundations

When to Apply

Apply information-theory when:

• Compressing prompts, retrieval contexts, logs, or feature sets
• Drift detection — distribution shift from baseline (KL, JS divergence)
• Feature selection by mutual information with target
• Retrieval re-ranking, MMR, or diversity-aware candidate selection
• Prompt-quality diagnosis via output-conditional entropy / Fano bound

Skip and use simpler alternatives when:

• Question is about causation, not information — use foundations-causal-inference
• Single-feature linear correlation is sufficient — Pearson r is cheaper than MI for monotonic continuous data
• Streaming data with hard latency budget — full MI/KL is too slow; use sketches or sampled approximations
• N samples too small for stable entropy estimate (rule of thumb n > 5 × #bins per variable)
• Problem is system-stability or feedback control — use foundations-control-theory
• Bits/nats unit doesn't map to a business decision — risk of treating it as decoration, not signal