Information Theory Foundations

When to Apply

Apply information-theory when:

• Compressing prompts, retrieval contexts, logs, or feature sets
• Drift detection — distribution shift from baseline (KL, JS divergence)
• Feature selection by mutual information with target
• Retrieval re-ranking, MMR, or diversity-aware candidate selection
• Prompt-quality diagnosis via output-conditional entropy / Fano bound
• Hallucination / abstention gating via semantic entropy over meaning-clustered samples (#1)
• RL post-training diagnostics — policy-entropy collapse is the dominant failure mode in RLVR (#1)
• Agent-to-agent message budgets and KV-cache handoff sizing, framed as a bottleneck/rate problem (#6, #8)