foundations-information-theory
Installation
SKILL.md
Information Theory Foundations
When to Apply
Apply when a decision depends on a defined distribution:
- Distribution shift or model-update comparison (KL direction, JS with a stated log base)
- Feature relevance or leakage screening by mutual information, with an estimator and a null baseline
- Model comparison across tokenizers (bits-per-byte or bits-per-character, not perplexity)
- Hallucination or abstention gating via semantic entropy over meaning clusters
- RL post-training health: policy-entropy collapse in RLVR
- Context, prompt, KV-cache or agent-handoff compression framed as a measured rate/task-loss tradeoff
- Error floors (Fano), coding limits (entropy, capacity, rate-distortion), model selection (MDL)
- LLM-serving questions: speculative-decoding acceptance, quantization or KV-cache compression claims, log loss vs Brier for calibration (references/llm-practice.md)