foundations-information-theory

Installation
SKILL.md

Information Theory Foundations

When to Apply

Apply when a decision depends on a defined distribution:

  • Distribution shift or model-update comparison (KL direction, JS with a stated log base)
  • Feature relevance or leakage screening by mutual information, with an estimator and a null baseline
  • Model comparison across tokenizers (bits-per-byte or bits-per-character, not perplexity)
  • Hallucination or abstention gating via semantic entropy over meaning clusters
  • RL post-training health: policy-entropy collapse in RLVR
  • Context, prompt, KV-cache or agent-handoff compression framed as a measured rate/task-loss tradeoff
  • Error floors (Fano), coding limits (entropy, capacity, rate-distortion), model selection (MDL)
  • LLM-serving questions: speculative-decoding acceptance, quantization or KV-cache compression claims, log loss vs Brier for calibration (references/llm-practice.md)
Installs
17
GitHub Stars
102
First Seen
Aug 12, 2026