Information: Entropy

In 1948, Claude Shannon defined information in terms of surprise. The information content of a message is the degree to which it reduces your uncertainty. A message you could have predicted perfectly carries zero information — it tells you nothing you didn't already know. A completely unexpected message carries maximum information. Shannon called this quantity entropy, borrowing the term from thermodynamics: like physical entropy, it measures disorder and unpredictability.

Shannon entropy is defined as H = −∑ p(x) log₂ p(x) across all possible outcomes. The maximum entropy of a source is achieved when all outcomes are equally likely — pure unpredictability. Minimum entropy is achieved when one outcome is certain — pure predictability. Applied practically: a quarterly report that always says roughly the same thing carries low entropy. A dataset where any measurement could be anything carries high entropy. Neither extreme is ideal — maximum entropy is overwhelming, minimum entropy is uninformative.

Norbert Wiener extended this framework through cybernetics to argue that information is what distinguishes organisation from chaos in any self-regulating system. A thermostat carries information about temperature; the information is what allows the system to maintain order. Wiener's key insight: the entropic arrow runs toward decay unless information is actively used to correct it. Systems without good information channels become entropic — they drift.

Andrei Kolmogorov gave entropy a computational interpretation: the algorithmic complexity of a string is the length of the shortest program that can generate it. A truly random sequence cannot be compressed — it has maximum Kolmogorov complexity. A highly ordered sequence can be compressed to a short description — it has low complexity. The two frameworks — Shannon's probabilistic entropy and Kolmogorov's algorithmic complexity — converge: low-entropy sources are compressible; high-entropy sources are not.

The practical application is calibrating attention and weight. When a source has low entropy (high predictability), each new message from it should update you very little. When a source has high entropy (high surprise rate), each new message carries real information and deserves genuine engagement. Most people give equal attention to all messages regardless of their information content — this is the calibration error this skill corrects.

Your Process

Step 1: Identify the Source and the Question Name the source being analysed (a data series, a person, a research domain, a sensor, a communication channel, a market) and the question the source is being asked to answer. Entropy is always relative to a question — the same source may be high-entropy with respect to one question and low-entropy with respect to another.