Home › Communication theory
Entropy and redundancy in human communication
In information theory, information is a measure of surprise. A message that you could have predicted tells you little; one that surprises you tells you a lot. Language is full of predictability, and that is why we can understand speech through noise.
Entropy: a measure of uncertainty
Claude Shannon (1948) defined the entropy of a source as the average amount of surprise per symbol:
H = −Σ p × log2(p) (in bits)
Here p is the probability of each possible symbol. A bit is the amount of information in one fair yes–no question.
| Source | Entropy |
|---|---|
| Fair coin (50% / 50%) | 1 bit |
| Biased coin (90% / 10%) | about 0.47 bits (more predictable) |
| Fair six-sided die | about 2.58 bits |
| 26 letters, all equally likely | about 4.70 bits per letter |
| English letters at their real frequencies (no context) | about 4.1 bits per letter |
Redundancy
Redundancy is the part of a message that is predictable, and therefore not strictly needed. If the maximum possible entropy is Hmax and the actual entropy is H:
Redundancy = 1 − H / Hmax
Shannon (1951) tested how well people could guess the next letter of English text. His results put the entropy of printed English at roughly 0.6 to 1.3 bits per letter once long stretches of context are taken into account, against about 4.7 bits if all letters were equally likely. In other words, most letters in ordinary text can be predicted from what came before. Try it: th_ qu_ck br_wn f_x is easy to complete.
Try it: entropy calculator
Enter the counts (or probabilities) of each category, for example how often four different words occurred in a sample.
Why redundancy helps listeners
- Noise tolerance. When part of the signal is lost, listeners use context to fill the gap, so redundancy acts as error correction.
- Hearing loss. People with hearing loss rely heavily on context, lip-reading and gesture, which is why a quiet room, clear lighting and a visible face help so much.
- Aphasia and language difficulty. Repeating key words, supporting speech with gestures and writing, and using predictable routines add redundancy.
- Speakers adapt. Predictable words tend to be pronounced less carefully, and unpredictable ones more carefully, a pattern described by Lindblom’s hyper- and hypo-articulation (H&H) theory (1990).
Cautions
- Entropy measures statistical unpredictability. It is not the same as meaning or importance.
- Estimates of the entropy of a language depend on the text, the context length and the method used.
- The calculator treats your categories as independent, so it cannot measure predictability from context.
Key idea. Predictability is not wasted effort. It is what makes spoken communication robust in the noisy real world.
Sources and further reading
- Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423 and 623–656.
- Shannon, C. E. (1951). Prediction and entropy of printed English. Bell System Technical Journal, 30(1), 50–64.
- Lindblom, B. (1990). Explaining phonetic variation: A sketch of the H&H theory. In W. J. Hardcastle & A. Marchal (Eds.), Speech Production and Speech Modelling. Kluwer.
- Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley.