wire / wire-format / shannon-entropy
H(X) = -Σ p(x) log2 p(x). Measures average information content in bits. Minimum bits to encode a random variable. Source coding theorem: can compress to H(X) bits per symbol (lossless) or H(X|Y) with side info.
aka: Shannon entropy
spec: Shannon information theory (1948)
H(X) = -Σ p(x) log2 p(x). Measures average information content in bits. Minimum bits to encode a random variable. Source coding theorem: can compress to H(X) bits per symbol (lossless) or H(X|Y) with side info.
| field | size | meaning |
|---|---|---|
| Entropy H(X) | bits | H(X) = -Σ p(x) log2 p(x). Uniform distribution maximizes entropy: H = log2(n) for n symbols. |
| Joint entropy H(X,Y) | bits | H(X,Y) = -Σ p(x,y) log2 p(x,y). H(X,Y) ≤ H(X)+H(Y). |
| Conditional entropy H(X|Y) | bits | H(X|Y) = H(X,Y) - H(Y). Information X provides given Y. |
| Mutual information I(X;Y) | bits | I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X). Symmetric measure of dependence. |
| Channel capacity | bits/use | C = max I(X;Y) over input distributions. Shannon's channel coding theorem: can achieve any rate R < C with arbitrarily low error. |
example:
00 00
Shannon entropy (placeholder)
agent: curl -H 'accept: application/json' wire.phall.io/wire-format/shannon-entropy
or /wire-format/shannon-entropy.json