Information Entropy¶

The probability-weighted average of logarithmic surprise across the outcomes of a discrete random variable.

Version: v1 · 2026-10-03 · History
Domain-specific #: 13330
Domain group: Formal Sciences
Origin domain: Information Theory
Subdomains: Discrete Information Theory, Probability → Information Theory
Aliases: Shannon Entropy

Core Idea¶

Information entropy assigns one number to the uncertainty of a finite discrete random variable by averaging the surprise of its possible outcomes. If $X$ has outcomes $x$ with probabilities $p(x)$, then, for logarithm base $b>1$,

\[H_b(X)=-\sum_x p(x)\log_b p(x)=\mathbb{E}[-\log_b p(X)],\]

with a zero-probability term defined by continuity as $0\log_b0=0$. The base fixes the unit: base two gives bits, base $e$ natural units, and base ten decimal digits. The measure is zero when one outcome is certain and reaches $\log_b m$ when all $m$ possible outcomes are equally likely. Shannon derived this functional from requirements for a measure of choice and used it to analyze discrete communication sources.^[1]

The identity is the distribution-level expected logarithmic surprise, not a particular communication device, code, ecological community, or realized outcome. One unlikely event can be very surprising while the distribution's average surprise remains modest; conversely, many similarly likely outcomes make the average larger. Shannon's source-coding results give the measure an operational role under a specified lossless source-and-channel model, but the coding result is a consequence with premises, not part of the formula's definition.^[1]

The same functional can be applied to the taxon identity of an organism sampled from a community, with true relative taxon abundances supplying $p$. In that setting it is called the Shannon diversity index. The numerical formula transfers literally, while “diversity” and the sampling model add ecological interpretation; no literal communication channel or code is implied.^[2]

Structural Signature¶

Sig role-phrases: discrete outcome variable → declared probability law → logarithmic outcome surprise → probability-weighted expectation → interpretation boundary.

Discrete outcome variable. Specify the mutually distinguished outcomes and the level of observation. “Next symbol,” “whole message,” and “taxon of a sampled individual” are different random variables and need not have the same entropy. The finite alphabet fixes the present entry's mathematical scope.^[1]^[2]
Declared probability law. Each outcome has a nonnegative weight summing to one. Entropy belongs to that law, not to the outcome labels alone. A finite-sample frequency vector is an estimate of a community's abundance law, not automatically the law itself.^[1]^[2]
Logarithmic outcome surprise. For an outcome with positive probability, $-\log_b p(x)$ decreases as that outcome becomes more likely. The logarithm makes products of independent probabilities additive on this score and the base chooses the unit.^[1]
Probability-weighted expectation. Weight each outcome's surprise by its own probability and sum. This aggregation distinguishes $H_b(X)$ from the self-information of a single realized $x$. It also places live Expected Value as a prerequisite, while the particular logarithmic score remains this entry's residual.^[1]
Interpretation boundary. The same number can support different downstream questions, but a source-coding rate requires source and coding assumptions; an ecological index requires a defined abundance/sampling unit. Marginal entropy, entropy rate, and an estimate from counts are not interchangeable.^[1]^[2]

For fixed alphabet size, equalizing the distribution raises entropy; concentrating all probability on one outcome lowers it to zero. These comparisons follow from the functional, whereas claims about physical heat, ecological value, or a code's attainable length require additional models.^[1]

What It Is Not¶

It is not one outcome's self-information. The quantity $-\log_b p(x)$ scores one possible or realized outcome; $H_b(X)$ averages that score over the entire distribution. Calling a highly surprising observation “high entropy” without stating the law confuses a draw with a property of its source.^[1]

It is not cross-entropy: that evaluates outcomes drawn under one law $p$ using a potentially different model $q$. Here the same $p$ supplies both weights and logarithmic probabilities. Nor is it conditional entropy, which averages uncertainty after a context is known. These neighboring functionals are related but answer different questions.^[1]

It is not automatically the entropy rate of a dependent process. $H(X_t)$ describes one time-indexed symbol's marginal law. Shannon's finite-state source rate averages state-conditional symbol entropies; dependence can reduce uncertainty about the next symbol even when each marginal remains balanced.^[1]

It is not thermodynamic entropy by synonym. Shannon noted a formal resemblance to statistical-mechanical formulas, but a thermodynamic state, macro/microstate specification, and physical units or constraints are not supplied by a bare distribution of message symbols or taxa. Similar mathematical form does not establish identity of physical interpretation.^[1]

Scope of Application¶

For a memoryless discrete source, the outcome variable can be the next emitted symbol and the distribution its symbol probabilities. Then $H(X)$ is the per-symbol source entropy, and Shannon's noiseless-channel theorem connects that rate to achievable faithful coding under its source/channel premises. One should not infer a one-shot code of precisely $H$ bits per symbol from this asymptotic statement; finite codeword restrictions and block length matter.^[1]

For a dependent discrete source, entropy remains defined for each marginal symbol or whole block, but the process rate is a different quantity. Shannon explicitly defines a finite-state source's per-symbol uncertainty by averaging the conditional distributions at its states. This is why retaining the time/history model is necessary when using entropy to reason about compression or transmission rate.^[1]

For ecological alpha diversity, let $X$ be the taxon identity of a randomly selected individual and $p(x)$ its community's true relative abundance. The resulting Shannon index responds to both richness and evenness. Willis and Martin distinguish the population abundance vector from counts observed in a finite survey and study estimation under ecological co-occurrence; a count-based plug-in calculation cannot silently be treated as the exact population index. Their analysis also excludes taxonomic-distance measures, so this index is not itself a phylogenetic diversity score.^[2]

This entry covers finite discrete outcome laws. Continuous differential entropy, quantum entropy, and thermodynamic entropy require different carriers or conventions and are not asserted as simple instances of this exact finite-discrete identity.

Clarity¶

Entropy clears up the phrase “unpredictable” by requiring a random variable, its probability law, and a scale. The same two labels may be nearly certain in one population and evenly balanced in another. Conversely, changing the labels without changing their probabilities leaves the entropy unchanged. The number thus measures distributional uncertainty, not semantic importance or the identity of the outcomes.^[1]

It also clarifies the difference between uncertainty before observing and surprise after observing. A rare event carries high self-information if it occurs, but the entropy averages over its rarity. The distinction prevents a vivid exceptional event from being substituted for the distribution's typical information burden.^[1]

Finally, Shannon's source entropy is a rate only after a source model is specified. When symbols depend on state, the next-symbol law conditioned on that state can be narrower than the marginal law. Stating whether a report gives $H(X_t)$, a conditional entropy, a block entropy, or a process rate resolves apparent numerical disagreements.^[1]

Manages Complexity¶

The distribution may have many outcome probabilities, yet $H$ compresses it to a single comparable scalar. That compression is useful for comparing average uncertainty on one declared alphabet and log base, and it underlies source-coding analysis when its premises are satisfied. In ecology it collapses a potentially long vector of taxon abundances while retaining sensitivity to both how many taxa occur and how evenly individuals are distributed among them.^[1]^[2]

The compression is deliberately lossy. Two distinct distributions may have equal entropy while putting their mass on different outcomes; the scalar does not reveal which event is rare, whether an event is consequential, or whether a sample missed taxa. A responsible use keeps the probability vector and observation model available whenever tail events, taxon identities, or sampling bias matter.^[2]

Separating the functional from interpretive consequences also reduces conceptual clutter. One can compute the same $-\sum p\log p$ in communications and ecology without dragging codewords into a community study or ecological “evenness” into a source theorem. The role mapping tells which parts are shared mathematics and which must be re-proved in the new setting.

Abstract Reasoning¶

First name an outcome variable and law. Then compute each positive-probability outcome's logarithmic surprise and average it under that same law. If all probability concentrates on one outcome, the value is zero; if it spreads uniformly over $m$ outcomes, the value is $\log_b m$. This lets an analyst compare uncertainty while keeping the alphabet and logarithm base fixed.^[1]

Next test what inference is wanted. For two independent random variables the joint entropy adds, whereas dependence makes joint entropy no greater than the sum of the marginals. Knowing a contextual variable can lower the remaining average uncertainty; therefore a marginal $H(X_t)$ is not sufficient evidence for the rate of a dependent source. Shannon states these relationships before giving his finite-state source-rate construction.^[1]

Finally distinguish the modeled law from observations. When $p$ is unknown and estimated from sampled counts, calculate or report the estimator as an estimator, and consider unobserved categories and sampling structure before interpreting differences as population diversity. Willis and Martin's networked-community setting shows why estimating the same Shannon functional can be a separate statistical problem from defining it.^[2]

Knowledge Transfer¶

Within information theory, the functional applies to different finite symbol alphabets without changing its roles: an outcome variable, its law, the logarithmic score, and expectation. It supports analysis of source uncertainty and, with source-model qualifications, lossless coding. Moving from a memoryless symbol to a state-dependent source does not license replacing a state-conditional rate with one marginal entropy.^[1]

The ecology transfer is literal reuse of the functional after species identity is made a discrete random variable. The numerical $H$ can be computed in bits if base two is chosen, even though ecologists often use natural logarithms; changing base rescales the result, not the rank within a fixed comparison. The coding theorem itself does not transfer as an ecological claim. The sampling unit, true abundance law, and estimation method now determine what the number means.^[2]

Beyond such probability models, “entropy” can be an analogy or a different formally defined quantity. The live prime Expected Value supplies the broad probability-weighted aggregation; a hypothetical prime of generic uncertainty might span more substrates, but that does not turn this exact logarithmic probability functional into a substrate-independent prime.

Examples¶

Memoryless two-symbol source¶

Suppose a source emits independent symbols $A$ and $B$ with probabilities $3/4$ and $1/4$. In bits, the next-symbol entropy is

\[H_2(X)=-\frac34\log_2\frac34-\frac14\log_2\frac14\approx0.811.\]

The result is below the one-bit maximum of a balanced two-symbol law: an observer expects less uncertainty because $A$ is more likely. Under the stipulated memoryless source and a suitable lossless block-coding model, this is also the per-symbol source rate relevant to Shannon's noiseless coding result. The numeric pair is a transparent worked example, not a dataset attributed to Shannon.^[1]

Mapped back: discrete outcome variable = next emitted symbol; declared probability law = $(3/4,1/4)$; logarithmic outcome surprise = $-\log_2(3/4)$ for $A$ or $-\log_2(1/4)$ for $B$; probability-weighted expectation = approximately $0.811$ bits; interpretation boundary = coding-rate interpretation depends on the memoryless/lossless source model, not on the isolated formula.

Two-taxon community¶

Suppose the true relative abundances of two taxa are $3/4$ and $1/4$. Let $X$ identify the taxon of an individual selected at random from that community. Its base-two Shannon index is again approximately $0.811$ bits; a second community with the same two taxa at $1/2$ each has one bit. The comparison shows greater evenness with unchanged richness. It does not assert that either community transmits messages, or that the index measures evolutionary distance between the taxa.^[2]

Mapped back: discrete outcome variable = selected individual's taxon; declared probability law = true abundance shares, not raw survey counts; logarithmic outcome surprise = $-\log_2$ of each share; probability-weighted expectation = $0.811$ versus $1$ bit; interpretation boundary = ecological diversity reading requires the community/sampling definition, while finite-count estimates and phylogenetic distance are separate.

Boundary: alternating symbols¶

A binary process that alternates deterministically but starts in either phase with equal probability has balanced one-position marginals: $H_2(X_t)=1$ bit. Once the phase is learned, all later symbols are fixed; its asymptotic entropy rate is zero. This is a constructed boundary example showing why marginal entropy cannot be substituted for process rate without a memoryless premise.^[1]

Mapped back: discrete outcome variable = the symbol at one fixed position; declared probability law = balanced marginal $(1/2,1/2)$; logarithmic outcome surprise = one bit for either symbol at that position; probability-weighted expectation = one bit of marginal entropy; interpretation boundary = dependence across positions defeats the additional memoryless premise needed to equate this marginal with the source's long-run rate.

Structural Tensions¶

T1 — Compact comparison versus outcome-level detail. A single entropy value makes large distributions comparable, but compression loses the identities and probabilities of particular outcomes. Retaining the full vector preserves rare-event or taxon-specific questions at the cost of a less compact summary; keeping only $H$ risks treating equal-entropy laws as interchangeable. Diagnostic: Does the decision depend solely on average surprise, or on which outcomes carry the probability mass?^[1]^[2]

T2 — Portable formula versus setting-specific warrant. Reusing $-\sum p\log p$ permits exact mathematical comparison between a symbol law and a taxon-abundance law, but the convenience tempts a coding or ecological interpretation where its supporting premises are absent. Insisting on every setting's full model protects inference but obscures the reusable functional. Diagnostic: Which role and calculation transfer unchanged, and which claimed consequence needs a new source, sampling, or coding premise?^[1]^[2]

Structural–Framed Character¶

Evaluative weight: The functional is value-neutral; high entropy is not inherently desirable, whether it reflects many plausible symbols or even taxon abundances. Human-practice dependence: The outcome partition and sampling unit are chosen by an analyst, but once those and $p$ are specified, the mathematical value does not depend on human approval. Institutional origin: Shannon's communications problem supplied the historical framing, not an institution whose rules determine the formula.^[1]

Vocabulary travel: Probability, logarithm and expectation travel literally across communication and ecological studies; “bits per symbol” and “alpha diversity” do not travel as interchangeable interpretations. Import versus recognition: The formula may be deliberately imported as an ecological index, but whether a proposed computation is this entropy must be recognized by matching its weighted-log roles, not by using the word “diversity.” Its character: strongly structural as a finite-discrete mathematical functional, yet domain-specific in the encyclopedia because its constitutive probability/logarithm identity and derived guarantees belong to information theory and probability, while applications carry distinct interpretive frames.^[1]^[2]

Structural Core vs. Domain Accent¶

The portable skeleton is probability-weighted aggregation of an outcome score. Live Expected Value supplies that operation and is the proposed composition parent. It does not itself supply the special score $-\log_b p(x)$ or Shannon's distributional uncertainty interpretation; those are the domain-specific difference. A still broader “uncertainty reduction” pattern is at most a future-prime question, not an established parent inferred from verbal similarity.

The accent is therefore not merely historical terminology. A finite discrete outcome law, logarithmic surprise of that same law, and expectation under it are necessary for the named identity. The formula can cross into ecology because species identity is rendered as such a law; that reuse shows breadth of application, not that the exact functional is substrate-free. Thermodynamic state variables, continuous densities, and quantum operators require their own constructions and cannot be pulled under this node by the shared word alone.

This entry presupposes Expected Value.

The structured proposal is a composition/presupposes edge to live Expected Value. Entropy is $\mathbb{E}[-\log_b p(X)]$: without probability-weighted expectation, the quantity would be an outcome's self-information rather than the distribution's entropy. This is an ingredient relation, not a claim that Information Entropy is a taxonomic subtype of every expectation.

Live Conditional Entropy is related but conditions on a known variable or context; it is not a strict upward genus of marginal entropy. Live Entropy (Thermodynamic Sense) has a different physical carrier and interpretation. The live domain-specific Binary Entropy Function is the two-outcome specialization; Cross Entropy introduces a second scoring distribution; Entropy Estimation concerns inference from samples. None replaces the current general discrete-distribution identity.

Relationships to Other Abstractions¶

Current abstraction Information Entropy Domain-specific

Parents (1) — more general patterns this builds on

Information Entropy presupposes Expected Value Prime

Shannon entropy presupposes probability-weighted expectation of outcome-level logarithmic surprise.

Hierarchy paths (3) — routes to 2 parentless roots

Information Entropy → Expected Value → Aggregation → Micro Macro Linkage

Show alternative paths (2)

Neighborhood in Abstraction Space¶

Information Entropy sits in a sparse region of the domain-specific corpus (72^nd percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Foundations of Probability & Inference (29 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-10-08

Not to Be Confused With¶

Self-information: $-\log_b p(x)$ for one outcome, before expectation.^[1]
Entropy rate: average new uncertainty per process symbol under its dependence model, not necessarily the entropy of one marginal symbol.^[1]
Conditional entropy: residual average uncertainty after a context is known; not the unconditional $H(X)$.^[1]
Cross-entropy: scoring one distribution with another model distribution, rather than using the same $p$ for both roles.
Shannon diversity estimate from observed counts: a statistical estimate of a population abundance functional, vulnerable to sampling assumptions and unobserved taxa.^[2]
Thermodynamic entropy: a physical state quantity with separate variables, units and premises; resemblance of logarithmic formulas does not identify it with this entry.^[1]

References¶

[1] Claude E. Shannon, “A Mathematical Theory of Communication”, Bell System Technical Journal 27 (1948), Harvard-hosted reprint, §§6–7, PDF pp.9–12, and §9, PDF pp.15–16. The original states the discrete entropy functional and its properties, distinguishes finite-state source entropy per symbol, and proves the qualified noiseless-channel theorem. registry ↩a ↩b ↩c ↩d ↩e ↩f ↩g ↩h ↩i ↩j ↩k ↩l ↩m ↩n ↩o ↩p ↩q ↩r ↩s ↩t ↩u ↩v ↩w ↩x ↩y ↩z ↩27 ↩28 ↩29 ↩30 ↩31

[2] Amy D. Willis and Bryan D. Martin, “Estimating Diversity in Networked Ecological Communities”, Biostatistics 23, no. 1 (2022): 207–222, §§1 and 2.1.1, especially equation (2.1). The original study defines the Shannon alpha-diversity index on relative taxon abundances and distinguishes the index from its count-data estimates and phylogenetic measures. registry ↩a ↩b ↩c ↩d ↩e ↩f ↩g ↩h ↩i ↩j ↩k ↩l ↩m ↩n