Elements of Information Theory

Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory. Wiley.

Type

Book

Intellectual base

Textbook

Year

2006

DOI

10.1002/047174882x

ISBN

9780471241959

Link

https://doi.org/10.1002/047174882X

Cited from

engineering_design, information_theory, mathematics, neuroscience, physics, statistics_experimental_design, synthesized

Cited by

20 citations across 18 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Channel
  • The two endpoints are a transmitter and a receiver; the medium is an abstract bit-pipe; the codebook is the set of binary input symbols \(\{0, 1\}\); and the noise profile is a single number, the crossover probability \(p\), the chance that a transmitted bit arrives flipped.

    This sourceStandard graduate text; defines the binary symmetric channel, its capacity C = 1 − H(p), and the channel-coding theorem.

• Compression
  • to reduce the number of symbols, bits, or physical resources required to store or transmit it — either losslessly (exact reconstruction possible)

    This sourceStandard graduate text; the data-compression chapters establish that lossless source coding achieves exact reconstruction at rates approaching the entropy. SUPPORTS marker 167 (lossless = exact reconstruction possible). DOI verified.

• Convexity
  • Probability. The set of probability distributions is convex, and Jensen's inequality is the canonical bound from which many results, including information-theoretic lower bounds, follow.

    This sourceThe simplex of probability distributions is convex, and Jensen's inequality is the canonical bound yielding information-theoretic results and lower bounds.

• Garbage In, Garbage Out
  • The formal backbone is the data-processing inequality — for any chain X → Y → Z, no processing of Y can increase its mutual information with X — and GIGO is the practitioner-level claim that holds even in substrates where the formal information measure is not cleanly defined.

    This sourceStates and proves the data-processing inequality: for a Markov chain X → Y → Z, I(X;Z) ≤ I(X;Y), so no processing of Y can increase its information about X, and any apparent cleanup requires a second informative channel.

• Irreducible Floor
  • Statistical learning: Bayes risk — the irreducible classification error given the problem's noise structure; model improvement is intra-regime, and lowering the floor requires changing the feature set, the labelling, or the problem definition.

    This sourceStandard reference (Fano's inequality) for the Bayes error rate as the irreducible lower bound on classification error given the problem's noise structure.

• Linearity
  • Listed in the references but not attached to a specific claim.
• Measure
  • In information theory Shannon entropy is built on a probability measure over outcomes, and mutual information compares two such measures.

    This sourceBuilds Shannon entropy and mutual information on a probability measure over outcomes.

• Measurement Uncertainty and Observational Noise
  • A thermometer with lower noise gives a more precise temperature reading; a quantum measurement device with lower noise still cannot simultaneously measure position and momentum with arbitrary precision, a separation Cover and Thomas (2006) underscore in distinguishing channel-noise-limited capacity from intrinsic source structure.

    This sourceStandard information-theory text: separates channel noise (an apparatus-and-environment property limiting capacity) from intrinsic source entropy (a property of the underlying signal), clarifying that noise is technological while source structure is fundamental.

• Observability
  • For stochastic systems, observability is characterized in information-theoretic terms (Fisher information, mutual information between state and observation history)
• Predictive Coding
  • Listed in the references but not attached to a specific claim.
• Signal Decay and Fadeout
  • Random loss or disappearance offers no such predictability, a distinction Cover and Thomas (2006) sharpen in their information-theoretic treatment of structured (lawful) versus random (entropic) information change.

    This sourceStandard information-theory text: separates channel noise (an apparatus-and-environment property limiting capacity) from intrinsic source entropy (a property of the underlying signal), clarifying that noise is technological while source structure is fundamental.

Domain-specific

• Additive White Gaussian Noise
  • Cover and Thomas give the standard information-theoretic treatment of Gaussian channels and their capacity.
• Binary entropy function
  • … demonstrates that p lies in the unit interval, the log base fixes the information unit, endpoint limits are used, symmetry holds under p versus one-minus-p, and the maximum occurs at one half. to An applied instance preserves the same invariant under a changed notation, scale, implementation, or empirical setting..
• Digital Data
  • Cover and Thomas formalize discrete sources, codes, entropy, and channels.
• Error-Correcting Codes with Feedback
  • Listed in the references but not attached to a specific claim.
• Krichevsky–Trofimov estimator
  • … identity and a convenient observable transfers from For a binary string containing m zeroes and n ones, the next-symbol estimates are (m+1/2)/(m+n+1) and (n+1/2)/(m+n+1). to A context-tree weighting compressor uses a KT probability estimate within every context before mixing predictions across tree depths..
• Shannon–Fano–Elias coding
  • For each ordered symbol, the method forms the cumulative probability below it plus half its probability and takes enough leading binary digits to distinguish its interval.
  • The entry records a descriptive analytical identity; practical use requires the governing domain's evidence, standards, and safety obligations.
• Typical set
  • For an iid or suitably stationary source, the epsilon-typical set contains length-n sequences whose negative log probability per symbol lies within epsilon of entropy.
  • The entry records a descriptive analytical identity; practical use requires the governing domain's evidence, standards, and safety obligations.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:6c2e08fea881 · see in the full table