Skip to content

Log-Sum Inequality

For nonnegative sequences aᵢ and bᵢ with totals A and B, the inequality ∑aᵢ log(aᵢ/bᵢ) ≥ A log(A/B), under standard zero conventions, with equality when the positive ratios aᵢ/bᵢ are constant.

Version
v1 · 2026-09-28 · History
Domain-specific #
10473
Domain group
Formal Sciences
Origin domain
Information Theory
Subdomain
Information Inequalities → Information Theory
Aliases
Log Sum Inequality, Log-Sum Lemma

Core Idea

The log-sum inequality says convex information cost cannot be reduced below the cost computed after aggregating corresponding masses. Local ratio variation adds nonnegative excess beyond the total ratio.

Its proof and equality condition reveal the mechanism: Jensen compares the weighted average of x log x with the function at the weighted average, and equality means all supported local ratios coincide.

Structural Signature

Sig role-phrases:

  • Sequence a — Supplies numerator masses and total A. It is input. Counterfactual: Positive a where b is zero yields an infinite term.
  • Sequence b — Supplies denominator masses, weights, and total B. It is reference. Counterfactual: B must support the normalization used in the proof.
  • Coordinate ratios — Measure local proportionality aᵢ/bᵢ. It is local quantity. Counterfactual: Undefined forms require limits rather than cancellation.
  • Convex function x log x — Creates the aggregation inequality. It is mathematical engine. Counterfactual: Changing the function requires a new convexity condition.
  • Totals A and B — Form the coarsened comparison. It is aggregate. Counterfactual: Aggregation cannot increase the resolved divergence contribution.
  • Equality condition — Requires common ratios across positive support. It is sharpness. Counterfactual: Zeros demand careful statement of which indices count.

What It Is Not

  • It is not the elementary identity log(xy)=log x+log y.
  • Negative inputs are outside the standard statement.
  • Zeros cannot be handled by ordinary division.
  • Equality is not merely A=B unless local ratios also match.
  • Closest near-miss. Gibbs' inequality is a probability-specialized consequence obtained by totals A=B=1; log-sum is the more general nonnormalized statement.

Scope of Application

  • Information theory. Proves divergence and data-processing bounds.
  • Statistics. Analyzes relative entropy and likelihood ratios.
  • Convex analysis. Provides an aggregation form of Jensen.
  • Probability. Derives Gibbs' inequality and related inequalities.

Clarity

State index set, nonnegativity, totals and whether positive, log base, zero and infinity conventions, support condition, finite versus extended-real result, equality condition including zero terms, normalization, partition or aggregation, and the precise consequence being derived.

Manages Complexity

The formula compresses support, normalization, convexity, and equality subtleties. Many incorrect applications are numerically plausible because zero terms or unnormalized measures are silently treated as probabilities.

Abstract Reasoning

  1. Verify aligned nonnegative sequences and support conditions.
  2. Compute totals and separate zero-total edge cases.
  3. Rewrite the sum using b-weights and f(x)=x log x.
  4. Apply Jensen under normalized weights and restore the scale B.
  5. Analyze equality and limiting zero cases before specializing to information quantities.

Knowledge Transfer

The convex aggregation pattern transfers to f-divergences and integral forms when the generating function and measure conditions are rebuilt. The logarithmic formula and equality condition should not be copied to nonconvex functions.

Examples

Canonical

For probability vectors p and q with q positive wherever p is positive, log-sum gives ∑pᵢ log(pᵢ/qᵢ) ≥ log(1)=0, establishing nonnegative KL divergence and its equality condition.

Mapped back: a → p; b → q; totals → 1 and 1; left → KL divergence; right → 0.

Applied / In Practice

Applying the displayed formula to a negative aᵢ makes x log x and the probability-style proof leave their real nonnegative domain, so the theorem does not apply.

Mapped back: input → negative term; domain → violated; verdict → outside theorem.

Structural Tensions

T1 — Fine-Grained Divergence versus Coarse Aggregation. Coordinate resolution reveals heterogeneity while merging categories loses distinguishability and lowers the bound.

Diagnostic: Which partition preserves information relevant to the claim?

T2 — Compact Formula versus Boundary Conventions. The positive-input proof is short while zero support cases determine whether values are finite, zero, or infinite.

Diagnostic: Are limiting conventions and support conditions explicit?

Structural–Framed Character

Log-Sum Inequality is structural as a convex lower bound under aggregation and framed by nonnegative mass ratios.

Structural Core vs. Domain Accent

The broad pattern is convexity penalizing heterogeneity hidden by aggregation. Information theory adds mass support, likelihood ratios, relative entropy, and data processing.

This entry presupposes Convexity.

  • Approved inequality root. No frozen parent entails the exact log-sum statement and equality condition.

  • Related — Jensen's inequality, Gibbs' inequality, Kullback–Leibler divergence, f-divergence, convexity, and data-processing inequality. They are proof engine, consequences, generalization, and interpretation.

Relationships to Other Abstractions

Local relationship map for Log-Sum InequalityParents appear above the current abstraction, mutual partners to the right, and children below. Node labels state whether each abstraction is prime or domain-specific; colors identify relation types.Log-Sum InequalityDOMAINPrime abstraction: Convexity — presupposesConvexityPRIME

Current abstraction Log-Sum Inequality Domain-specific

Parents (1) — more general patterns this builds on

  • Log-Sum Inequality presupposes Convexity Prime

    Log-Sum Inequality presupposes Convexity: the parent's defining role is necessary to the child's frozen mechanism or criterion.

Hierarchy path (1) — routes to 1 parentless root

Neighborhood in Abstraction Space

Log-Sum Inequality sits in a crowded region of the domain-specific corpus (27th percentile for distinctiveness): several abstractions share nearly its structure, so a description that fits it tends to fit its neighbors too.

Family — Matrices, Measures & Numeric Structures (30 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-10-08

Not to Be Confused With

  • Gibbs' inequality. Tell: Is the normalized probability consequence.
  • Jensen's inequality. Tell: Is the general convex theorem used in the proof.
  • Sum–log inequality. Tell: May refer informally to unrelated concavity bounds for log.
  • KL divergence. Tell: Is a quantity whose nonnegativity follows from log-sum.

References

  • Frozen Wikipedia discovery revision: https://en.wikipedia.org/wiki/Log_sum_inequality (revision 1354492474).
  • Preserved source candidate: http://files.ele-math.com/articles/jmi-10-01.pdf
  • Preserved source candidate: http://www.renyi.hu/~csiszar/Publications/Information_Theory_and_Statistics:_A_Tutorial.pdf
  • Preserved source candidate: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture3.pdf
  • Preserved source candidate: http://www.inference.phy.cam.ac.uk/mackay/itila/book.html

The frozen Wikipedia revision is discovery provenance. The retained source set was reviewed for identity, formal or operational relation, and scope. The encyclopedia's structural synthesis is bounded to those claims; a thin authority surface is recorded as a nonblocking source-strengthening repair rather than concealed.