Variational Message Passing

Compile mean-field variational Bayes updates into local exchanges of moments and natural-parameter contributions on a probabilistic graph, iteratively increasing an evidence lower bound without claiming exact posterior recovery.

Version

v4 · 2026-09-07 · History

Domain-specific #

3060

Origin domain

statistics

Subdomain

variational inference in probabilistic graphical models

Aliases

VMP

Core Idea

Variational Message Passing (VMP) is a deterministic approximate-inference algorithm that turns mean-field variational Bayes updates into reusable local computations on a probabilistic graphical model. Instead of deriving a new global optimization program for every model, VMP lets each node or factor contribute a message determined by its local conditional distribution and the current moments of neighboring approximate distributions. The receiving variable combines those contributions to update the natural parameters of its own variational factor.

Let observed variables be (x), latent variables be \(z=(z_1,\ldots,z_m)\), and the joint model be (p(x,z)). Under the factorization.

Scope of Application

The classical VMP scope is a directed acyclic Bayesian network whose node conditionals form a conjugate-exponential model. This includes many constructions from discrete, Gaussian, Gamma, Poisson, and related families when every parent-child relationship preserves the needed conjugacy. Winn and Bishop show that the same machinery covers common models such as mixtures, factor analysis, probabilistic principal-component analysis, hidden Markov models, and Kalman-filter-like systems assembled from suitable building blocks.

Clarity

A claimed VMP implementation should answer seven questions.

1. What is the graph factorization? List variables, observations, and local conditional or factor functions. 2. What is the variational family? State exactly which variables or blocks are independent under (q). 3. What objective is optimized? Write the ELBO or equivalent KL identity rather than saying only “approximate inference.”
2. What does each message contain? Identify sufficient-statistic moments, natural-parameter contributions, or the justified extension used.

Manages Complexity

Variational updates for a new graphical model can be lengthy and error-prone when derived monolithically. VMP manages this complexity by using the graph as a computational interface. Each local distribution or factor owns a small operator that converts incoming moments into outgoing natural-parameter contributions. The global algorithm is assembled by composing those operators along existing edges.

Abstract Reasoning

The abstraction licenses conditional inferences.

• If every updated (q_i) is replaced by its exact mean-field coordinate optimum while the other factors are fixed, the ELBO cannot decrease at that step.
• If the ELBO decreases, at least one update is approximate, damped in an unaccounted way, numerically unstable, or implemented incorrectly; the event requires diagnosis rather than being called ordinary VMP convergence.
• If all local conditionals are conjugate exponential and the graph compiler has correct operators, model-specific updates can be generated from local types and connectivity.
• If a factor breaks conjugacy, classical closed-form VMP is not licensed merely because neighboring values can be called messages.

Knowledge Transfer

VMP transfers literally across probabilistic models when each new model maps to the same roles: graph factors, approximate distributions, sufficient-statistic moments, natural parameters, coordinate updates, and an ELBO. A Gaussian-mixture model, a conjugate state-space model, and a hierarchical discrete model have different semantic variables but can reuse the same inference architecture.

The abstraction also transfers among implementations. A graphical interface such as VIBES, a probabilistic programming compiler such as Infer.NET, or a hand-coded factor graph may schedule the updates differently while implementing the same local variational equations.

Relationships to Other Abstractions

Current abstraction Variational Message Passing Domain-specific

Parents (1) — more general patterns this builds on

• Variational Message Passing is a kind of Bayesian Updating Prime

  Bayesian Updating is the minimal live parent.

Hierarchy paths (5) — routes to 3 parentless roots

• Variational Message Passing → Bayesian Updating → Inductive Reasoning

Show alternative paths (4)

Neighborhood in Abstraction Space

Variational Message Passing sits in a sparse region of the domain-specific corpus (86^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (1565 abstractions)

Nearest neighbors

• Variational Bayesian Methods — 0.84
• Ruzsa–Szemerédi Problem — 0.82
• Method of Moments — 0.80
• Probabilistic Graphical Model — 0.79
• Focused Information Criterion — 0.79

Computed from structural-signature embeddings · 2026-09-08