Variational Message Passing¶

Compile mean-field variational Bayes updates into local exchanges of moments and natural-parameter contributions on a probabilistic graph, iteratively increasing an evidence lower bound without claiming exact posterior recovery.

Version: v4 · 2026-09-07 · History
Domain-specific #: 3060
Origin domain: statistics
Subdomain: variational inference in probabilistic graphical models
Aliases: VMP

Core Idea¶

Variational Message Passing (VMP) is a deterministic approximate-inference algorithm that turns mean-field variational Bayes updates into reusable local computations on a probabilistic graphical model. Instead of deriving a new global optimization program for every model, VMP lets each node or factor contribute a message determined by its local conditional distribution and the current moments of neighboring approximate distributions. The receiving variable combines those contributions to update the natural parameters of its own variational factor.

Let observed variables be (x), latent variables be \(z=(z_1,\ldots,z_m)\), and the joint model be (p(x,z)). Under the factorization

\[ q(z)=\prod_{i=1}^{m}q_i(z_i), \]

variational inference maximizes the evidence lower bound

\[ \mathcal L(q)=\mathbb E_q[\log p(x,z)]-\mathbb E_q[\log q(z)]. \]

Holding all factors except (q_i) fixed gives the coordinate optimum

\[ \log q_i^*(z_i)=\mathbb E_{q_{-i}}[\log p(x,z)]+\text{constant}. \]

Winn and Bishop's VMP contribution is not this variational identity alone. It is the compilation of that identity into local messages for conjugate-exponential Bayesian networks: parent-to-child messages carry expected sufficient statistics, child-to-parent messages carry contributions to natural parameters, and a node's updated variational distribution is formed from the incoming contributions^[1]. Exact coordinate updates increase or leave unchanged the lower bound and repeat until a stopping rule is met.

The result remains approximate. Convergence of the lower bound means that the chosen coordinate procedure has reached a stationary solution for the restricted variational family; it does not show that the approximate posterior equals the true posterior or that the best global solution was found.

Structural Signature¶

The mandatory roles are:

probabilistic graphical model — a Bayesian network or factorized probabilistic model with local factors and observed and latent variables;
variational factorization — usually a nodewise mean-field approximation \(q(z)=\prod_i q_i(z_i)\), or an explicitly documented extension;
variational objective — the ELBO, equivalently reverse-KL minimization under the ordinary derivation;
local conjugate-exponential structure — exponential-family sufficient statistics and natural parameters whose expectations and updates can be computed locally;
moment messages — current expectations of sufficient statistics sent from a variable or parent toward connected factors or children;
natural-parameter contributions — local factor or child contributions sent back toward a variable or parent;
coordinate update — combination of incoming contributions to replace one (q_i) by its conditional variational optimum;
schedule and initialization — an order and starting state for repeated local updates; and
diagnostic — evaluation of the lower bound, convergence tolerance, iteration limit, and checks on approximation quality outside the objective itself.

The recognition path is:

factorized probabilistic model + declared mean-field family -> derive local moment/natural-parameter messages -> combine incoming messages into one variational-factor update -> iterate coordinate updates -> monitor ELBO and approximation diagnostics

In a conjugate-exponential model, the updated (q_i) remains in the same exponential family as the corresponding conditional. If its natural parameter is λ_i, a factor-graph formulation can express the update schematically as

\[ \lambda_i^{\text{new}}=\sum_{a\in N(i)}\lambda_{a\to i}, \]

where each factor-to-variable message is the expected log-factor contribution under current neighboring beliefs. The exact notation differs between directed-node and factor-graph presentations; locality, variational factorization, and coordinate optimization are invariant.

What It Is Not¶

VMP is not Variational Bayesian Methods as a whole. The broader family includes gradient-based, stochastic, amortized, automatic-differentiation, black-box, and structured approximations that need not compile into the VMP message algebra. VMP is one stable algorithmic realization of mean-field variational Bayes.

It is not generic coordinate-ascent variational inference merely renamed. Any mean-field coordinate optimum has the exponential expected-log-joint form, but VMP additionally exploits graph locality and reusable exponential-family message operators so updates can be assembled from neighboring factors instead of re-derived globally.

It is not belief propagation or the sum-product algorithm. Sum-product messages combine local functions and incoming messages to compute marginals, exactly on appropriate acyclic factor graphs and approximately when used in loopy settings^[2]. VMP messages implement coordinate optimization of a factorized variational posterior and track an ELBO. A tree does not generally make the VMP mean-field approximation exact.

It is not prime:message_passing. That live prime requires autonomous holders of private state, discrete addressed messages as the sole interaction medium, intermediating channels, asynchrony, and no shared memory. VMP “messages” are local algebraic quantities in an inference representation. They need not travel asynchronously or between autonomous agents, and they may be computed sequentially in one process.

It is not expectation propagation, which performs different local projection operations and has different divergence and convergence behavior. It is not Gibbs sampling or MCMC, which represents posterior information through samples rather than a deterministically optimized factorized density. It is not one software package: VIBES and Infer.NET are implementations, not the abstraction.

Scope of Application¶

The classical VMP scope is a directed acyclic Bayesian network whose node conditionals form a conjugate-exponential model. This includes many constructions from discrete, Gaussian, Gamma, Poisson, and related families when every parent-child relationship preserves the needed conjugacy. Winn and Bishop show that the same machinery covers common models such as mixtures, factor analysis, probabilistic principal-component analysis, hidden Markov models, and Kalman-filter-like systems assembled from suitable building blocks^[1].

In probabilistic programming, VMP supports modular inference because the operator for a local factor can be reused wherever that factor appears. A model compiler can inspect factor types, select message operators, generate a schedule, and update distributions without asking the user to hand-derive every variational equation. Infer.NET exposes VMP alongside expectation propagation and Gibbs sampling as a distinct inference algorithm^[3].

The scope has hard boundaries. Plain classical VMP does not supply closed-form messages whenever conjugacy fails or the required expectations cannot be evaluated. Logistic likelihoods, softmax factors, nonlinear deterministic relations, and awkward truncations may require auxiliary variational bounds, quadrature, sampling, gradients, or specialized nonconjugate operators. Winn and Bishop already described auxiliary-bound extensions; Knowles and Minka later formalized Non-Conjugate Variational Message Passing for binary and multinomial regression^[4]. These are extensions, not evidence that classical VMP has no assumptions.

Nodewise mean field also removes posterior dependencies among different (q_i) factors. Structured variational message passing can preserve selected dependencies, but it must declare the new factorization and message semantics. The node is broad enough to recognize documented VMP extensions, yet its classical recognition test remains the baseline.

Clarity¶

A claimed VMP implementation should answer seven questions.

What is the graph factorization? List variables, observations, and local conditional or factor functions.
What is the variational family? State exactly which variables or blocks are independent under (q).
What objective is optimized? Write the ELBO or equivalent KL identity rather than saying only “approximate inference.”
What does each message contain? Identify sufficient-statistic moments, natural-parameter contributions, or the justified extension used.
Why is each local computation tractable? Establish conjugacy or name the bound, quadrature, or approximation that replaces it.
What is the update schedule and initialization? These can affect the attained stationary point.
How is success evaluated? ELBO monotonicity and stabilization test execution, while predictive checks or comparison with stronger inference assess posterior adequacy.

For a factor (f_a(z_a)) neighboring variable (z_i), a common modern statement is

\[ m_{a\to i}(z_i)\propto \exp\left(\mathbb E_{q(z_{a\setminus i})}[\log f_a(z_a)]\right), \quad q_i(z_i)\propto\prod_{a\in N(i)}m_{a\to i}(z_i). \]

This notation makes the difference from sum-product visible: neighboring variables are averaged in log-factor space under the current variational distribution, and factor messages multiply to form the coordinate-optimal (q_i).

Manages Complexity¶

Variational updates for a new graphical model can be lengthy and error-prone when derived monolithically. VMP manages this complexity by using the graph as a computational interface. Each local distribution or factor owns a small operator that converts incoming moments into outgoing natural-parameter contributions. The global algorithm is assembled by composing those operators along existing edges.

The factorization also limits state. Rather than storing or integrating one high-dimensional joint posterior, the algorithm maintains tractable distributions or moments for individual variables or blocks. Conjugacy ensures that a local update changes distribution parameters without changing the distribution family. Natural parameters add, so contributions from multiple children can be accumulated without recreating the entire joint calculation.

The ELBO provides a common diagnostic across the assembled operations. A decreasing ELBO after an allegedly exact coordinate update signals an implementation, scheduling, or arithmetic problem. However, monotonicity is only a necessary internal consistency check. A restrictive mean-field family can converge smoothly to a poor approximation.

Modularity creates a second economy: once Gaussian-likelihood, Gamma-precision, categorical, Dirichlet, and other operators are verified, they can be reused in mixtures, hierarchies, time series, and composite models. This reusable local algebra is the residual that prevents VMP from collapsing into the broad Variational Bayesian Methods node.

Abstract Reasoning¶

The abstraction licenses conditional inferences.

If every updated (q_i) is replaced by its exact mean-field coordinate optimum while the other factors are fixed, the ELBO cannot decrease at that step^[5].
If the ELBO decreases, at least one update is approximate, damped in an unaccounted way, numerically unstable, or implemented incorrectly; the event requires diagnosis rather than being called ordinary VMP convergence.
If all local conditionals are conjugate exponential and the graph compiler has correct operators, model-specific updates can be generated from local types and connectivity.
If a factor breaks conjugacy, classical closed-form VMP is not licensed merely because neighboring values can be called messages. The analyst must supply an additional bound or another expectation/projection method and identify the resulting algorithm.
If the ELBO stabilizes, one may infer coordinate stationarity under the schedule and tolerance, not exact posterior recovery, global optimality, or calibrated uncertainty.
If two schedules or initializations reach different ELBOs, local optima or saddle behavior are plausible; the higher bound does not alone prove better predictive calibration.
If the factorization forbids a strong posterior dependency, no amount of coordinate iteration can recover that dependency. The variational family, rather than execution, is the limiting component.
If the graph is a tree, belief propagation may be exact under its conditions, but VMP can remain approximate because mean-field independence, not graph cycles alone, creates error.

These rules make VMP inspectable as an algorithm rather than a stylistic description of local computation.

Knowledge Transfer¶

VMP transfers literally across probabilistic models when each new model maps to the same roles: graph factors, approximate distributions, sufficient-statistic moments, natural parameters, coordinate updates, and an ELBO. A Gaussian-mixture model, a conjugate state-space model, and a hierarchical discrete model have different semantic variables but can reuse the same inference architecture.

The abstraction also transfers among implementations. A graphical interface such as VIBES, a probabilistic programming compiler such as Infer.NET, or a hand-coded factor graph may schedule the updates differently while implementing the same local variational equations. Parallel and reactive execution are engineering variations only if they preserve the variational fixed-point equations and objective semantics.

Outside Bayesian computation, “message passing” transfers broadly but VMP does not. Distributed services, graph neural networks, and social networks can exchange messages without any variational family, posterior, sufficient statistics, or ELBO. The portable pieces are Optimization, approximation, factorization, and local computation. Their combination with Bayesian graphical-model semantics defines the domain-specific survivor.

Examples¶

Unknown Gaussian mean and precision. Let observations \(x_1,\ldots,x_N\) follow a Gaussian likelihood with unknown mean μ and precision τ, with a Gaussian prior for μ and a Gamma prior for τ. Choose \(q(\mu,\tau)=q_\mu(\mu)q_\tau(\tau)\). The current moments of τ contribute to the natural parameters of the Gaussian \(q_\mu\); the current moments of μ and μ² contribute to the Gamma \(q_\tau\). Alternating these updates changes only family parameters, increases the ELBO under exact updates, and exposes mean-field error by dropping posterior dependence between μ and τ.

Conjugate Gaussian mixture. A model contains categorical assignment variables, mixture weights with a Dirichlet prior, and component parameters with conjugate priors. Local messages update expected component counts, assignment probabilities, and component sufficient statistics. Responsibilities influence parameter moments, and those moments return to influence responsibilities. This is VMP when the cycle is coordinate optimization of a declared factorized posterior, not merely because quantities travel around a graph.

Nonconjugate logistic factor. A Bernoulli observation linked through a logistic sigmoid to a continuous latent variable breaks the classical conjugate-exponential path. Introducing a justified auxiliary variational bound can restore tractable local updates; NCVMP can instead define specialized factor updates. Calling the unchanged logistic likelihood “conjugate” would be a category error.

Belief-propagation nonexample. Sum-product on a tree passes products and marginalizations of local functions and can return exact marginals. It has a graphical message schedule, but no mean-field ELBO coordinate optimization is required. It is belief propagation, not VMP.

Generic variational-inference nonexample. A neural amortized variational model maximizes a stochastic ELBO with reparameterized gradients. Unless its computation is organized as VMP local message operators and coordinate updates, it instantiates Variational Bayesian Methods but not Variational Message Passing.

Structural Tensions¶

Local modularity versus global dependence. Local factor operators make inference reusable, while mean-field factorization deletes dependencies that may be scientifically important. Diagnostic: which posterior correlations are structurally impossible under (q)?

Conjugate tractability versus model expressiveness. Conjugacy gives closed-form messages, but realistic likelihoods may be nonconjugate. Auxiliary bounds widen scope at the cost of another approximation layer.

Monotone objective versus posterior accuracy. ELBO improvement certifies coordinate progress under the chosen family, not calibration against the true posterior. External diagnostics remain necessary.

Scheduling flexibility versus stationary-point dependence. Local updates can be ordered flexibly, but initialization and schedule may change the attained solution. Reproducibility requires recording both.

Automation versus operator correctness. Compilers remove repetitive derivation, yet one incorrect factor operator can contaminate every model using it. Local validation and bound checks become infrastructure obligations.

Message vocabulary versus semantic collision. The word “message” invites comparison with distributed systems and belief propagation. VMP's algebraic quantities have neither the live prime's communication commitments nor sum-product's marginalization rule.

Structural–Framed Character¶

VMP is strongly structural. A candidate can be recognized from its objective, factorization, graph locality, message types, update rule, and convergence diagnostic without relying on a particular institution, vendor, or evaluator. The algorithm can be implemented independently from its historical software.

Its aggregate framedness is approximately 0.16. Modeling choices determine the graph, variational family, initialization, schedule, and acceptable tolerance, but those choices fill explicit roles rather than constitute the identity by convention. The name is scholarly terminology, while execution consequences are mathematical. An exact coordinate update either matches the expected-log-joint optimum or it does not.

The classification remains domain-specific. General local optimization and compositional computation are portable; posterior distributions, exponential-family moments, natural parameters, KL direction, and the ELBO are not optional accents.

Structural Core vs. Domain Accent¶

The structural core is:

factor a global objective -> assign local state summaries -> exchange local contributions -> update one component -> repeat until objective stabilization

That skeleton can describe distributed optimization, constraint propagation, and iterative solvers. Optimization supplies the choice set, objective, and local-stationarity logic.

The domain accent specifies (p(x,z)), factorized (q(z)), reverse KL, the ELBO, expected log factors, exponential-family sufficient statistics, natural parameters, conjugacy, posterior approximation, and probabilistic-graph neighborhoods. Removing these roles yields generic local iterative optimization, not VMP.

This also explains why the accepted Variational Bayesian Methods node does not close the candidate. It owns the broader posterior-to-optimization transformation. VMP adds a recognizable algorithmic compilation discipline: how the coordinate optimum is decomposed into factor-local reusable messages and recombined into distributions. That residual supports a subtype node rather than a mere example paragraph.

Bayesian Updating is the minimal live parent. VMP specializes revision from a joint Bayesian model and observations to an approximate posterior distribution when exact updating is intractable. The approximation and factorization qualifications are essential; VMP does not replace Bayes' rule with unrelated message traffic.

Optimization is a strong related prime because VMP performs coordinate ascent on an ELBO over a restricted distribution family. It is not selected as an additional edge because one minimal parent is required and Bayesian Updating better preserves the inferential type.

Message Passing is lexical and structurally adjacent only in a broad computational sense. Its live identity requires autonomous private-state holders, addressed messages, asynchronous channels, and no shared substrate, so it is not a subsumption parent for VMP.

Factorization, Approximation, Probability, and Statistical Inference supply structural ingredients or a wider environment. The immediate field genus, Variational Bayesian Methods, is accepted in the workspace but is not yet a live endpoint; parent locality should be revisited when it becomes live.

Relationships to Other Abstractions¶

Current abstraction Variational Message Passing Domain-specific

Parents (1) — more general patterns this builds on

Variational Message Passing is a kind of Bayesian Updating Prime

Bayesian Updating is the minimal live parent.

Neighborhood in Abstraction Space¶

Variational Message Passing sits in a sparse region of the domain-specific corpus (86^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (1565 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-09-08

Not to Be Confused With¶

Variational Bayesian Methods / variational inference: the broad distributional-optimization family; VMP is one graph-local algorithmic implementation.
Coordinate-ascent variational inference: the coordinate optimizer underlying classical VMP; not all hand-derived CAVI is organized into reusable graph messages.
Belief propagation / sum-product: local marginalization messages, exact on suitable trees; different objective and approximation logic.
Expectation propagation: local moment-projection method with different divergence and convergence behavior.
Generic message passing: communication among autonomous state holders or any graph-local exchange; insufficient without variational semantics.
Gibbs sampling: stochastic conditional sampling rather than deterministic optimization of (q).
Expectation-maximization: alternates latent expectations and parameter optimization, usually without maintaining a full variational posterior over all unknown parameters.
Infer.NET or VIBES: software systems that implement VMP and other functions, not synonyms for the algorithm.
Non-Conjugate VMP: an extension that supplies message rules where classical conjugate-exponential updates do not directly apply.
Factor graph: a representation of function factorization; it supports many algorithms besides VMP.

References¶

[1] Winn, John M. and Bishop, Christopher M. “Variational Message Passing”. Journal of Machine Learning Research, 2005. The originating paper: section 3 defines the VMP message algebra for conjugate-exponential Bayesian networks - parent-to-child messages are expected natural statistics, child-to-parent messages are natural-parameter contributions, and a node's updated variational factor is their sum. The paper names exactly this class - hidden Markov models, Kalman filters, factor analysers, principal component analysers, and mixtures and hierarchical mixtures of these - as special cases of its conjugate-exponential architecture (section 4 sets out the allowable building blocks); the Gaussian mixture is the case worked through in full. registry ↩a ↩b

[2] Kschischang, Frank R., Frey, Brendan J., and Loeliger, Hans-Andrea. “Factor Graphs and the Sum-Product Algorithm,”. IEEE Transactions on Information Theory, 2001. The paper that states the sum-product rule in this form: a message is the local function times the incoming messages, summarized for the outgoing variable - yielding exact marginals on cycle-free factor graphs and, applied unchanged to graphs with cycles, results that are no longer exact summaries. registry ↩

[3] Microsoft Research and .NET Foundation and Contributors. “Working with different inference algorithms”. Infer.NET User Guide. The Infer.NET user guide lists exactly these three selectable inference algorithms - expectation propagation, variational message passing and Gibbs sampling - and compares them in one table on determinism, exactness, convergence and efficiency (a max-product mode is noted separately as experimental). registry ↩

[4] Knowles, David A. and Minka, Thomas P. “Non-Conjugate Variational Message Passing for Multinomial and Binary Regression”. Advances in Neural Information Processing Systems 24 (NIPS 2011), 2011. Knowles and Minka's paper is the source for the second half of this sentence: NCVMP as the extension of VMP past conjugate exponential families, demonstrated on logistic binary and multinomial regression with a tighter softmax bound. The prior auxiliary-bound work is Winn and Bishop (2005), section 6.1. registry ↩

[5] Blei, David M., Kucukelbir, Alp, and McAuliffe, Jon D. “Variational Inference”. Journal of the American Statistical Association, 2017. Section 2.4 of the review gives this update - each factor set proportional to the exponentiated expected log complete conditional, the others held fixed - and states that coordinate ascent climbs the ELBO monotonically to a local optimum. registry ↩