Skip to content

Transactional Outbox

Atomic-publish reliability pattern — instantiates Topic-Brokered Event Distribution

Captures an event in the same local transaction as the state change that caused it, so a committed change is never published without its event and an event is never published without its change.

A Transactional Outbox closes the gap between changing state and announcing the change. A service that updates its database and then publishes an event has two separate operations that can fail independently: commit the change, crash, and the event is lost; publish the event, fail to commit, and subscribers hear about a change that never happened. Its defining move is to fold the announcement into the same local transaction as the change — the event is written to an outbox table atomically with the business write, so the two commit together or not at all. A separate relay later reads the outbox and publishes what it finds. The result eliminates the lost-event and phantom-event failures at their source: the database transaction becomes the single point where both facts are made true at once.

Example

A bank's funds-transfer service must debit an account and publish a payment.settled event that notifications, the general ledger, and fraud monitoring all consume. Doing the debit and the publish as two steps invites disaster: crash after the debit but before the publish, and the money has moved while every downstream system believes it hasn't. With an outbox, the service — inside the one transaction that debits the account — also inserts a payment.settled row into an outbox table. The commit makes both durable together. A relay process then polls the outbox, hands each unsent row to the publish path, and marks it published; if the relay dies mid-run, the unsent rows are simply still there to retry. The debit and its announcement are now inseparable — there is no interleaving of failures that can produce one without the other.

How it works

  • Write the event with the change — the event is inserted into an outbox table within the business transaction, so atomic commit covers both.
  • Relay reads and publishes — a poller (or a change-data-capture tail of the database log) reads unsent outbox rows and emits them through the normal publish path.
  • The outbox row is a delivery ledger — each row's state (pending → published) is a per-event record of whether the announcement has actually gone out.
  • At-least-once, by construction — a relay that publishes but crashes before marking a row will re-publish it, so consumers must expect duplicates.

Tuning parameters

  • Relay mechanism — polling the outbox table (simple, adds query load and latency) versus change-data-capture on the DB log (lower latency, more infrastructure).
  • Poll interval and batch size — tighter polling cuts publish latency but presses on the database; larger batches amortize cost at the cost of freshness.
  • Idempotency key — the deduplication token carried on each event so consumers can absorb the inevitable duplicates safely.
  • Published-row retention — how long sent rows are kept before pruning, trading auditability against outbox-table bloat.
  • Ordering scope — whether events are relayed in per-aggregate order or best-effort, which downstream consumers may or may not depend on.

When it helps, and when it misleads

Its strength is decisive: it removes the entire class of dual-write bugs[1] — the lost event and the phantom event — by making the announcement share the change's fate. Its failure mode is the duplicate: at-least-once relay means an event can be published more than once, so a consumer that is not idempotent will double-count, double-charge, or double-notify. A second trap is latency and bloat — a lagging relay delays every downstream reaction, and an un-pruned outbox grows without limit. The classic misuse is treating the committed outbox row as delivered when it is only captured — the event still has to be relayed — or reaching for an outbox where the situation actually demands synchronous, read-your-write consistency. The discipline is to ship an idempotency key on every event, monitor relay lag as a first-class signal, and prune published rows on a schedule.

How it implements the components

  • event_message_contract — it persists the fully-formed, contract-shaped event durably at the moment of the state change, so the announcement is captured exactly and atomically (distinct from enforcing the contract on the wire, which the Publish API or Producer SDK does at emit time).
  • delivery_trace_record — the outbox table is the produce-side ledger of every event owed and whether it has been published, the authoritative record that nothing committed was left un-announced.

It captures and tracks, but does not define or version the schema it stores (schema_compatibility_matrix — that's Schema Registry), emit to the broker itself (its relay hands off to the Publish API or Producer SDK), or route the event once it is published (broker_routing_boundary — Topic Exchange or Event Bus).

Notes

The outbox is a producer-side reliability pattern; it says nothing about whether consumers receive or process the event, which is the job of delivery acknowledgement, durable queues, and dead-letter handling downstream. It guarantees only that everything the producer committed will eventually be published at least once — the strongest promise achievable without a distributed transaction across the database and the broker.

References

[1] The dual-write problem — updating two independent systems (here a database and a message broker) in sequence, with no atomic guarantee across them, so a failure between the two leaves them inconsistent. The transactional outbox is the standard resolution: it replaces the second remote write with a local one inside the same transaction, then relays it asynchronously. ↩