Skip to content

Log Compaction

Retention procedure — instantiates Event-Log-Centered Modeling

Reclaims space by keeping only the latest or still-necessary record per key and discarding superseded history, under a retention policy that must never break the ability to rebuild state.

An append-only log grows forever, which is both its virtue and its bill. Log Compaction is the maintenance process that keeps the bill payable without betraying the virtue. It walks the log and, for each key, retains only the record that still matters — usually the latest value, sometimes a semantically necessary subset — while discarding entries that later events have made redundant. Its defining constraint is subtle: compaction must shrink the log without changing what you can rebuild from it. Removing intermediate updates to a key that only cares about its final value is safe; removing history that some projection or audit still needs is data loss dressed as housekeeping. So compaction is really a governed discard — half storage engineering, half retention policy — deciding not just what is redundant but what may lawfully be dropped, kept, or redacted, and for how long.

Example

An IoT platform ingests device-state events: each thermostat reports its setpoint every few seconds, producing millions of SetpointReported events per device per month. Ninety-nine percent of them are superseded the instant the next report lands. A compaction process runs over the device-state topic keyed by device_id: for each device it keeps the latest reported setpoint and drops the redundant history, collapsing a month of chatter into one current record per device. A new consumer that only needs "what is each device's setpoint now" can bootstrap from the compacted log in seconds instead of replaying millions of stale updates.

The retention side is where judgment lives. The same platform is legally required to purge a device's data on account deletion and to keep thermostat-fault events for two years for safety recalls. So compaction is not a blind "keep latest" — its policy distinguishes state topics (compact aggressively to latest-per-key) from event topics that must be retained in full, and it honors redaction requests by removing personal data even from history that is otherwise kept. The output is a log that is smaller and still fit for every rebuild the platform legitimately needs — and no rebuild it must not permit.

How it works

Compaction is distinguished from ordinary deletion by what it is allowed to forget:

  • Key the log. Records are grouped by the key whose value they update (device, account, aggregate), so redundancy can be judged per key.
  • Keep the necessary, drop the superseded. For each key, retain the latest or semantically-required record and discard entries no downstream rebuild depends on.
  • Bind it to a retention and access policy. What may be dropped, what must be kept, what must be redacted, and for how long — driven by law and by which projections still read the history.
  • Preserve rebuildability. The invariant throughout: any state or projection the system is entitled to reconstruct must still be reconstructable from the compacted log.

Tuning parameters

  • Compaction key — what defines "same record." A coarse key collapses aggressively and saves the most space but can merge things that should stay distinct; a fine key preserves detail at higher volume.
  • Retention horizon — how long superseded history is kept before compaction removes it. Longer horizons preserve auditability and late rebuilds; shorter ones save storage and shrink privacy exposure.
  • Compaction trigger — by size, by age, or on a schedule. Aggressive triggers keep the log lean but spend I/O and can compact data still in use; lazy triggers let the log bloat.
  • Redaction handling — whether personal data is purged in place or tombstoned. In-place purge satisfies erasure duties cleanly but rewrites history; tombstoning preserves structure at the cost of retaining a marker.
  • Topic classification — which streams are compactible state versus must-retain events. Misclassifying an event topic as state is how compaction quietly destroys audit history.

When it helps, and when it misleads

Its strength is that it lets an event-sourced system keep the append-only promise affordably: fast bootstrap for state consumers, bounded storage, and a lawful way to honor deletion and redaction without abandoning the log model. Real log systems ship this as a first-class feature.[1]

It misleads when "redundant" is judged too generously. History that looks superseded to one consumer may be exactly what an auditor, a bitemporal query, or a not-yet-written projection needs — and once compaction has dropped it, no replay can bring it back, so the failure is silent and permanent. The classic misuse is running compaction to reclaim space on a topic that is really an immutable audit record, quietly converting a source of truth into a lossy cache. The discipline is to classify each stream explicitly as compactible state or must-retain event, to derive the retention horizon from real downstream and legal needs rather than disk pressure, and to treat any irreversible drop as a decision requiring sign-off, not a background job.

How it implements the components

Log Compaction realizes the retention-and-space side of the archetype — the components that keep the log affordable and lawful:

  • snapshot_and_compaction_policy — it defines what is kept, collapsed, or discarded per key, and when compaction runs.
  • retention_redaction_and_access_policy — it governs how long history is held, what must be purged or redacted, and who may read it.

It reclaims space but neither serves as truth nor rebuilds: the canonical, authoritative log is Append-Only Event Store, and fast reconstruction from a retained checkpoint is Snapshot Plus Replay.

Notes

Compaction sits in genuine tension with the archetype's obligations of auditability and completeness. The more aggressively it forgets, the cheaper and faster the system — but the archetype's guarantee is precisely that history is not silently erased. Compaction is only safe where the discarded history is provably redundant to every legitimate rebuild; on any stream where that cannot be shown, retention, not compaction, is the correct default.

References

[1] Log compaction — as implemented in systems like Apache Kafka — retains at least the latest value for each key and removes superseded records, so the compacted log still fully determines current state per key while shedding redundant history. It is a keyed retention operation, not a time-based deletion. ↩