Commit Log and Compaction Cycle¶
Append-and-compact protocol — instantiates Fast–Slow Store Coupling
Records every change first as an append-only log entry, then periodically compacts the accumulated log into compact stable state — reclaiming space and dropping superseded records.
The defining move of this mechanism is that the fast store never overwrites — it only appends. Every insert, update, and delete lands at the tail of a sequential log, which is the cheapest and most crash-safe write a machine can make. Durability is therefore bought before integration is even attempted: the moment a change is fsynced to the log, it cannot be lost, even though the durable state it belongs in has not yet been formed. The slow side of the coupling is a separate, periodic compaction pass that folds the accumulated log entries into a compact, deduplicated representation — keeping the latest value for each key, discarding everything the latest value superseded, and physically reclaiming the space. Fast capture and slow integration are thus split cleanly in time: write fast now, tidy later, on a cadence of the system's own choosing.
Example¶
A team runs a metrics database ingesting readings from a fleet of factory sensors. Writes arrive in unpredictable bursts, and the one thing they cannot tolerate is losing a reading during a power blip. So the storage engine is built log-structured: each incoming sample is appended to a write-ahead log and mirrored into an in-memory table, and the write is acknowledged the instant the log entry is durable. No sensor waits for the data to be sorted, indexed, or merged with anything.
When the in-memory table fills, it is flushed as an immutable sorted file, and the log segment behind it is retired. Over a day the disk accumulates dozens of these files, many holding stale versions of keys that were overwritten later. A background compaction pass then merges overlapping files, keeps only the newest value for each series, drops the tombstones left by deletes, and rewrites the survivors into a smaller, tidier set. The outcome: writes stayed microsecond-cheap during the day, and the durable store still ends the day coherent and compact — because the two jobs were never asked of the same operation.
How it works¶
- Append, then acknowledge. Every mutation is written sequentially to the log and only then confirmed. Reads consult the in-memory table plus the on-disk files, newest first.
- Flush on fill. When the memory table crosses its size bound, it is sealed and written out as an immutable run; the corresponding log segment can be released.
- Compact on a trigger. A separate pass merges runs when their number or overlap crosses a threshold, resolving each key to its latest value and physically dropping superseded records and tombstones.
- Tune the two amplifications against each other. More aggressive compaction means fewer files to read (lower read cost) but more rewriting (higher write cost); the cadence is the knob that trades one for the other.
Tuning parameters¶
- Compaction trigger — fire on file count, total size, or overlap ratio. Eager compaction keeps reads fast but rewrites data more often; lazy compaction saves write effort but lets read cost and disk footprint balloon.
- Memtable size — how much is buffered before a flush. Larger buffers make bigger, better-sorted runs but widen the window of unflushed (log-only) state and raise memory use.
- Compaction strategy — leveled (tight, read-optimized, write-heavy) versus tiered/size-based (write-friendly, more read fan-out). The choice follows the read/write mix of the workload.
- Tombstone retention — how long deletion markers survive before being dropped. Too short risks resurrecting deleted data across replicas; too long bloats the store.
- Log retention — how much log to keep after flush for recovery and replication before it is truncated.
When it helps, and when it misleads¶
Its strength is write-heavy, crash-sensitive workloads: sequential appends are the fastest durable write available, and integration is deferred to a moment the system picks rather than one the user waits on. It also degrades gracefully — a crash loses at most the unflushed tail, which the log replays on restart.
Its signature failure mode is write amplification: the same logical datum is physically rewritten several times as it is repeatedly compacted, so a workload that looks light on paper can saturate disk bandwidth with background merges.[n1] The classic misuse is running it under an update-in-place workload where nearly every key is rewritten constantly — the log fills with garbage faster than compaction can clear it, read fan-out explodes, and compaction "storms" starve foreground work. The guarding discipline is to match compaction strategy and cadence to the real read/write ratio and to watch amplification as a first-class metric, not to assume that because writes are cheap the store is free.
How it implements the components¶
This mechanism fills the capture-and-fold subset of the archetype's machinery:
interference_and_overwrite_guard— the append-only log is the guard: a change is made durable at the tail before any later write can overwrite or crowd out the value it replaced, so no fragile trace is lost pre-consolidation.transfer_trigger_or_cadence— flush-on-fill plus threshold-triggered compaction are the explicit cadence that moves data from log to stable runs.consolidation_transform— compaction is the transform: it merges sorted runs, resolves each key to one value, and rewrites into compact form.promotion_and_eviction_rule— compaction promotes the latest version of each key and evicts superseded versions and tombstoned keys, keeping the store bounded.
It does not route reads against a separate authoritative origin or expire on a freshness clock (freshness_and_authority_marker, read_write_routing_rule) — that is Edge Cache with Origin Synchronization; and it runs no transfer_backlog_dashboard to meter ingestion lag against an SLA — that belongs to its nearest twin, Staging Table to Canonical Warehouse Pipeline, which validates rows into a schema rather than compacting an append-only log.
Related¶
- Instantiates: Fast–Slow Store Coupling — supplies the capture-then-fold coupling: append-only fast log, periodic compaction into durable state.
- Sibling mechanisms: Edge Cache with Origin Synchronization · Hot/Warm/Cold Storage Tiering · Inbox-to-Archive Processing Workflow · Incident Timeline to Permanent Runbook Update · Local Inventory Cache with System-of-Record Refresh · Scratchpad-to-Knowledge-Base Curation · Short-Term to Long-Term Memory Consolidation Routine · Staging Table to Canonical Warehouse Pipeline · Write-Back Cache with Durable Backing Store
Editorial Notes¶
Form Classification¶
Form family: Control, Automation & Runtime
Rationale: Records every change first as an append-only log entry, then periodically compacts the accumulated log into compact stable state — reclaiming space and dropping superseded records, making its operative form a live operational control that automatically routes, enforces, adapts, or responds during execution.
Independent corroboration: The frozen evidence defines Commit Log and Compaction Cycle as 'Records every change first as an append-only log entry, then periodically compacts the accumulated log into compact stable state — reclaiming space and dropping superseded records', so its operative form is Control, Automation & Runtime.
Nearest alternative: Protocol, Workflow & Routine — Automatic append, flush, and compaction triggers continuously govern live storage behavior rather than relying on an operator-run workflow.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Database and storage-system engineering established append-only commit logs followed by periodic compaction into deduplicated stable state.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
[n1] Write amplification is the ratio of bytes physically written to storage over bytes logically written by the application. Log-structured designs trade cheap sequential writes for repeated rewriting during compaction, so amplification — not raw write latency — is usually the metric that governs how such a store scales. ↩