Designing Data-Intensive Applications¶
Kleppmann, M. (2017). Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. O’Reilly Media.
Cited by¶
32 citations across 32 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Custody Transfer
- Denormalization
- An e-commerce catalog that denormalizes a seller's rating into every product document — dropping read latency from a join-and-lookup to a single document read, at the price of a background job touching thousands of documents whenever a rating changes
This sourceTreats denormalization, derived/materialized data, and the write-side fan-out cost of keeping duplicated facts in sync (e.g., a value copied into many documents) against read-side gains.
- An e-commerce catalog that denormalizes a seller's rating into every product document — dropping read latency from a join-and-lookup to a single document read, at the price of a background job touching thousands of documents whenever a rating changes
- Fault Tolerance
- Fault tolerance clarifies that designing for perfect conditions is a planning failure, that robust systems must explicitly enumerate and plan for failure modes, that redundancy must address the actual failure modes expected (correlated vs independent), that trade-offs between cost and resilience are unavoidable and must be made explicitly, and that failure- mode analysis (FMEA) is a systematic design activity rather than an afterthought
This sourceTreats reliability and faults (Ch. 1), replication (Ch. 5), and the trouble with distributed systems including correlated/common-mode failures, partial failures, and operational pitfalls of untested failover (Ch. 8).
- Fault tolerance clarifies that designing for perfect conditions is a planning failure, that robust systems must explicitly enumerate and plan for failure modes, that redundancy must address the actual failure modes expected (correlated vs independent), that trade-offs between cost and resilience are unavoidable and must be made explicitly, and that failure- mode analysis (FMEA) is a systematic design activity rather than an afterthought
- Partition
- The hash-partitioning guarantee that any key routes to exactly one shard transfers to jurisdictional design, where the same engineering move — a routing function plus a canonical block list — produces agency mandates without overlap or gap.
This sourceChapter 6 (Partitioning) covers key-range and hash partitioning/sharding, under which each key (row) routes to exactly one partition — the database 'exactly one shard' guarantee. Resolves needs-source-433; the transfer to jurisdictional design (routing function plus canonical block list) is the prime's own cross-substrate analogy.
- The hash-partitioning guarantee that any key routes to exactly one shard transfers to jurisdictional design, where the same engineering move — a routing function plus a canonical block list — produces agency mandates without overlap or gap.
- Pull Flow
- The kanban-pull insight transferred into stream-processing back-pressure, where consumer demand pulls events and producers pause when consumers slow, and into request-driven serverless compute; the mechanism is identical.
This sourceTreats back-pressure in stream processing, where consumer demand pulls events and producers pause when consumers slow — the kanban-pull logic ported to data systems.
- The kanban-pull insight transferred into stream-processing back-pressure, where consumer demand pulls events and producers pause when consumers slow, and into request-driven serverless compute; the mechanism is identical.
- Serialization
- Writing serializes oral knowledge into text; notation serializes performance into a score; photography serializes a three-dimensional scene into a flat image; the central dogma serializes a folded protein's information into a linear nucleotide sequence and back; legal documentation serializes a verbal agreement into a contract; and flat-pack serializes an assembled object for shipment.
This sourceChapter 4 ('Encoding and Evolution') covers serialization formats (JSON, XML, Protocol Buffers, Avro, Thrift) and schema evolution / backward- and forward-compatibility as a discipline carried into API design; frames serialization as the structured-to-self-contained-bytes round trip.
- Writing serializes oral knowledge into text; notation serializes performance into a score; photography serializes a three-dimensional scene into a flat image; the central dogma serializes a folded protein's information into a linear nucleotide sequence and back; legal documentation serializes a verbal agreement into a contract; and flat-pack serializes an assembled object for shipment.
- Suspension
- A data-warehouse schema built for monthly batch reporting holds into a real-time architecture, dissonant against every new use case, discharged by a step-by-step migration with adapters and dual writes.
This sourceIncremental data-system migration with adapters and dual writes rather than a big-bang rewrite, the stepwise resolution of a legacy-schema mismatch.
- A data-warehouse schema built for monthly batch reporting holds into a real-time architecture, dissonant against every new use case, discharged by a step-by-step migration with adapters and dual writes.
Domain-specific¶
- CAP Theorem
- This forced choice defines the two canonical design families: CP systems (Bigtable, Spanner, etcd, HBase) refuse client requests or error during partition rather than serve stale data
This sourceWhy a linearizable store must block or return errors on disconnected replicas during a network fault rather than serve stale data.
Supported in partVerified against the source
- This forced choice defines the two canonical design families: CP systems (Bigtable, Spanner, etcd, HBase) refuse client requests or error during partition rather than serve stale data
- Stream Processing
- Earlier systems — Kafka for durable log storage and replay, Apache Storm for low-latency processing, Spark Streaming for micro-batch approximation — each resolved the event-time/processing-time tension differently, and the design space they explored shaped what is now the standard architecture: a durable ordered log as the input channel, a stateful dataflow graph as the compute layer, and a key-partitioned state store that allows processing to scale horizontally and recover from failure by replaying the log from the last committed offset.
This sourceTextbook treatment of log-based message brokers showing an ordered log consumed through tracked offsets and replayable from the last recorded offset after a failure. Textbook treatment presenting log-based message brokers (Kafka's family) as the durable-log messaging tier that took its ideas from databases.
Supported in partVerified against a saved copy of the source
“several assumptions: restarting a failed task must replay the same messages in the same order”
- Earlier systems — Kafka for durable log storage and replay, Apache Storm for low-latency processing, Spark Streaming for micro-batch approximation — each resolved the event-time/processing-time tension differently, and the design space they explored shaped what is now the standard architecture: a durable ordered log as the input channel, a stateful dataflow graph as the compute layer, and a key-partitioned state store that allows processing to scale horizontally and recover from failure by replaying the log from the last committed offset.
Mechanisms¶
- Anti-Entropy Reconciliation Exchange
- The subtler trap is that the exchange only converges replicas to agreement, not to correctness: reconcile by wall-clock "last write wins" instead of real causal order and the sweep will faithfully propagate a value that silently erased a concurrent update — every replica now agrees on the wrong answer.
This sourceKleppmann explains that timestamp-based last-write-wins reconciliation can make replicas converge by silently discarding concurrent writes, so convergence does not itself guarantee correctness.
- The subtler trap is that the exchange only converges replicas to agreement, not to correctness: reconcile by wall-clock "last write wins" instead of real causal order and the sweep will faithfully propagate a value that silently erased a concurrent update — every replica now agrees on the wrong answer.
- Cache Invalidation Review
- Its failure mode is that the hard part is genuinely hard — a rule that under-fires serves stale reads that look authoritative, and stale data is worse than obviously-missing data because nobody double-checks it.
This sourceExplains that caches not invalidated when source data changes can return stale data as normal read results.
- Its failure mode is that the hard part is genuinely hard — a rule that under-fires serves stale reads that look authoritative, and stale data is worse than obviously-missing data because nobody double-checks it.
- Data Validation Schema
- Validating at write keeps trusted storage clean; validating at read tolerates messy intake but defers the risk.
This sourceContrasts schema-on-write, in which a database enforces structure when data is written, with schema-on-read, in which structure is interpreted when data is read.
- Validating at write keeps trusted storage clean; validating at read tolerates messy intake but defers the risk.
- Durable Queue with Acknowledgement
- The guarantee is at-least-once, and the duplicates it implies are not optional — a consumer that assumes each message arrives exactly once will double-apply on every redelivery.
This sourceExplains that at-least-once delivery permits redelivery and therefore exposes consumers to duplicate messages.
- The guarantee is at-least-once, and the duplicates it implies are not optional — a consumer that assumes each message arrives exactly once will double-apply on every redelivery.
- Event Replay Deduplication
- Idempotent consumers are the standard way real systems get "exactly-once effects" out of "at-least-once delivery."
This sourceExplains how retried message delivery can still produce an effectively once-only processing result.
- Idempotent consumers are the standard way real systems get "exactly-once effects" out of "at-least-once delivery."
- Idempotency and Deduplication Ledger
- Because at-least-once delivery is the normal, correct behavior of any durable pipeline, something must absorb the duplicates it produces — and a ledger keyed on change identity does exactly that, turning "the message came twice" into a no-op with a truthful reply.
This sourceExplains that retrying stream-processing work can duplicate effects, requiring those duplicates to be absorbed safely.
- Because at-least-once delivery is the normal, correct behavior of any durable pipeline, something must absorb the duplicates it produces — and a ledger keyed on change identity does exactly that, turning "the message came twice" into a no-op with a truthful reply.
- Idempotency Key & Deduplication Store
- Its strength is converting an at-least-once delivery world — the only kind a retrying distributed system can cheaply offer — into effectively exactly-once behaviour at the operation level, which is what makes safe retries, recovery replay, and duplicate messages tolerable at all.
This sourceShows how idempotence and deduplication make retried or replayed operations produce the same effect as a single execution.
- Its strength is converting an at-least-once delivery world — the only kind a retrying distributed system can cheaply offer — into effectively exactly-once behaviour at the operation level, which is what makes safe retries, recovery replay, and duplicate messages tolerable at all.
- Identifier Lifecycle Register
- The systems term for the preserved-but-dead marker is a tombstone.
This sourceUses tombstone for a retained deletion record that marks earlier values as dead for later compaction.
- The systems term for the preserved-but-dead marker is a tombstone.
- Last-Write-Wins Register
- Its danger is that the loss is silent and invisible: two users make concurrent edits, both receive success responses, and one edit vanishes with no error and no trace
This sourceShows that last-write-wins can silently discard concurrent writes without reporting an error to the application.
- Its danger is that the loss is silent and invisible: two users make concurrent edits, both receive success responses, and one edit vanishes with no error and no trace
- Leader-Based Replication
- The classic and most dangerous failure is split-brain — a partition leaves two nodes each believing they lead, both accepting divergent writes.
This sourceDescribes split brain as a network partition leaving multiple nodes believing they are leader and accepting divergent writes.
- The classic and most dangerous failure is split-brain — a partition leaves two nodes each believing they lead, both accepting divergent writes.
- Leader Election and Term Protocol
- But the checkpoint store now rejects any write not stamped with the current term 9 — the old manager's term-8 write is fenced out and fails.
This sourceKleppmann's fencing pattern has storage reject an old lower-numbered token after it has processed a write carrying a newer token.
- But the checkpoint store now rejects any write not stamped with the current term 9 — the old manager's term-8 write is fenced out and fails.
- Lease and Fencing-Token Protocol
- Its correctness is only as good as its assumptions: leases lean on bounded clock skew, and fencing requires the protected resource to actually check the token — a resource that ignores tokens gets no protection at all.
This sourceExplains that lease safety depends on timing assumptions and that fencing works only when the protected resource validates and rejects stale tokens.
- Its correctness is only as good as its assumptions: leases lean on bounded clock skew, and fencing requires the protected resource to actually check the token — a resource that ignores tokens gets no protection at all.
- Lease Expiry Sweep
- Worse, a holder whose lease expired might resume — after a pause or a network partition — and act on a resource that has already been reassigned, unless it is fenced.
This sourceShows how an expired lease holder can resume and act on a reassigned resource, and how fencing tokens make the resource reject the stale operation.
- Worse, a holder whose lease expired might resume — after a pause or a network partition — and act on a resource that has already been reassigned, unless it is fenced.
- Lease, Lock, or Reservation Token
- The standard guard is a fencing token: a monotonically increasing number issued with each lease that the resource itself checks, rejecting any write from a holder once a newer lease has been handed out.
This sourceDefines fencing tokens as monotonically increasing lease numbers that the protected resource checks to reject operations from stale lease holders.
- The standard guard is a fencing token: a monotonically increasing number issued with each lease that the resource itself checks, rejecting any write from a holder once a newer lease has been handed out.
- Materialized View or Cache
- The derived copy can drift from its base, and a stale result served as truth is the failure that gives cache invalidation its reputation as one of computing's genuinely hard problems.
This sourceExplains that caches and derived data can become stale when underlying data changes, making invalidation and consistency between copies intrinsically difficult.
- The derived copy can drift from its base, and a stale result served as truth is the failure that gives cache invalidation its reputation as one of computing's genuinely hard problems.
- Multi-Leader Replication with Conflict Resolution
- Last-write-wins is the usual culprit: it silently discards the losing concurrent write, so data loss masquerades as consistency, and under clock skew it can even drop the write that actually happened later.
This sourceExplains that last-write-wins discards concurrent writes silently and that clock skew can cause a causally later write to be lost.
- Last-write-wins is the usual culprit: it silently discards the losing concurrent write, so data loss masquerades as consistency, and under clock skew it can even drop the write that actually happened later.
- Resolution Event Commit
- Commit atomically under a guard. Perform the pending→terminal transition as a conditional, compare-and-set write
This sourceDefines compare-and-set as a conditional write that takes effect only when the current value still matches the value previously read.
- Commit atomically under a guard. Perform the pending→terminal transition as a conditional, compare-and-set write
- Serialization with Reconstruction Schema
- Formats built for exactly this — Apache Avro, Protocol Buffers — codify the rules: optional fields with defaults, reserved tags, reader-writer schema resolution.
This sourceExplains schema-evolution rules for Protocol Buffers and Avro, including defaults, stable tags, and reader–writer schema resolution.
- Formats built for exactly this — Apache Avro, Protocol Buffers — codify the rules: optional fields with defaults, reserved tags, reader-writer schema resolution.
- Serialized Job Envelope
- Because durable delivery is usually at-least-once, a non-idempotent job can also take effect twice.
This sourceExplains that at-least-once delivery can repeat message processing and that side effects must therefore be made idempotent.
- Because durable delivery is usually at-least-once, a non-idempotent job can also take effect twice.
- Staged Resynchronization Protocol
- The discipline is to fence old epochs, keep the reference independently checkable and genuinely contestable, add hysteresis to reopening, preserve competing versions until evidence discriminates, and never let restored agreement stand in for restored function.
This sourceShows how fencing tokens reject stale actors and how concurrent versions can be retained for later resolution.
- The discipline is to fence old epochs, keep the reference independently checkable and genuinely contestable, add hysteresis to reopening, preserve competing versions until evidence discriminates, and never let restored agreement stand in for restored function.
- Subscriber Change Webhook
- Its failure mode follows from at-least-once delivery: webhooks can be delivered late, more than once
This sourceExplains that at-least-once delivery may redeliver a message, so consumers must tolerate duplicates.
- Its failure mode follows from at-least-once delivery: webhooks can be delivered late, more than once
- Subscription Health Dashboard
- Its failure mode is measuring delivery while a subscriber silently fails at processing: a dashboard can glow green because events were handed over, even as the consumer drops them on the floor after receipt
This sourceDistinguishes successful message delivery from successful downstream processing and explains how a consumer can receive and then fail to process an event.
- Its failure mode is measuring delivery while a subscriber silently fails at processing: a dashboard can glow green because events were handed over, even as the consumer drops them on the floor after receipt
- Synchronization Conflict Queue
- By making an unresolved conflict a visible, owned item that constrains the action it touches, the queue ensures consequential decisions know whether the state they consume is accepted or still disputed — the opposite of a green light over a silent overwrite.
This sourceWarns that last-write-wins conflict resolution can silently discard concurrent writes even after reporting them as successful.
- By making an unresolved conflict a visible, owned item that constrains the action it touches, the queue ensures consequential decisions know whether the state they consume is accepted or still disputed — the opposite of a green light over a silent overwrite.
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Read against the text for 2 of 32 citations: 2 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Links previously used in the corpus¶
Before the registry existed this work was also linked 3 other ways.
- https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/copyright-page01.html ×4
- https://dataintensive.net/ ×3
- https://www.worldcat.org/isbn/9781491903063 ×1
Registry ID ref:252c81b2d9a6 · see in the full table