Designing Data-Intensive Applications¶
Kleppmann, M. (2017). Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. O'Reilly Media.
Cited by¶
11 citations across 11 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Custody Transfer
- Denormalization
- An e-commerce catalog that denormalizes a seller's rating into every product document — dropping read latency from a join-and-lookup to a single document read, at the price of a background job touching thousands of documents whenever a rating changes
This sourceTreats denormalization, derived/materialized data, and the write-side fan-out cost of keeping duplicated facts in sync (e.g., a value copied into many documents) against read-side gains.
- An e-commerce catalog that denormalizes a seller's rating into every product document — dropping read latency from a join-and-lookup to a single document read, at the price of a background job touching thousands of documents whenever a rating changes
- Fault Tolerance
- Fault tolerance clarifies that designing for perfect conditions is a planning failure, that robust systems must explicitly enumerate and plan for failure modes, that redundancy must address the actual failure modes expected (correlated vs independent), that trade-offs between cost and resilience are unavoidable and must be made explicitly, and that failure- mode analysis (FMEA) is a systematic design activity rather than an afterthought
This sourceTreats reliability and faults (Ch. 1), replication (Ch. 5), and the trouble with distributed systems including correlated/common-mode failures, partial failures, and operational pitfalls of untested failover (Ch. 8).
- Fault tolerance clarifies that designing for perfect conditions is a planning failure, that robust systems must explicitly enumerate and plan for failure modes, that redundancy must address the actual failure modes expected (correlated vs independent), that trade-offs between cost and resilience are unavoidable and must be made explicitly, and that failure- mode analysis (FMEA) is a systematic design activity rather than an afterthought
- Partition
- The hash-partitioning guarantee that any key routes to exactly one shard transfers to jurisdictional design, where the same engineering move — a routing function plus a canonical block list — produces agency mandates without overlap or gap.
This sourceChapter 6 (Partitioning) covers key-range and hash partitioning/sharding, under which each key (row) routes to exactly one partition — the database 'exactly one shard' guarantee. Resolves needs-source-433; the transfer to jurisdictional design (routing function plus canonical block list) is the prime's own cross-substrate analogy.
- The hash-partitioning guarantee that any key routes to exactly one shard transfers to jurisdictional design, where the same engineering move — a routing function plus a canonical block list — produces agency mandates without overlap or gap.
- Pull Flow
- The kanban-pull insight transferred into stream-processing back-pressure, where consumer demand pulls events and producers pause when consumers slow, and into request-driven serverless compute; the mechanism is identical.
This sourceTreats back-pressure in stream processing, where consumer demand pulls events and producers pause when consumers slow — the kanban-pull logic ported to data systems.
- The kanban-pull insight transferred into stream-processing back-pressure, where consumer demand pulls events and producers pause when consumers slow, and into request-driven serverless compute; the mechanism is identical.
- Serialization
- Writing serializes oral knowledge into text; notation serializes performance into a score; photography serializes a three-dimensional scene into a flat image; the central dogma serializes a folded protein's information into a linear nucleotide sequence and back; legal documentation serializes a verbal agreement into a contract; and flat-pack serializes an assembled object for shipment.
This sourceChapter 4 ('Encoding and Evolution') covers serialization formats (JSON, XML, Protocol Buffers, Avro, Thrift) and schema evolution / backward- and forward-compatibility as a discipline carried into API design; frames serialization as the structured-to-self-contained-bytes round trip.
- Writing serializes oral knowledge into text; notation serializes performance into a score; photography serializes a three-dimensional scene into a flat image; the central dogma serializes a folded protein's information into a linear nucleotide sequence and back; legal documentation serializes a verbal agreement into a contract; and flat-pack serializes an assembled object for shipment.
- Suspension
- A data-warehouse schema built for monthly batch reporting holds into a real-time architecture, dissonant against every new use case, discharged by a step-by-step migration with adapters and dual writes.
This sourceIncremental data-system migration with adapters and dual writes rather than a big-bang rewrite, the stepwise resolution of a legacy-schema mismatch.
- A data-warehouse schema built for monthly batch reporting holds into a real-time architecture, dissonant against every new use case, discharged by a step-by-step migration with adapters and dual writes.
Mechanisms¶
- Event Replay Deduplication
- Idempotent consumers are the standard way real systems get "exactly-once effects" out of "at-least-once delivery."
This sourceExplains how retried message delivery can still produce an effectively once-only processing result.
- Idempotent consumers are the standard way real systems get "exactly-once effects" out of "at-least-once delivery."
- Identifier Lifecycle Register
- The systems term for the preserved-but-dead marker is a tombstone.
This sourceUses tombstone for a retained deletion record that marks earlier values as dead for later compaction.
- The systems term for the preserved-but-dead marker is a tombstone.
- Serialization with Reconstruction Schema
- Formats built for exactly this — Apache Avro, Protocol Buffers — codify the rules: optional fields with defaults, reserved tags, reader-writer schema resolution.
This sourceExplains schema-evolution rules for Protocol Buffers and Avro, including defaults, stable tags, and reader–writer schema resolution.
- Formats built for exactly this — Apache Avro, Protocol Buffers — codify the rules: optional fields with defaults, reserved tags, reader-writer schema resolution.
- Staged Resynchronization Protocol
- The discipline is to fence old epochs, keep the reference independently checkable and genuinely contestable, add hysteresis to reopening, preserve competing versions until evidence discriminates, and never let restored agreement stand in for restored function.
This sourceShows how fencing tokens reject stale actors and how concurrent versions can be retained for later resolution.
- The discipline is to fence old epochs, keep the reference independently checkable and genuinely contestable, add hysteresis to reopening, preserve competing versions until evidence discriminates, and never let restored agreement stand in for restored function.
Verification¶
This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.
Links previously used in the corpus¶
Before the registry existed this work was also linked 3 other ways.
- https://dataintensive.net/ ×3
- https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/copyright-page01.html ×1
- https://www.worldcat.org/isbn/9781491903063 ×1
Registry ID ref:252c81b2d9a6 · see in the full table