This repository explains the concepts from "Designing Data-Intensive Applications" by Martin Kleppmann, with chapter-by-chapter breakdowns and practical examples.
Read it chapter by chapter, or study one topic at a time with the Reading paths and Topics by theme sections below.
-
Chapter 1: Reliable, Scalable, and Maintainable Applications - Foundation chapter covering the three fundamental concerns in software systems: reliability (fault tolerance), scalability (handling growth), and maintainability (operability, simplicity, evolvability).
-
Chapter 2: Data Models and Query Languages - Comprehensive exploration of data models including relational (SQL), document (NoSQL), and graph databases, along with their query languages, trade-offs, and when to use each model.
-
Chapter 3: Storage and Retrieval - Deep dive into how databases store and retrieve data, covering hash indexes, SSTables, LSM-trees, B-trees, and the differences between OLTP (row-oriented) and OLAP (column-oriented) storage engines.
-
Chapter 4: Encoding and Evolution - Comprehensive guide to data encoding formats (JSON, XML, Thrift, Protocol Buffers, Avro), schema evolution strategies, backward/forward compatibility, and dataflow patterns through databases, services, and message queues.
-
Chapter 5: Replication - Explores replication strategies including single-leader, multi-leader, and leaderless replication, along with handling consistency challenges and conflicts.
-
Chapter 6: Partitioning - Covers partitioning strategies (sharding) for scalability, including key-range and hash-based partitioning, secondary indexes, rebalancing techniques, and request routing approaches.
-
Chapter 7: Transactions - Deep dive into ACID properties, isolation levels (Read Committed, Snapshot Isolation, Serializable), concurrency control mechanisms (2PL, MVCC, SSI), and distributed transactions with two-phase commit.
-
Chapter 8: The Trouble with Distributed Systems - Explores the fundamental challenges of distributed systems including unreliable networks, clock synchronization issues, process pauses, partial failures, Byzantine faults, and system models for reasoning about failures.
-
Chapter 9: Consistency and Consensus - Comprehensive coverage of consistency models (from eventual consistency to linearizability), ordering guarantees, causality, consensus algorithms (2PC, Paxos, Raft), and coordination services (ZooKeeper, etcd, Consul) for building fault-tolerant distributed systems.
-
Chapter 10: Batch Processing - In-depth exploration of batch processing systems from Unix tools to MapReduce and modern dataflow engines (Spark, Flink), covering distributed filesystems (HDFS), join algorithms, graph processing (Pregel model), fault tolerance strategies, and the evolution toward declarative SQL interfaces.
-
Chapter 11: Stream Processing - Comprehensive guide to stream processing covering event streams, message brokers (Kafka), change data capture (CDC), event sourcing, stream joins (stream-stream, stream-table), windowing operations, handling time and late events with watermarks, fault tolerance with exactly-once semantics, and comparison of stream processing frameworks.
-
Chapter 12: The Future of Data Systems - Integrating everything together: data integration patterns, unbundling databases, derived data with Lambda and Kappa architectures, CQRS, end-to-end correctness with idempotence, enforcing constraints in distributed systems, trust and verification with audit logs, and ethical considerations for privacy and fairness in data systems design.
Every link below jumps straight to the section that covers a topic, so you don't need to read the whole chapter.
- Reading paths are ordered lists of sections for a specific goal.
- Topics by theme list each concept with:
- Start here: the section that explains it.
- Also see: other sections, often in other chapters, where the idea comes up again.
- Read first: what you need to know for the section to make sense. Skip it if you already know the material.
- Ch 3 · The Simplest Database
- Ch 3 · Hash Indexes
- Ch 3 · SSTables and LSM-Trees
- Ch 3 · B-Trees
- Ch 3 · B-Tree vs LSM-Tree
- Ch 7 · How Durability Works (write-ahead log)
- Ch 3 · Transaction Processing vs Analytics
- Ch 3 · Column-Oriented Storage
- Ch 1 · Approaches for Coping with Load
- Ch 5 · Leaders and Followers
- Ch 5 · Synchronous vs. Asynchronous Replication
- Ch 5 · Problems with Replication Lag
- Ch 5 · Leaderless Replication
- Ch 6 · Partitioning of Key-Value Data
- Ch 6 · Partitioning and Secondary Indexes
- Ch 6 · Rebalancing Partitions
- Ch 6 · Request Routing
- Ch 7 · The ACID Properties
- Ch 7 · Read Committed
- Ch 7 · Snapshot Isolation
- Ch 7 · Lost Updates
- Ch 7 · Write Skew and Phantoms
- Ch 7 · Serializable Isolation
- Ch 7 · Two-Phase Commit
- Ch 8 · Faults and Partial Failures
- Ch 8 · Unreliable Networks
- Ch 8 · Unreliable Clocks
- Ch 8 · Process Pauses
- Ch 8 · The Truth is Defined by the Majority
- Ch 9 · Linearizability
- Ch 9 · The Cost of Linearizability (CAP)
- Ch 9 · Causality
- Ch 9 · Total Order Broadcast
- Ch 9 · Raft Consensus Algorithm
- Ch 9 · Apache ZooKeeper
- Ch 4 · Modes of Dataflow
- Ch 11 · Message Brokers vs Event Logs
- Ch 11 · Apache Kafka Architecture
- Ch 11 · Change Data Capture
- Ch 11 · Event Sourcing
- Ch 11 · Time in Stream Processing
- Ch 11 · Fault Tolerance
- Ch 12 · Data Integration
- Ch 12 · Derived Data (Lambda and Kappa)
- Ch 12 · End-to-End Argument for Data Systems
- Ch 1 · Describing Load and Describing Performance
- Ch 3 · B-Tree vs LSM-Tree
- Ch 5 · Leaders and Followers
- Ch 5 · Problems with Replication Lag
- Ch 5 · Writing and Reading with Quorum
- Ch 6 · Consistent Hashing
- Ch 6 · Skewed Workloads and Hot Spots
- Ch 7 · Isolation Levels
- Ch 9 · The Cost of Linearizability (CAP)
- Ch 9 · Consensus Algorithms
- Ch 11 · Apache Kafka Architecture
- Ch 12 · Exactly-Once Semantics
| Topic | Start here | Also see | Read first |
|---|---|---|---|
| Faults vs failures | Ch 1 · Reliability | Hardware Faults, Software Errors, Human Errors | — |
| Load parameters and fan-out | Ch 1 · Describing Load | — | — |
| Latency percentiles (p99) | Ch 1 · Describing Performance | — | Describing Load |
| Scaling up vs scaling out | Ch 1 · Approaches for Coping with Load | Ch 6 · Why Partition Data? | Describing Load |
| Operability, simplicity, evolvability | Ch 1 · Maintainability | Ch 4 · Schema Evolution | — |
| Topic | Start here | Also see | Read first |
|---|---|---|---|
| Relational vs document model | Ch 2 · Relational Model vs Document Model | Comparison, Choosing a Data Model | — |
| One-to-many, many-to-many relationships | Ch 2 · One-to-Many Relationships | Many-to-One and Many-to-Many | Relational vs Document |
| Schema-on-read vs schema-on-write | Ch 2 · Schema Flexibility | Ch 4 · Schema Evolution | Relational vs Document |
| Data locality | Ch 2 · Data Locality | — | The Document Model |
| Declarative vs imperative queries | Ch 2 · Declarative vs Imperative | Ch 10 · Declarative Query Languages | — |
| MapReduce as a query model | Ch 2 · MapReduce | Aggregation Pipeline, Ch 10 · MapReduce Job Execution | Declarative vs Imperative |
| Graph data models | Ch 2 · Graph Databases | Property Graphs, Triple Stores, When to Use Graph Databases | — |
| Topic | Start here | Also see | Read first |
|---|---|---|---|
| JSON, XML, CSV and their pitfalls | Ch 4 · JSON, XML, and CSV | Language-Specific Formats | — |
| Thrift and Protocol Buffers | Ch 4 · Thrift and Protocol Buffers | Binary Encoding | JSON, XML, and CSV |
| Backward and forward compatibility | Ch 4 · Schema Evolution | Adding Fields, Removing Fields, Changing Field Types | Thrift and Protocol Buffers |
| Avro writer's and reader's schema | Ch 4 · Avro | Writer's vs Reader's Schema, Avro vs Thrift/Protocol Buffers | Schema Evolution |
| Dataflow through databases, REST/RPC, messaging | Ch 4 · Modes of Dataflow | Ch 11 · Message Brokers vs Event Logs | Schema Evolution |
| Topic | Start here | Also see | Read first |
|---|---|---|---|
| What partitioning is | Ch 6 · What is Partitioning? | Partitioning vs. Replication | Ch 5 · Leaders and Followers |
| Key-range vs hash partitioning | Ch 6 · Partitioning of Key-Value Data | By Key Range, By Hash of Key | What is Partitioning? |
| Consistent hashing | Ch 6 · Consistent Hashing | Don't Hash Mod N | Partitioning by Hash of Key |
| Hot spots and skew | Ch 6 · Skewed Workloads and Hot Spots | Ch 10 · Handling Skew | Partitioning by Hash of Key |
| Local vs global secondary indexes | Ch 6 · Partitioning and Secondary Indexes | Comparison: Local vs. Global, Ch 3 · Secondary Indexes | Partitioning of Key-Value Data |
| Rebalancing strategies | Ch 6 · Rebalancing Partitions | Comparison of Rebalancing Strategies, Automatic vs. Manual | Partitioning of Key-Value Data |
| Request routing and gossip | Ch 6 · Request Routing | Coordination Service, Gossip Protocol, Ch 9 · Service Discovery | Rebalancing Partitions |
| Topic | Start here | Also see | Read first |
|---|---|---|---|
| What ACID means | Ch 7 · The ACID Properties | Atomicity, Consistency, Isolation, Durability | — |
| Abort and retry | Ch 7 · Abort and Retry | Ch 12 · Duplicate Suppression | Atomicity |
| Choosing an isolation level | Ch 7 · Isolation Levels | Choosing an Isolation Level, Real-World Database Isolation | Isolation |
| Read committed (dirty reads and writes) | Ch 7 · Read Committed | No Dirty Reads, No Dirty Writes | Isolation Levels |
| Snapshot isolation and MVCC | Ch 7 · Snapshot Isolation | MVCC | Read Committed |
| Lost updates | Ch 7 · Lost Updates | Solutions to Lost Updates | Snapshot Isolation |
| Write skew and phantoms | Ch 7 · Write Skew and Phantoms | Solutions to Write Skew, Ch 12 · Uniqueness Constraints | Lost Updates |
| Serial execution, 2PL, SSI | Ch 7 · Serializable Isolation | Actual Serial Execution, Two-Phase Locking, SSI | Write Skew and Phantoms |
| Distributed transactions and 2PC | Ch 7 · Distributed Transactions | Two-Phase Commit, Ch 9 · Two-Phase Commit, Ch 9 · Three-Phase Commit | Atomicity |
| Topic | Start here | Also see | Read first |
|---|---|---|---|
| Single source of truth | Ch 12 · Data Integration | Better Approach: Single Source of Truth | Ch 11 · Change Data Capture |
| Unbundling databases | Ch 12 · Unbundling Databases | Composing Data Storage Technologies | Data Integration |
| CQRS | Ch 12 · Designing Applications Around Dataflow | — | Ch 11 · Event Sourcing |
| Lambda vs Kappa architecture | Ch 12 · Derived Data | Lambda Architecture, Kappa Architecture | Ch 10 · MapReduce Job Execution, Ch 11 · Stream Analytics |
| Exactly-once and idempotence end to end | Ch 12 · End-to-End Argument | Exactly-Once Semantics, Duplicate Suppression, Ch 11 · Idempotence | Ch 7 · Abort and Retry |
| Constraints without coordination | Ch 12 · Enforcing Constraints | Uniqueness Constraints, Timeliness and Integrity, Coordination-Avoidance | Ch 9 · Total Order Broadcast |
| Auditing and verification | Ch 12 · Trust, But Verify | Auditing, Designing for Auditability | — |
| Privacy and ethics | Ch 12 · Doing the Right Thing | Privacy and Data Protection | — |