Bird
Raised Fist0

Durability - How Systems Guarantee Data is Never Lost

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Durability - How Systems Guarantee Data is Never Lost
Ensuring data persistence and fault tolerance at scale with concrete durability guarantees
mediumNon-Functional Requirements FrameworkFreq: common
⚙️
Functional Requirements
  • Persist all incoming data writes without loss under normal and failure conditions
  • Support atomic write operations with durability guarantees
  • Provide data replication across multiple nodes or data centers for fault tolerance
  • Allow data recovery after crashes or hardware failures without data loss
  • Support read-after-write consistency for critical use cases
  • Enable configurable retention policies for data lifecycle management
📊
Non-Functional Requirements
  • Durability SLA of 99.999% (five nines) uptime for data availability
  • Support 2 billion daily active users generating 40 messages per user per day
  • Write latency P99 under 100ms for durability confirmation
  • Data retention configurable up to 30 days with replication factor of 3
  • System must handle peak write QPS of at least 1 million writes/sec
❓
Clarifying Questions to Ask
  • What is the expected data write and read throughput at peak and average?
  • What is the acceptable latency for write durability confirmation?
  • Is cross-region replication required or only within a single data center?
  • What is the maximum data retention period and replication factor?
  • Are there any specific consistency models required (e.g., strong, eventual)?
  • What failure scenarios must be tolerated (node crash, network partition, disk failure)?
Out of Scope
✗ Data analytics or querying beyond simple reads
✗ Complex transactional consistency beyond durability guarantees
✗ User authentication and authorization mechanisms
✗ Data schema evolution and versioning
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates 40 messages per day
  • Average message size is 100 bytes
  • Replication factor of 3 for durability
⚡
QPS Calculation
write qps2,000,000,000 users × 40 msgs/user/day ÷ 86,400 sec/day ≈ 925,926 msg/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1,851,852 msg/sec
read qpsAssuming read-to-write ratio 1:1: 925,926 msg/sec
💾
Storage Estimate
daily storage2B × 40 × 100 bytes = 8,000,000,000,000 bytes = 8 TB/day
retention 30 days8 TB/day × 30 days = 240 TB
with replication240 TB × 3 = 720 TB total storage
🖥️
Server Estimate
storage nodesAssuming each storage node handles 1 TB data and 10K writes/sec, need 720 nodes for storage capacity and throughput
write coordinator nodesAssuming each node handles 50K write QPS, need 40 nodes for write coordination
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Client Writes |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Write Coordinator|
          +-------+--------+
                  |
                  v
          +-------+--------+       +----------------+
          | Kafka Cluster  |------>| Replication    |
          +-------+--------+       | & Storage Nodes|
                  |                +-------+--------+
                  |                        |
                  v                        v
          +-------+--------+       +----------------+
          | Storage Nodes  |<------| Replication    |
          +-------+--------+       +----------------+
                  |
                  v
          +-------+--------+
          | Read API Layer |
          +----------------+
🧩
Components
Write Coordinator
Receives client writes, validates, and orchestrates replication and persistence
Why thisStateless microservice layer enables horizontal scaling and isolates client-facing logic from storage internals
Why notDirect client writes to storage nodes rejected due to complexity in client-side sharding and lack of centralized write control
Kafka Cluster
Durable distributed log to buffer writes and enable asynchronous replication
Why thisKafka provides high-throughput, partitioned, replicated logs with configurable durability guarantees and retention
Why notRabbitMQ rejected due to lower throughput and weaker durability guarantees at 1.8M writes/sec scale
Storage Nodes (Cassandra)
Persist data with replication and support fast writes and reads
Why thisApache Cassandra offers tunable consistency, linear write scalability, and built-in replication across nodes
Why notPostgreSQL rejected due to lack of native horizontal scaling and complex manual sharding at 1.8M writes/sec
Replication Service
Ensures data is replicated across multiple storage nodes and data centers for durability
Why thisCustom replication logic layered on Kafka consumer groups ensures ordered, reliable replication with failure detection
Why notRelying solely on Cassandra’s internal replication rejected due to slower cross-datacenter replication and weaker failure isolation
Read API Layer
Serves read requests with read-after-write consistency where required
Why thisStateless read service caches hot data and routes queries to storage nodes with consistency level control
Why notDirect client reads from storage rejected due to complexity in consistency enforcement and caching
🔄
Data Flow
1
Client → Write Coordinator
Client sends write request
Write Coordinator validates and assigns partition key
2
Write Coordinator → Kafka Cluster
Write Coordinator appends message to Kafka topic partition
Kafka ensures durable, ordered log with replication factor 3
3
Kafka Cluster → Replication Service
Replication Service consumes messages from Kafka
Ensures messages are replicated to multiple storage nodes
4
Replication Service → Storage Nodes
Replication Service writes data to Cassandra nodes
Writes use QUORUM consistency for durability
5
Storage Nodes → Read API Layer
Storage nodes serve read requests
Read API enforces read-after-write consistency where needed
6
Read API Layer → Client
Read API returns data to client
Client receives durable, consistent data

Practice

(1/5)
1. A social media platform must handle 1 million tweet writes per second and fan-out to millions of followers asynchronously. Which component best decouples tweet ingestion from timeline delivery to ensure durability and scalability?
easy
A. Use a relational database with immediate fan-out triggers on write
B. Use a distributed log system like Kafka partitioned by user ID
C. Use an in-memory cache like Redis for immediate fan-out
D. Use a direct synchronous RPC call to each follower's timeline service

Solution

  1. Step 1: Identify the role of the distributed log

    Kafka partitions by user ID to preserve ordering and durability, decoupling ingestion from delivery.

  2. Step 2: Eliminate other options

    Relational DB triggers (A) cause synchronous blocking and don't scale; Redis (C) lacks durability and persistence; synchronous RPC (D) blocks on follower availability.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Kafka partitions ensure durable, ordered, asynchronous fan-out [check]
Hint: Distributed log decouples ingestion and delivery [check]
Common Mistakes:
2. In a distributed lock system using ZooKeeper, at step 4 the client attempts to acquire the lock by creating an ephemeral sequential node. What happens if the client crashes immediately after creating this node but before confirming lock ownership?
easy
A. ZooKeeper automatically deletes the ephemeral node, releasing the lock
B. The ephemeral node remains until manually deleted, causing a deadlock
C. The client retains the lock until a timeout expires
D. The lock is transferred to the next waiting client immediately

Solution

  1. Step 1: Understand ephemeral node semantics

    Ephemeral nodes in ZooKeeper are automatically deleted when the client session ends (e.g., crash), preventing deadlocks (A) and stale locks.

  2. Step 2: Trace lock ownership after crash

    Since the ephemeral node is deleted, the next client waiting for the lock will be notified, but the lock is not immediately transferred (D) until they acquire it explicitly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Ephemeral node deletion on client crash -> lock released [check]
Hint: Ephemeral nodes vanish on client crash, freeing locks [check]
Common Mistakes:
3. For handling concurrent seat bookings, which architecture better suits a high-traffic ticketing system requiring low latency and strong consistency?
medium
A. Use a centralized relational database with serializable transactions to lock seats
B. Use client-side seat locking with server-side eventual consistency checks
C. Use a message queue to serialize all booking requests for each seat
D. Use a distributed cache with optimistic locking and retry logic for seat reservation

Solution

  1. Step 1: Analyze concurrency and latency requirements

    High traffic and low latency require minimizing blocking and centralized bottlenecks.

  2. Step 2: Evaluate architectures

    Centralized DB serializable transactions cause contention and latency; message queue serialization adds delay; client-side locking is unreliable. Distributed cache with optimistic locking allows concurrent attempts with retries, balancing consistency and performance.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Optimistic locking in distributed cache balances concurrency and consistency [check]
Hint: Optimistic locking + retry beats centralized locks under load [OK]
Common Mistakes:
4. For the review system, which architecture better supports real-time aggregation of review scores for millions of listings with minimal latency, given that eventual consistency is acceptable?
medium
A. Use a stream processing system like Apache Flink to aggregate reviews in real-time
B. Use a relational database with complex joins and transactions for immediate consistency
C. Use batch processing with daily MapReduce jobs to update aggregated scores
D. Use a NoSQL document store with denormalized review aggregates updated on each write

Solution

  1. Step 1: Identify real-time aggregation needs

    Millions of listings require low-latency updates; immediate consistency is not mandatory.

  2. Step 2: Evaluate architectures

    Relational DB (A) is costly and slow at scale; batch jobs (C) have high latency; NoSQL denormalization (D) risks write amplification and stale data. Stream processing (B) provides scalable, low-latency aggregation with eventual consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stream processing enables scalable real-time aggregation with eventual consistency [check]
Hint: Stream processing fits real-time aggregation with eventual consistency [check]
Common Mistakes:
5. For the For-You feed algorithm, which architecture best balances freshness of recommendations with scalability when serving millions of users?
medium
A. Use a monolithic service that recomputes recommendations synchronously on every user request
B. Use a client-side recommendation engine to reduce server load
C. Use an offline batch job to precompute recommendations periodically and serve from cache
D. Use a real-time streaming pipeline to update recommendations instantly per user action

Solution

  1. Step 1: Evaluate synchronous recomputation

    Monolithic synchronous (A) causes high latency and poor scalability.

  2. Step 2: Compare batch vs streaming

    Streaming (C) offers freshness but is complex and costly at scale; client-side (D) lacks data for quality recommendations.

  3. Step 3: Select batch precompute

    Batch jobs (B) balance freshness and scalability by precomputing and caching recommendations.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Batch precompute caches recommendations for scalable freshness [check]
Hint: Batch precompute balances freshness and scale best [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard