Bird
Raised Fist0

Durability - How Systems Guarantee Data is Never Lost

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Durability - How Systems Guarantee Data is Never Lost
45 min · senior
senior

You are designing a durability subsystem for a 50M DAU messaging platform that guarantees no user data is ever lost, even under failures. The system must handle 50 million daily active users generating 10 billion messages per day. Business requires end-to-end encryption at rest and in transit, offline message delivery, and a P99 end-to-end latency under 200ms for message persistence confirmation.

Constraints
  • End-to-end encryption mandatory for data at rest and in transit
  • Offline message delivery with guaranteed eventual consistency
  • P99 write latency under 200ms for message durability confirmation
  • System must tolerate node failures without data loss
  • Support 10 billion messages per day (~115,740 msg/sec average)
  • Data retention for 30 days with efficient compaction
📋
Guided Steps
1/6
Step 1
Choosing the Write Path for Durability
How will you design the write path to guarantee no data loss while meeting P99 < 200ms latency?
💡 Hint
Consider a distributed log system with replication and durability guarantees.
Key Decision
Use Apache Kafka with topic partitioning keyed by conversation_id to ensure ordered, replicated writes with configurable durability (acks=all).
Interviewer Expectation
Mid: Names Kafka for durability. Senior: Explains conversation_id partition key ensures ordering and localizes writes to a partition, enabling efficient replication and recovery.
⚠️Common Wrong Answer
Using a traditional RDBMS for writes causes high latency and poor horizontal scalability at 10B messages/day.
Step 2
Data Storage and Retention Strategy
What storage system will you use to persist messages long-term and support efficient range queries for offline delivery?
💡 Hint
Look for a wide-column store optimized for high write throughput and range scans.
Key Decision
Use Apache Cassandra with partition_key=conversation_id to store messages; LSM-tree design supports high write throughput and efficient cursor-based sync.
Interviewer Expectation
Mid: Suggests Cassandra for scalability. Senior: Explains why conversation_id as partition key enables O(1) range scans for client sync and why alternatives like MongoDB fail at this scale due to poor write amplification.
⚠️Common Wrong Answer
Using MongoDB leads to write bottlenecks and inefficient range queries causing high sync latency.
Step 3
Ensuring End-to-End Encryption
How will you guarantee end-to-end encryption for data at rest and in transit without compromising durability?
💡 Hint
Separate encryption keys per conversation and encrypt data before persistence.
Key Decision
Implement client-side encryption with keys managed via AWS KMS; encrypted payloads stored in Kafka and Cassandra, TLS used for in-transit encryption.
Interviewer Expectation
Mid: Mentions TLS and encryption. Senior: Details client-side encryption to prevent server-side data exposure and key management via AWS KMS for secure key rotation.
⚠️Common Wrong Answer
Encrypting only at the server side exposes data during transit and storage, violating end-to-end encryption requirements.
Step 4
Offline Message Delivery and Sync
How will clients efficiently sync missed messages after offline periods without overwhelming the system?
💡 Hint
Use cursor-based incremental sync with backpressure and batching.
Key Decision
Clients maintain last received message offset per conversation; sync API queries Cassandra with range scans using conversation_id and offset; server batches results and applies rate limiting.
Interviewer Expectation
Mid: Describes cursor-based sync. Senior: Explains how conversation_id partitioning enables efficient range scans and how batching and rate limiting prevent overload during mass reconnects.
⚠️Common Wrong Answer
Full table scans or naive polling cause system overload and high latency during client reconnect storms.
Step 5
Handling Failures Without Data Loss
How will the system handle node failures or network partitions to guarantee no data loss?
💡 Hint
Use replication with quorum writes and durable commit logs.
Key Decision
Kafka configured with replication factor ≥3 and acks=all ensures writes are committed to multiple brokers; Cassandra uses replication factor 3 with QUORUM consistency for reads/writes; automated failover and repair processes ensure durability.
Interviewer Expectation
Mid: Mentions replication. Senior: Explains how Kafka acks=all and Cassandra QUORUM consistency prevent data loss during broker/node failures and how repair mechanisms fix inconsistencies.
⚠️Common Wrong Answer
Using replication factor 1 or acks=1 risks data loss during failures.
Step 6
API Design for Durability System
Design 3 core API endpoints for message persistence, client sync, and acknowledgment.
💡 Hint
POST /messages to send and persist a message with durability confirmationGET /messages/sync to fetch missed messages using conversation_id and offsetPOST /messages/ack to acknowledge message receipt and update durability state
Key Decision
Define RESTful endpoints: POST /messages accepts encrypted payload and returns durable write confirmation; GET /messages/sync supports cursor-based incremental fetch; POST /messages/ack updates delivery status in Cassandra.
Interviewer Expectation
Mid: Names endpoints and basic functionality. Senior: Details request/response payloads, idempotency, encryption handling, and how acknowledgments update delivery metadata for offline delivery guarantees.
⚠️Common Wrong Answer
Using HTTP polling endpoints without cursor support causes excessive load and poor scalability.
🔌
API Design Challenge
Sketch 3 core API endpoints for the durability system that handle message persistence, client sync, and acknowledgment.
Hints
  • How does the client send a message and receive durability confirmation?
  • How does the client synchronize missed messages after offline periods?
  • How does the recipient acknowledge message receipt to update durability state?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with conversation_id partition key for durable, ordered writes
  • Use Cassandra with conversation_id partition key for efficient offline sync
  • Implement client-side encryption with AWS KMS for end-to-end encryption
  • Design cursor-based incremental sync API to avoid overload
  • Configure replication and consistency to prevent data loss on failures
Good to Have - 30%
  • TLS for in-transit encryption
  • Batching and rate limiting on sync API to handle reconnect storms
  • Idempotent API design for message persistence and acknowledgment
Bonus - 10%
  • Automated repair and failover mechanisms
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

This durability system design ensures zero data loss for a high-scale messaging platform by leveraging Apache Kafka for durable, ordered writes with replication, Apache Cassandra for scalable long-term storage and efficient offline sync, and client-side encryption managed via AWS KMS for end-to-end security. The system supports offline message delivery with cursor-based incremental sync APIs that include batching and rate limiting to prevent overload. Replication factors and consistency levels are configured to guarantee durability despite node failures. RESTful APIs provide idempotent endpoints for message persistence, sync, and acknowledgment.

Key Decisions
  • Use Apache Kafka with acks=all and replication factor ≥3 to ensure durable, replicated writes with ordering guaranteed by conversation_id partition key
  • Persist messages in Apache Cassandra using conversation_id as partition key to enable efficient range queries for offline sync with QUORUM consistency to prevent data loss
  • Implement client-side encryption with keys managed by AWS KMS to ensure end-to-end encryption at rest and in transit, preventing server-side data exposure
  • Use TLS for all in-transit data to secure communication between clients and servers
  • Design cursor-based incremental sync API that uses conversation_id and last received offset to fetch missed messages with batching and rate limiting to avoid system overload
  • Configure Kafka and Cassandra replication and consistency settings to tolerate node failures without data loss
  • Design RESTful APIs with idempotency and encryption handling for message persistence, sync, and acknowledgment to maintain durability guarantees
  • Apache Kafka with conversation_id as partition key for ordered, replicated durable writes
  • Apache Cassandra with conversation_id partition key for scalable, efficient range queries
  • Client-side encryption with AWS KMS-managed keys for end-to-end encryption compliance
  • TLS for in-transit encryption between clients and servers
  • Cursor-based incremental sync API to support offline message delivery without overload
  • Kafka replication factor ≥3 and acks=all to guarantee no data loss on broker failures
  • Cassandra replication factor 3 with QUORUM consistency for fault-tolerant reads/writes
  • RESTful API endpoints: POST /messages, GET /messages/sync, POST /messages/ack with idempotency and encryption

Practice

(1/5)
1. A real-time analytics system must ingest 1 million events per second and provide near-instant query results. Which component best handles the ingestion to balance latency and throughput?
easy
A. A relational database with strong consistency guarantees
B. A batch processing system that aggregates events every minute
C. A distributed log system like Apache Kafka with partitioned topics
D. A single-threaded in-memory queue to ensure ordering

Solution

  1. Step 1: Identify ingestion requirements

    High throughput and low latency ingestion require a scalable, partitioned system that can handle parallel writes.

  2. Step 2: Evaluate components

    Relational DBs (A) can't handle this write volume with low latency; single-threaded queues (C) limit throughput; batch systems (D) increase latency.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Kafka partitions enable parallel ingestion with low latency [check]
Hint: Partitioned logs scale writes and keep latency low [check]
Common Mistakes:
2. For a system supporting multiple client types (web, mobile, IoT), which architecture best balances maintainability and client-specific optimization?
medium
A. Single API Gateway forwarding requests directly to microservices with client logic inside services
B. Separate BFF layers per client type behind a common API Gateway
C. One monolithic backend serving all clients with feature flags for client differences
D. Client apps call microservices directly, bypassing API Gateway and BFF

Solution

  1. Step 1: Evaluate client-specific optimization needs

    Separate BFF layers per client type allow tailored APIs, improving UX and reducing client complexity.

  2. Step 2: Assess maintainability

    Separate BFF layers per client type behind a common API Gateway isolates client-specific logic in BFFs, keeping microservices generic and maintainable. Single API Gateway forwarding requests directly to microservices with client logic inside services mixes client logic into services, increasing coupling. One monolithic backend serving all clients with feature flags for client differences's monolith reduces flexibility and scalability. Client apps call microservices directly, bypassing API Gateway and BFF breaks encapsulation and complicates security.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Separate BFFs isolate client logic, improving maintainability and optimization [check]
Hint: Separate BFFs isolate client logic, improving maintainability [check]
Common Mistakes:
3. If the Kafka cluster responsible for video upload event streaming goes down, what exactly breaks in the upload processing pipeline?
medium
A. Users cannot upload videos because the frontend blocks until Kafka responds
B. Video metadata extraction and transcoding jobs stall, causing delayed availability of videos
C. The For-You feed algorithm stops recommending new videos immediately
D. The CDN fails to cache videos, causing playback failures

Solution

  1. Step 1: Trace Kafka's role in upload pipeline

    Kafka buffers upload events for async processing; frontend uploads don't block on Kafka (A wrong).

  2. Step 2: Identify failure impact

    Metadata extraction and transcoding (B) stall but do not cause immediate video unavailability; feed algorithm (D) depends on processed data, not Kafka directly.

  3. Step 3: Recognize CDN caching dependency

    Without Kafka, processed videos don't reach CDN timely, causing cache misses and playback failures (C).

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Kafka outage delays CDN caching, causing playback failures [check]
Hint: Kafka outage stalls processing, causing CDN cache misses [OK]
Common Mistakes:
4. For internal service communication requiring bidirectional streaming with mobile clients that have intermittent connectivity, which approach is best?
medium
A. Use gRPC bidirectional streaming directly for real-time updates
B. Use gRPC unary calls with client-side polling and exponential backoff
C. Use WebSockets with JSON payloads for flexible streaming
D. Use REST long polling to simulate streaming behavior

Solution

  1. Step 1: Analyze client connectivity constraints

    Mobile clients often have intermittent connectivity and poor support for HTTP/2 streaming.

  2. Step 2: Evaluate streaming options

    Direct gRPC streaming (A) is unreliable on mobile. WebSockets (C) add complexity and JSON is less efficient. REST long polling (D) is less efficient and higher latency. Unary calls with polling and backoff (B) handle intermittent connectivity gracefully.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Mobile intermittent -> unary calls + polling with backoff [check]
Hint: Mobile clients -> unary calls + polling, not streaming [check]
Common Mistakes:
5. For a real-time multiplayer game backend requiring low latency and consistent player state, which architecture is more appropriate and why?
medium
A. Stateful servers storing player state only in a distributed cache without persistence
B. Stateless microservices with external database for player state
C. Stateless servers with client-side state management and server validation
D. Stateful game servers maintaining player state in memory with periodic persistence

Solution

  1. Step 1: Analyze latency and consistency needs

    Real-time games require fast access to player state with strong consistency.

  2. Step 2: Evaluate architecture trade-offs

    Stateful servers with in-memory state reduce latency and allow immediate state updates; periodic persistence balances durability.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stateful in-memory state + persistence -> low latency, consistent [check]
Hint: In-memory stateful servers best for real-time consistency [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard