Bird
Raised Fist0

Reliability and Fault Tolerance - Designing for Failure Not If

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Reliability and Fault Tolerance - Designing for Failure Not If
45 min · senior
senior

You are designing a messaging platform for a startup with 50 million daily active users (DAU). The system must guarantee message delivery even under partial failures, network partitions, and server crashes. Users expect end-to-end encryption, offline message delivery, and a P99 latency under 200ms for message send and receive operations. The system should gracefully handle failures without data loss or duplication and scale linearly with user growth.

Constraints
  • End-to-end encryption mandatory for all messages at rest and in transit
  • Offline message delivery with guaranteed eventual consistency
  • P99 latency for message send and receive under 200ms
  • System must tolerate partial data center outages without user impact
  • Support 50M DAU with peak message throughput of 1M messages/sec
📋
Guided Steps
1/5
Step 1
Choosing the Messaging Backbone
Which messaging infrastructure will you use to ensure ordered, durable, and scalable message delivery?
💡 Hint
Consider partitioning keys that guarantee ordering and linear scalability under 1M msg/sec.
Key Decision
Use Apache Kafka with conversation_id as the partition key to ensure all messages in a conversation are ordered and processed by the same consumer instance.
Interviewer Expectation
Mid: Names Kafka for durability and ordering. Senior: Explains why conversation_id is chosen over sender_id to avoid out-of-order delivery and enable efficient cursor-based sync.
⚠️Common Wrong Answer
Using RabbitMQ without partitioning leads to bottlenecks and unordered message delivery at scale.
Step 2
Data Storage for Offline and Durable Delivery
What storage system will you use to persist messages for offline delivery and fault tolerance?
💡 Hint
Look for a storage system optimized for high write throughput and efficient range queries for message sync.
Key Decision
Use Apache Cassandra with partition_key=conversation_id to enable linear scalable writes and efficient range scans for cursor-based offline sync.
Interviewer Expectation
Mid: Suggests Cassandra for scalability. Senior: Details how partitioning by conversation_id enables O(1) range scans and why relational DBs fail at this scale.
⚠️Common Wrong Answer
Using a relational database causes write bottlenecks and complex joins, failing to meet 1M msg/sec throughput.
Step 3
Ensuring End-to-End Encryption and Key Management
How will you implement end-to-end encryption while enabling offline delivery and fault tolerance?
💡 Hint
Consider client-side encryption with secure key exchange and server-side storage of ciphertext only.
Key Decision
Implement client-side encryption using libsodium with keys exchanged via a secure key management service like HashiCorp Vault, storing only ciphertext in Kafka and Cassandra.
Interviewer Expectation
Mid: Mentions client-side encryption. Senior: Explains key rotation, secure key storage, and how encryption integrates with offline delivery without server access to plaintext.
⚠️Common Wrong Answer
Encrypting only in transit (TLS) exposes plaintext at rest, violating security constraints.
Step 4
Designing for Failure and Recovery
How will your system detect failures and recover without data loss or duplication?
💡 Hint
Think about heartbeat mechanisms, idempotent writes, and consumer offset management.
Key Decision
Use Redis with TTL-based heartbeats to track consumer liveness, Kafka’s exactly-once semantics with idempotent producers, and consumer offset commits to enable safe retries and recovery.
Interviewer Expectation
Mid: Describes heartbeat and retries. Senior: Details failure detection sequence, how Redis TTL expiry triggers failover, and how Kafka’s idempotency prevents duplicates.
⚠️Common Wrong Answer
Relying on manual retries without idempotency causes message duplication and inconsistent state.
Step 5
API Design for Reliable Messaging
Design 3 core API endpoints that support sending encrypted messages, syncing offline messages, and acknowledging receipt.
💡 Hint
Consider request/response patterns that minimize latency and support exactly-once delivery semantics.
Key Decision
POST /messages/send for encrypted message submission with immediate ack, GET /messages/sync with cursor-based pagination for offline sync, and POST /messages/ack to confirm receipt and enable exactly-once delivery.
Interviewer Expectation
Mid: Names endpoints and basic functionality. Senior: Explains how each endpoint supports reliability, encryption, and offline delivery with minimal latency.
⚠️Common Wrong Answer
Using HTTP polling for sync causes excessive load and latency spikes at 50M DAU.
🔌
API Design Challenge
Sketch 3 core API endpoints to support reliable, encrypted messaging with offline delivery.
Hints
  • How does the client send an encrypted message and receive acknowledgement?
  • How does the client synchronize missed messages after offline periods?
  • How does the recipient acknowledge message receipt to ensure exactly-once delivery?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with conversation_id partition key for ordered, scalable messaging
  • Use Cassandra with conversation_id partition key for durable offline storage
  • Implement client-side encryption with secure key management
  • Design APIs supporting send, sync, and ack with reliability guarantees
Good to Have - 30%
  • Use Redis TTL heartbeats for failure detection and failover
  • Leverage Kafka exactly-once semantics and idempotent producers
  • Explain failure recovery sequence step-by-step
  • Optimize API latency to meet P99 < 200ms
Bonus - 10%
  • Discuss multi-region data center failover strategies
  • Include metrics and monitoring for reliability and latency
Anti-patterns to Avoid
HTTP polling at 50M DAU causes 50 million reconnects per minute just to check for messages, collapsing the system under load
Using relational databases for message storage leads to write bottlenecks and poor scalability, failing to meet throughput requirements
Encrypting only in transit exposes plaintext at rest, violating security requirements and risking data breaches
Ignoring idempotency causes message duplication and inconsistent user state, degrading user experience
No failure detection mechanism leads to prolonged outages and data loss, impacting reliability
Using sender_id as Kafka partition key causes out-of-order delivery and inconsistent synchronization, breaking message ordering guarantees
✅
Model Answer
▼ Reveal

The system uses Apache Kafka partitioned by conversation_id for ordered, scalable messaging, Apache Cassandra for durable offline storage, and client-side encryption with libsodium and Vault for security. Redis heartbeats detect consumer failures enabling fast failover. Kafka's exactly-once semantics with idempotent producers prevent message duplication. APIs use cursor-based pagination for efficient offline sync and support low latency with POST /messages/send, GET /messages/sync, and POST /messages/ack endpoints.

Key Decisions
  • Partition Kafka topics by conversation_id to guarantee message ordering and scalability
  • Use Cassandra partitioned by conversation_id for high write throughput and efficient range queries
  • Implement client-side encryption with secure key exchange and store only ciphertext on servers
  • Use Redis TTL heartbeats to detect consumer failures and trigger failover quickly
  • Leverage Kafka exactly-once semantics and idempotent producers to avoid duplicates
  • Design APIs with cursor-based pagination and immediate acknowledgements to meet P99 latency < 200ms
  • Apache Kafka with conversation_id partition key ensures ordered, scalable message streaming
  • Apache Cassandra with conversation_id partition key enables high write throughput and efficient offline sync
  • Client-side encryption with libsodium and HashiCorp Vault for secure key management and end-to-end encryption
  • Redis TTL-based heartbeat mechanism detects consumer failures enabling fast failover
  • Kafka exactly-once semantics and idempotent producers prevent message duplication during retries
  • Cursor-based pagination in APIs supports efficient offline message synchronization
  • POST /messages/send, GET /messages/sync, POST /messages/ack endpoints designed for low latency and reliability
  • TLS for in-transit encryption complements end-to-end encryption for defense in depth

Practice

(1/5)
1. A routing system must quickly update estimated travel times on roads based on live traffic data arriving at high velocity. Which component is best suited to handle this real-time data ingestion and update?
easy
A. Use a distributed stream processing system like Apache Kafka with real-time consumers
B. Use a local in-memory cache updated manually by operators
C. Use a REST API polling service to fetch traffic data every 5 minutes
D. Use a relational database with batch updates every 10 minutes

Solution

  1. Step 1: Identify real-time ingestion needs

    Relational DB with batch updates (A) is too slow for live traffic; polling (C) adds latency and overhead; manual updates (D) are not scalable or real-time.

  2. Step 2: Evaluate streaming system suitability

    Kafka (B) supports high-throughput, low-latency ingestion and real-time processing, ideal for live traffic updates.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka streams handle high-velocity data ingestion with low latency [check]
Hint: Stream processing fits real-time high-velocity data ingestion [check]
Common Mistakes:
2. In the request flow for posting a new answer, at step 5 the Feed Generator consumes messages from Kafka. What happens if the Feed Generator detects a malformed message at this step?
easy
A. The Feed Generator sends the malformed message to a dead-letter queue for later inspection
B. The Feed Generator pauses consumption and retries the malformed message until fixed
C. The Feed Generator discards the message and continues processing the next message
D. The Feed Generator crashes, causing the entire feed update pipeline to stop

Solution

  1. Step 1: Understand Feed Generator error handling

    Discarding (A) loses data silently; pausing and retrying (B) blocks the pipeline indefinitely; crashing (D) causes system downtime.

  2. Step 2: Dead-letter queue usage

    Sending malformed messages to a dead-letter queue (C) isolates bad data for manual or automated inspection without blocking the main pipeline.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dead-letter queue isolates bad messages without blocking processing [check]
Hint: Dead-letter queues isolate bad messages, keep pipeline flowing [check]
Common Mistakes:
3. For a mobile chat app requiring low latency and offline support, which architecture is best suited to handle message synchronization and delivery?
medium
A. Use WebSockets with server push and no offline caching
B. Use gRPC streaming for real-time message delivery to mobile clients
C. Use REST APIs with client-side local storage and background sync
D. Use polling every few seconds to fetch new messages

Solution

  1. Step 1: Analyze mobile client constraints

    Mobile clients often have intermittent connectivity and limited support for persistent streaming (gRPC streaming in Use gRPC streaming for real-time message delivery to mobile clients is unreliable on mobile). WebSockets without offline caching (Use WebSockets with server push and no offline caching) fail offline support. Polling (Use polling every few seconds to fetch new messages) adds latency and battery drain.

  2. Step 2: Evaluate REST with local storage

    REST APIs combined with client-side local storage and background sync enable offline message access and eventual consistency with low latency.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    REST + local storage -> offline support and sync [check]
Hint: Offline support needs local storage, not persistent streaming [check]
Common Mistakes:
4. For a stock exchange system requiring low-latency order matching and high availability, which architecture is best suited to handle failover without losing in-flight orders?
medium
A. Use a distributed matching engine with partitioned order books per stock symbol and asynchronous replication
B. Use a single centralized matching engine with synchronous replication to a hot standby
C. Use multiple independent matching engines with eventual consistency and conflict resolution
D. Use a cloud-managed database with built-in failover and no dedicated matching engine

Solution

  1. Step 1: Analyze architecture requirements

    Low latency and high availability require partitioning to avoid bottlenecks and asynchronous replication to avoid blocking on failover.

  2. Step 2: Evaluate options

    Use a single centralized matching engine with synchronous replication to a hot standby blocks on synchronous replication causing latency; Use multiple independent matching engines with eventual consistency and conflict resolution risks conflicts and inconsistent order books; Use a cloud-managed database with built-in failover and no dedicated matching engine lacks dedicated matching engine causing latency and complexity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Partitioned matching engine + async replication -> low latency + availability [check]
Hint: Partition and async replication for low latency failover [check]
Common Mistakes:
5. For managing user playlists with frequent updates and offline sync, which storage architecture is best suited given the need for eventual consistency and offline support?
medium
A. Conflict-free replicated data type (CRDT) based distributed store
B. Centralized NoSQL store with eventual consistency but no offline sync
C. Strongly consistent relational database with synchronous replication
D. Local device-only storage with manual sync to server

Solution

  1. Step 1: Identify requirements

    Playlists require offline edits and eventual sync with conflict resolution.

  2. Step 2: Evaluate architectures

    Relational DB (A) blocks offline edits; centralized NoSQL (C) lacks offline sync; local-only (D) risks data loss and complex merges. CRDTs (B) enable conflict-free merges and offline edits.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    CRDTs enable offline edits with automatic conflict resolution [check]
Hint: CRDTs handle offline sync and conflicts automatically [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard