Bird
Raised Fist0

Reliability and Fault Tolerance - Designing for Failure Not If

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Reliability and Fault Tolerance - Designing for Failure Not If
Design systems assuming failure is inevitable; architect for graceful degradation and rapid recovery.
mediumNon-Functional Requirements FrameworkFreq: high
⚙️
Functional Requirements
  • System must detect and isolate component failures automatically without human intervention.
  • Support graceful degradation of service during partial outages to maintain core functionality.
  • Provide automatic failover and recovery mechanisms for critical components.
  • Maintain data consistency and durability despite node or network failures.
  • Emit detailed monitoring and alerting signals for failure detection and diagnosis.
  • Support rolling upgrades and configuration changes without downtime.
📊
Non-Functional Requirements
  • System uptime SLA of 99.99% (approx 52.56 minutes downtime per month).
  • Recovery time objective (RTO) under 1 minute for critical failures.
  • Recovery point objective (RPO) of zero data loss for writes acknowledged to clients.
  • Support 2 billion daily active users generating 40 messages per user per day.
  • End-to-end request latency P99 under 100ms even during partial failures.
❓
Clarifying Questions to Ask
  • What are the critical components whose failure impacts system availability the most?
  • What failure modes are most common or most impactful (hardware, network, software bugs)?
  • Is eventual consistency acceptable in some parts of the system or is strong consistency required everywhere?
  • What is the expected failure detection latency and alerting mechanism?
  • Are there existing SLAs or compliance requirements that dictate recovery objectives?
  • What is the expected scale in terms of users, requests per second, and data volume?
Out of Scope
✗ Designing the core business logic or feature set of the application.
✗ Security and compliance requirements beyond basic fault tolerance.
✗ User interface or client-side error handling.
✗ Cost optimization or infrastructure budgeting.
📌
Assumptions
  • 2 billion daily active users (DAU).
  • Each user generates 40 messages per day on average.
  • Average message size is 100 bytes.
  • Data retention period is 30 days.
  • Replication factor for durability is 3.
⚡
QPS Calculation
write qps2,000,000,000 users × 40 msgs/user/day ÷ 86,400 seconds/day ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 = 1,851,852 writes/sec
read qpsAssuming 3 reads per write (for fanout, queries): 925,926 × 40 × 3 ÷ 86,400 ≈ 3,703,703 reads/sec
💾
Storage Estimate
daily storage2B × 40 × 100 bytes = 8,000,000,000,000 bytes = 8 TB/day
retention storage8 TB/day × 30 days = 240 TB
with replication240 TB × 3 replicas = 720 TB total storage
🖥️
Server Estimate
write pathKafka cluster with 100 brokers (each handles ~20K writes/sec) to handle 1.85M writes/sec peak
storageCassandra cluster with 200 nodes (each node ~3.6 TB data) to store 720 TB replicated data
🏗️
Architecture Diagram
Architecture Diagram
          +-------------+
          |  Clients    |
          +------+------+
                 |
          +------+-------+
          |  API Gateway |
          +------+-------+
                 |
          +------+-------+
          |  Kafka Cluster|
          +------+-------+
                 |
          +------+-------+
          | Cassandra DB |
          +------+-------+
                 |
          +------+-------+
          | Monitoring & |
          | Alerting Svc |
          +--------------+
🧩
Components
API Gateway
Entry point for client requests; performs load balancing and request validation.
Why thisEnvoy proxy supports circuit breaking and retries natively, enabling graceful degradation under failure.
Why notNginx lacks built-in circuit breaking and advanced health checks needed for fault tolerance at 1.8M QPS.
Kafka Cluster
Durable, distributed write-ahead log for ingesting messages with replication and partitioning.
Why thisKafka provides strong durability guarantees with configurable replication and supports high throughput (millions of writes/sec).
Why notRabbitMQ does not scale linearly beyond hundreds of thousands of messages/sec and lacks partitioned logs.
Cassandra DB
Distributed NoSQL database for storing messages with tunable consistency and multi-datacenter replication.
Why thisCassandra’s masterless architecture avoids single points of failure and supports linear horizontal scaling to hundreds of nodes.
Why notPostgreSQL requires manual sharding and has write bottlenecks at >900K writes/sec.
Monitoring & Alerting Service
Collects metrics and logs from all components to detect failures and trigger alerts.
Why thisPrometheus with Alertmanager supports multi-dimensional metrics and alerting rules with sub-minute detection latency.
Why notNagios is less scalable and slower in detecting transient failures at this scale.
Failover Controller
Automates detection of node failures and triggers failover or recovery workflows.
Why thisKubernetes operators provide declarative failure detection and automated pod/node recovery with minimal downtime.
Why notManual failover processes are error-prone and too slow for 1-minute RTO requirements.
🔄
Data Flow
1
Clients → API Gateway
Send message write/read requests
API Gateway performs validation and rate limiting
2
API Gateway → Kafka Cluster
Publish messages to Kafka topics
Kafka partitions messages by user ID for load distribution
3
Kafka Cluster → Cassandra DB
Consume messages and persist to Cassandra
Consumer groups ensure exactly-once processing with idempotency
4
Clients → API Gateway
Read requests query Cassandra
Reads use local consistency level to balance latency and durability
5
All Components → Monitoring & Alerting Service
Emit health and performance metrics
Alerts trigger Failover Controller on anomalies
6
Failover Controller → Kafka/Cassandra/K8s
Initiate failover or restart failed nodes
Automated recovery minimizes downtime and data loss

Practice

(1/5)
1. In a distributed lock system using ZooKeeper, at step 4 the client attempts to acquire the lock by creating an ephemeral sequential node. What happens if the client crashes immediately after creating this node but before confirming lock ownership?
easy
A. ZooKeeper automatically deletes the ephemeral node, releasing the lock
B. The ephemeral node remains until manually deleted, causing a deadlock
C. The client retains the lock until a timeout expires
D. The lock is transferred to the next waiting client immediately

Solution

  1. Step 1: Understand ephemeral node semantics

    Ephemeral nodes in ZooKeeper are automatically deleted when the client session ends (e.g., crash), preventing deadlocks (A) and stale locks.

  2. Step 2: Trace lock ownership after crash

    Since the ephemeral node is deleted, the next client waiting for the lock will be notified, but the lock is not immediately transferred (D) until they acquire it explicitly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Ephemeral node deletion on client crash -> lock released [check]
Hint: Ephemeral nodes vanish on client crash, freeing locks [check]
Common Mistakes:
2. At 10x scale, which component is most likely to become the bottleneck first in the typeahead system, and why?
hard
A. The front-end API servers, because they cannot handle the increased number of concurrent connections
B. The personalization data store, because user profile reads increase linearly and cannot be sharded easily
C. The ranking service, because ranking computations grow exponentially with candidate suggestions
D. The Trie cache servers, because maintaining low-latency prefix lookups requires large in-memory state that is hard to shard further

Solution

  1. Step 1: Analyze component scalability

    Personalization stores can be sharded; ranking computations scale linearly; front-end servers can be horizontally scaled.

  2. Step 2: Identify Trie cache bottleneck

    Trie caches hold large in-memory prefix trees; sharding by prefix has limits due to prefix overlap and memory footprint, causing latency and memory bottlenecks.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Trie cache memory and shard limits -> first bottleneck at 10x scale [check]
Hint: Trie cache memory limits cause first bottleneck at scale [check]
Common Mistakes:
3. At 10x scale, which component in a consistent hashing distributed cache system is most likely to fail first, and why?
hard
A. Consistent hashing ring metadata service, because it becomes a bottleneck updating and distributing ring changes rapidly
B. Cache nodes, because their memory capacity limits are exceeded by increased key volume
C. Client-side hashing logic, because it cannot handle the increased number of nodes efficiently
D. Network switches, because increased traffic saturates bandwidth causing packet loss

Solution

  1. Step 1: Identify scaling bottlenecks

    Cache nodes scale horizontally; memory limits can be increased. Client hashing logic is local and scales well. Network switches scale with infrastructure upgrades.

  2. Step 2: Consistent hashing ring metadata bottleneck

    The ring metadata service must update and distribute node membership changes; at 10x scale, frequent updates cause high coordination overhead and latency, becoming the first failure point.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Metadata service update overhead limits scaling first [check]
Hint: Metadata service update overhead limits scaling first [check]
Common Mistakes:
4. At 10x scale, which component in an HTTP/3 deployment is most likely to become the bottleneck first and why?
hard
A. WebSocket servers because HTTP/3 does not support WebSockets natively
B. TLS handshake CPU load on servers due to QUIC's per-connection encryption overhead
C. UDP packet loss rate increases causing retransmissions and latency spikes
D. TCP connection limits because HTTP/3 uses TCP under the hood

Solution

  1. Step 1: Identify HTTP/3 transport and encryption

    HTTP/3 uses QUIC over UDP with built-in TLS encryption per connection, increasing CPU load for handshake and encryption at scale.

  2. Step 2: Eliminate other bottlenecks

    UDP packet loss is network-level, not server bottleneck; WebSockets are not natively supported by HTTP/3 but unrelated to scaling HTTP/3 itself; HTTP/3 does not use TCP.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    QUIC's per-connection TLS handshake CPU load limits HTTP/3 scale [check]
Hint: QUIC TLS handshake CPU load limits HTTP/3 scale first [check]
Common Mistakes:
5. At 10x traffic scale, which load balancer component is most likely to become the bottleneck first and why?
hard
A. The IP Hash function, because hashing client IPs becomes computationally expensive at scale
B. The weighted server list, because updating weights dynamically causes synchronization delays
C. The centralized connection tracker, because Least Connections requires global state updates for every connection change
D. The Round Robin pointer, because cycling through servers sequentially cannot handle high concurrency

Solution

  1. Step 1: Identify statefulness in load balancing algorithms

    Least Connections requires tracking active connections per server globally, needing frequent updates.

  2. Step 2: Analyze scaling impact

    At 10x scale, the centralized connection tracker becomes a bottleneck due to synchronization overhead and contention.

  3. Step 3: Evaluate other components

    IP Hash is stateless and fast; weighted lists update less frequently; Round Robin pointer is lightweight and easily sharded.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Global connection tracking bottlenecks Least Connections at high scale [check]
Hint: Global connection tracking bottlenecks Least Connections at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard