Bird
Raised Fist0

Reliability and Fault Tolerance - Designing for Failure Not If

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
API Gateway

Receives client request and validates payload

API Gateway accepts HTTP request and prepares event for Kafka

Protocol:HTTP/REST
BeforeClient request is pending delivery to backend
AfterRequest is validated and ready to be published to Kafka
Kafka Cluster

Appends event to partitioned log for durability

Kafka stores event with partition key for ordering guarantees

Protocol:Kafka
BeforeEvent is ready to be persisted in Kafka
AfterEvent durably committed in Kafka; downstream consumers notified
Cassandra DB

Persists event data for long-term storage and querying

Cassandra writes event data with tunable consistency

Protocol:CQL
BeforeEvent is available in Kafka for consumption
AfterEvent data stored in Cassandra; queryable for analytics
Monitoring & Alerting Service

Collects metrics and event logs for health monitoring

Service receives Kafka and Cassandra metrics to detect anomalies

Protocol:Metrics/Events
BeforeSystem components operate normally with event flow
AfterMonitoring detects system health and triggers alerts if needed
Failover Controller

Monitors alerts and initiates failover on detected failures

Controller switches traffic or restarts components upon failure

Protocol:Metrics/Events
BeforeMonitoring alerts indicate degraded or failed components
AfterFailover Controller activates recovery to maintain availability
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────┐
│  API Gateway  │
└───────┬───────┘
        │ HTTP/REST
        ▼
┌───────────────┐
│ Kafka Cluster │
└───────┬───────┘
        │ Kafka
        ▼
┌───────────────┐
│ Cassandra DB  │
└───────┬───────┘
        │ CQL
        ▼
┌───────────────────────────────┐
│ Monitoring & Alerting Service  │
└───────────────┬───────────────┘
                │ Metrics/Events
                ▼
       ┌───────────────────┐
       │ Failover Controller│
       └───────────────────┘
Design Rationale

This architecture ensures reliability by decoupling request handling and storage. The API Gateway validates and forwards requests to Kafka, which provides durable, ordered event streaming. Cassandra DB persists data for fault-tolerant storage. The Monitoring & Alerting Service continuously collects metrics to detect failures early. Upon detecting issues, the Failover Controller triggers failover mechanisms to maintain system availability. This design embraces failure as inevitable, enabling graceful recovery and minimizing downtime.

Primary Focus

✓

Practice

(1/5)
1. A global web service must route user requests to the nearest healthy data center with minimal latency and automatic failover. Which component is best suited to handle this requirement?
easy
A. Deploy a Global Server Load Balancer (GSLB) that monitors data center health and routes accordingly
B. Configure client-side logic to select the closest data center IP address
C. Use DNS-based load balancing with low TTL values to direct traffic
D. Rely on BGP routing policies to steer traffic to preferred data centers

Solution

  1. Step 1: Identify component for global health-aware routing

    DNS with low TTL (Use DNS-based load balancing with low TTL values to direct traffic) can cause caching issues and slow failover; client-side logic (Configure client-side logic to select the closest data center IP address) is unreliable and hard to maintain; BGP routing (Rely on BGP routing policies to steer traffic to preferred data centers) is coarse-grained and slow to converge.

  2. Step 2: Understand GSLB capabilities

    GSLB actively monitors data center health and latency, dynamically routing users to the best endpoint, providing low latency and automatic failover.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    GSLB monitors health and routes globally with failover [check]
Hint: GSLB actively monitors and routes globally [check]
Common Mistakes:
2. For the review system, which architecture better supports real-time aggregation of review scores for millions of listings with minimal latency, given that eventual consistency is acceptable?
medium
A. Use a stream processing system like Apache Flink to aggregate reviews in real-time
B. Use a relational database with complex joins and transactions for immediate consistency
C. Use batch processing with daily MapReduce jobs to update aggregated scores
D. Use a NoSQL document store with denormalized review aggregates updated on each write

Solution

  1. Step 1: Identify real-time aggregation needs

    Millions of listings require low-latency updates; immediate consistency is not mandatory.

  2. Step 2: Evaluate architectures

    Relational DB (A) is costly and slow at scale; batch jobs (C) have high latency; NoSQL denormalization (D) risks write amplification and stale data. Stream processing (B) provides scalable, low-latency aggregation with eventual consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stream processing enables scalable real-time aggregation with eventual consistency [check]
Hint: Stream processing fits real-time aggregation with eventual consistency [check]
Common Mistakes:
3. If the gRPC server's message queue is removed, what exactly breaks in the message delivery flow?
medium
A. Clients receive duplicate messages due to lack of deduplication
B. The server blocks sending until the client acknowledges, causing backpressure
C. Messages are lost if the client is slow or disconnected, with no retry mechanism
D. The server switches to synchronous unary calls, increasing latency

Solution

  1. Step 1: Role of message queue in gRPC server

    The queue buffers messages for slow or disconnected clients, enabling retries and flow control.

  2. Step 2: Effect of removing the queue

    Without the queue, messages sent to slow/disconnected clients are dropped immediately, causing message loss without retries.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    No queue -> message loss on slow/disconnected clients [check]
Hint: No queue -> messages lost if client slow or disconnected [check]
Common Mistakes:
4. For a globally distributed database that must minimize read latency during network partitions, which architecture is the best trade-off according to PACELC?
medium
A. Use synchronous replication with strong consistency guarantees
B. Use a quorum-based protocol for both reads and writes
C. Use a single primary data center with read-only replicas elsewhere
D. Use eventual consistency with asynchronous replication and local reads

Solution

  1. Step 1: Analyze PACELC trade-offs for read latency during partitions

    Minimizing read latency favors local reads and asynchronous replication despite weaker consistency.

  2. Step 2: Evaluate architectures

    Synchronous replication (A) and quorum (D) increase latency; single primary (C) risks unavailability if primary is unreachable.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Eventual consistency + async replication -> low read latency under partition [check]
Hint: Eventual consistency beats sync replication for latency under partition [check]
Common Mistakes:
5. If the TCP retransmission timer is disabled in a file transfer system, what failure occurs when a packet is lost?
medium
A. The sender immediately switches to UDP to resend the packet
B. The sender never retransmits the lost packet, causing the transfer to stall indefinitely
C. The receiver requests retransmission via duplicate ACKs, triggering recovery
D. The connection resets automatically to recover from loss

Solution

  1. Step 1: Understand TCP retransmission timer role

    The retransmission timer triggers resending lost packets when ACKs are not received.

  2. Step 2: Trace failure without timer

    Without the timer, lost packets are never retransmitted, causing the sender to wait indefinitely and stall the transfer.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    No retransmission timer -> stalled transfer on loss [check]
Hint: No retransmission timer -> lost packets never resent [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard