Bird
Raised Fist0

Circuit Breaker Pattern - Preventing Cascade Failures Across Services

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
Circuit Breaker Proxy

Receives client request and queries circuit state

Checks if circuit is open or closed before forwarding request

Protocol:HTTP
BeforeRequest arrives; circuit state unknown to proxy
AfterCircuit state fetched from State Store; decision pending
Circuit Breaker State Store

Returns current circuit state for the requested service

Reads state from Redis with TTL to ensure freshness

Protocol:Redis GET
BeforeCircuit state stored with last update timestamp
AfterCircuit state delivered to proxy (open/closed/half-open)
Circuit Breaker Proxy

Decides to allow or block the request based on state

If circuit is open, blocks request; else forwards downstream

Protocol:HTTP Forward or Block
BeforeCircuit state known; request pending decision
AfterRequest either forwarded or blocked with error response
Metrics & Monitoring

Receives metrics about request outcome and circuit state

Publishes metrics to Kafka for aggregation and alerting

Protocol:Kafka Publish
BeforeMetrics buffer empty for this request
AfterMetrics recorded including success/failure and state
State Propagation Service

Propagates updated circuit state to all cluster nodes

Sends internal RPC to synchronize circuit states

Protocol:Internal RPC
BeforeCircuit state updated locally on one node
AfterAll nodes have consistent circuit state
Health Check & Recovery Mgr

Performs health checks and triggers circuit recovery

If service is healthy, closes circuit to allow requests

Protocol:Internal RPC
BeforeCircuit is open due to previous failures
AfterCircuit transitions to half-open or closed after checks
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│   Circuit Breaker Proxy       │
└───────────────┬───────────────┘
                │ HTTP Request
                ▼
┌───────────────────────────────┐
│ Circuit Breaker State Store    │
└───────────────┬───────────────┘
                │ Redis GET/SET
                ▼
┌───────────────────────────────┐
│  Metrics & Monitoring          │
└───────────────┬───────────────┘
                │ Kafka Publish
                ▼
┌───────────────────────────────┐
│ State Propagation Service      │
└───────────────┬───────────────┘
                │ Internal RPC
                ▼
┌───────────────────────────────┐
│ Health Check & Recovery Mgr    │
└───────────────────────────────┘
Design Rationale

The Circuit Breaker Proxy intercepts incoming requests and queries the Circuit Breaker State Store to determine if the circuit is open, closed, or half-open. This state is stored with TTL in Redis to ensure freshness. Based on the state, the proxy either forwards the request or blocks it to prevent cascade failures. Metrics about request outcomes and circuit states are published to the Metrics & Monitoring component via Kafka for alerting and analysis. The State Propagation Service synchronizes circuit states across cluster nodes using internal RPC calls, ensuring consistency. Meanwhile, the Health Check & Recovery Manager performs periodic health checks on downstream services and triggers circuit recovery by transitioning states appropriately. This architecture prevents cascading failures by isolating failing services and enabling controlled recovery.

Primary Focus

✓

Practice

(1/5)
1. A distributed messaging system must handle 1 million writes per second with guaranteed message durability and ordered delivery per topic partition. Which component is best suited to buffer and persist these writes before delivery?
easy
A. Use an in-memory Redis queue for fast writes but no persistence guarantees
B. Use a simple file system append-only log on a single server
C. Use a relational database with row-level locking for message storage
D. Use a Kafka cluster with partitioned logs and replication for durability and ordering

Solution

  1. Step 1: Identify buffering and durability needs

    Redis (A) is fast but lacks durability guarantees; relational DB (C) cannot handle 1M writes/sec efficiently due to locking and transaction overhead; file system log (D) on a single server is a single point of failure and bottleneck.

  2. Step 2: Kafka's partitioned log with replication

    Kafka (B) partitions data for parallelism, replicates for durability, and guarantees ordering within partitions, making it ideal for high-throughput durable buffering.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka partitions + replication -> high throughput + durability [check]
Hint: Partitioned logs with replication ensure durability and ordering [check]
Common Mistakes:
2. In the request flow of a global load balancing system, at Step 3 the GSLB receives a DNS query for a user in Europe. What does the GSLB do at this step?
easy
A. Returns the IP of the closest data center without checking health status
B. Checks data center health and latency metrics before returning the IP address
C. Forwards the DNS query to the local data center's DNS server for resolution
D. Caches the previous DNS response indefinitely to reduce load

Solution

  1. Step 1: Understand GSLB behavior at DNS query

    GSLB does not blindly return IPs (Returns the IP of the closest data center without checking health status) or forward queries (Forwards the DNS query to the local data center's DNS server for resolution). It actively checks health and latency to pick the best endpoint.

  2. Step 2: Evaluate caching behavior

    Indefinite caching (Caches the previous DNS response indefinitely to reduce load) would prevent dynamic failover and load balancing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    GSLB checks health and latency before DNS response [check]
Hint: GSLB checks health before DNS response [check]
Common Mistakes:
3. If the message queue responsible for decoupling job application submissions from downstream processing crashes, what is the immediate failure observed in the system?
medium
A. Applications are lost silently with no retry or alert
B. The system automatically switches to a backup queue with no impact
C. Application submission API blocks until the queue recovers, causing client timeouts
D. Applications are accepted but downstream components never receive them, causing processing delays

Solution

  1. Step 1: Understand queue role in decoupling

    The queue buffers submissions so downstream processing is asynchronous.

  2. Step 2: Trace failure impact

    If queue crashes, submissions accepted by API but not delivered downstream, causing processing delays (C). Silent loss (A) is prevented by persistence; blocking API (B) contradicts decoupling; automatic failover (D) is ideal but not guaranteed.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Queue crash -> accepted but undelivered applications -> delayed processing [check]
Hint: Queue crash delays downstream, submissions accepted but unprocessed [check]
Common Mistakes:
4. If the health check module fails to run, what is the exact failure observed in the load balancer's routing behavior?
medium
A. Load balancer reroutes traffic randomly ignoring server health
B. Load balancer blocks all traffic until health checks resume
C. Load balancer routes only to a cached subset of servers marked healthy previously
D. Load balancer continues routing to all servers, including unhealthy ones, causing client errors

Solution

  1. Step 1: Understand health check module role

    Health checks detect unhealthy servers; without them, no updates occur to remove bad servers.

  2. Step 2: Trace routing behavior without health checks

    Load balancer routes to all servers indiscriminately, including unhealthy ones, causing client errors.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    No health checks -> no removal of unhealthy servers -> client errors [check]
Hint: No health checks means no unhealthy server removal [check]
Common Mistakes:
5. At 10x current scale, which component in TikTok's architecture is most likely to fail first and why?
hard
A. The WebSocket servers maintaining live connections, since stateful connections cannot be easily sharded
B. The relational database storing user profiles, due to write amplification from frequent updates
C. The Feed Ranking Service, as it requires low-latency access to large user history data causing memory bottlenecks
D. The video transcoding cluster, because CPU-intensive jobs cannot be parallelized beyond a limit

Solution

  1. Step 1: Analyze stateful connection scaling

    WebSocket servers (D) hold state per connection; scaling requires complex connection migration or sticky sessions.

  2. Step 2: Compare other bottlenecks

    Relational DB (A) can be sharded; transcoding (B) parallelizes; Feed Ranking (C) uses caching and sharding.

  3. Step 3: Identify first failure point

    WebSocket servers hit connection limits first due to stateful nature and limited horizontal scaling.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Stateful WebSocket servers bottleneck first due to connection affinity [check]
Hint: Stateful WebSocket servers limit horizontal scaling first [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard