Bird
Raised Fist0

Circuit Breaker Pattern - Preventing Cascade Failures Across Services

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Circuit Breaker Pattern - Preventing Cascade Failures Across Services
Design a circuit breaker system to prevent cascading failures in microservices at scale
mediumMicroservices InfrastructureFreq: common
⚙️
Functional Requirements
  • Detect service call failures and latency spikes in real-time
  • Automatically open circuit breakers to stop calls to failing services
  • Provide configurable thresholds for error rates, latency, and retry limits
  • Allow half-open state to test service recovery before closing circuit
  • Expose metrics and health status for monitoring and alerting
  • Support distributed tracing integration to correlate failures
📊
Non-Functional Requirements
  • Support 2 billion daily active users generating up to 925K service calls per second
  • Circuit breaker state changes must propagate within 100ms P99 latency
  • System availability of 99.99% to avoid false positives causing downtime
  • Handle burst traffic spikes up to 3x baseline QPS without degradation
  • Minimal overhead: circuit breaker checks add less than 1ms latency per call
❓
Clarifying Questions to Ask
  • What is the expected failure detection granularity? Per endpoint, per service, or per instance?
  • Are circuit breaker states shared globally or isolated per client/service?
  • What is the maximum acceptable delay for propagating circuit breaker state changes?
  • Should the system support manual override or only automatic state transitions?
  • Is integration with existing monitoring and alerting systems required?
  • What is the expected failure rate threshold to trigger circuit breaker open state?
Out of Scope
✗ Design of the underlying microservices themselves
✗ Service discovery and load balancing mechanisms
✗ Detailed retry/backoff algorithms beyond circuit breaker states
✗ End-user facing UI for circuit breaker configuration
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates 40 service calls per day on average
  • Average payload size per service call is 100 bytes
  • Circuit breaker state changes occur at 0.1% of total calls
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 calls/sec
peak write qps925,926 × 3 (burst factor) ≈ 2,777,778 calls/sec
read qpsAssuming 2 reads per write for state checks: 925,926 × 2 ≈ 1,851,852 reads/sec
💾
Storage Estimate
daily storage bytes925,926 calls/sec × 100 bytes × 86,400 sec ≈ 8 TB/day
retention days7
total storage bytes8 TB × 7 days = 56 TB
with replication56 TB × 3 replicas = 168 TB total storage
🖥️
Server Estimate
circuit breaker serviceAssuming 10,000 RPS per server, need 278 servers at peak QPS
state store clusterUsing Cassandra with 100 MB/s write throughput per node, need ~30 nodes for writes
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |  Client Microservice  |
          +----------+-----------+
                     |
                     | Service call with circuit breaker check
                     v
          +----------------------+
          | Circuit Breaker Proxy |
          +----------+-----------+
                     |
          +----------+-----------+
          | Circuit Breaker State |
          |       Store (Cassandra)|
          +----------+-----------+
                     |
          +----------+-----------+
          |  Target Microservice  |
          +----------------------+
                     |
          +----------+-----------+
          | Metrics & Monitoring  |
          |      (Prometheus)     |
          +----------------------+
🧩
Components
Circuit Breaker Proxy
Intercepts service calls, checks circuit breaker state, and decides to allow or short-circuit calls
Why thisEnvoy proxy with native circuit breaker filters supports sub-millisecond latency and dynamic config updates
Why notCustom in-app circuit breaker adds latency and complexity; Envoy offloads logic and scales independently
Circuit Breaker State Store
Stores circuit breaker states (closed, open, half-open) and metrics per service endpoint
Why thisApache Cassandra provides linear write scalability and low-latency reads for 925K+ writes/sec with tunable consistency
Why notPostgreSQL requires manual sharding and cannot handle 2.7M writes/sec peak without bottlenecks
Metrics & Monitoring
Aggregates error rates, latencies, and state changes for alerting and dashboards
Why thisPrometheus supports high-cardinality metrics and integrates with Grafana for real-time monitoring
Why notElasticsearch is slower for time-series metrics and less efficient for real-time alerting
State Propagation Service
Distributes circuit breaker state changes to all proxies with <100ms latency
Why thisApache Kafka provides durable, ordered event streaming with low latency and consumer groups for scaling
Why notRedis Pub/Sub lacks durability and ordering guarantees, risking inconsistent states
Health Check & Recovery Manager
Periodically tests half-open circuit breakers to detect service recovery and close circuits
Why thisKubernetes CronJobs combined with custom health check microservice enables automated recovery testing
Why notManual recovery testing is error-prone and slow, risking prolonged downtime
🔄
Data Flow
1
Client Microservice → Circuit Breaker Proxy
Client sends service call intercepted by proxy
Proxy checks circuit breaker state before forwarding
2
Circuit Breaker Proxy → Circuit Breaker State Store
Reads current circuit breaker state for target service endpoint
Low-latency Cassandra read to decide allow or short-circuit
3
Circuit Breaker Proxy → Target Microservice
Forwards call if circuit is closed or half-open
Short-circuits call immediately if circuit is open
4
Target Microservice → Circuit Breaker Proxy
Returns response or error
Proxy records success/failure metrics
5
Circuit Breaker Proxy → Metrics & Monitoring
Sends metrics on call success, failure, and latency
Prometheus scrapes metrics for alerting
6
Metrics & Monitoring → State Propagation Service
Triggers circuit breaker state changes based on thresholds
Kafka streams state change events to all proxies for sync
🗄️
circuit_breaker_states
PK: service_id
service_id (UUID)endpoint (text)timestamp (timestamp)state (text)error_rate (float)latency_ms (int)
Stores circuit breaker states and metrics per service endpoint over time
🗄️
metrics_aggregation
PK: service_id
service_id (UUID)metric_type (text)timestamp (timestamp)value (double)
Aggregated metrics like error rates and latencies for alerting
🔌
GET
GetCircuitBreakerState
Request
{
  "service_id": "UUID",
  "endpoint": "string"
}
Response
{
  "state": "string",
  "error_rate": "float",
  "latency_ms": "int"
}
Fetch current circuit breaker state and metrics for a service endpoint
🔌
POST
UpdateCircuitBreakerState
Request
{
  "service_id": "UUID",
  "endpoint": "string",
  "state": "string",
  "error_rate": "float",
  "latency_ms": "int"
}
Response
{
  "success": "boolean"
}
Update circuit breaker state after evaluation or recovery test
🔌
GET
GetMetrics
Request
{
  "service_id": "UUID",
  "metric_type": "string",
  "start_time": "timestamp",
  "end_time": "timestamp"
}
Response
{
  "values": "array of {timestamp: timestamp, value: double}"
}
Retrieve aggregated metrics for monitoring and alerting

Practice

(1/5)
1. In the digital wallet system's transaction processing flow, at Step 4 the Ledger Service applies the debit and credit entries. What happens if the Ledger Service detects a mismatch in the double-entry records at this step?
easy
A. The Ledger Service retries the transaction indefinitely until the entries match
B. The Ledger Service marks the transaction as failed and triggers an alert for manual reconciliation
C. The Ledger Service ignores the mismatch and commits the debit entry only
D. The Ledger Service rolls back the entire transaction and deletes all related entries

Solution

  1. Step 1: Understand Ledger Service's double-entry validation

    The Ledger Service enforces double-entry consistency; mismatches indicate data corruption or bugs.

  2. Step 2: Evaluate failure handling

    Retrying indefinitely (A) risks blocking; ignoring mismatch (C) breaks accounting; deleting entries (D) risks data loss. Marking failed and alerting (B) enables safe manual reconciliation.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Mismatch triggers failure mark and alert -> safe reconciliation [check]
Hint: Mismatch triggers failure mark and alert, not silent retry [OK]
Common Mistakes:
2. In the message delivery trace, at Step 4 the Delivery Worker checks the consumer offset stored in ZooKeeper before sending messages. What happens if the offset is missing at this step?
easy
A. The Delivery Worker resets the offset to the earliest available message in the partition
B. The Delivery Worker blocks indefinitely waiting for the offset to appear
C. The Delivery Worker skips the message and moves to the next offset
D. The Delivery Worker deletes the partition and requests a new one

Solution

  1. Step 1: Understand consumer offset role

    The offset tracks the last consumed message; if missing, the consumer has no progress marker.

  2. Step 2: Delivery Worker behavior on missing offset

    Kafka semantics dictate resetting to earliest offset to avoid message loss; skipping (A) would lose messages; blocking (C) causes deadlock; deleting partition (D) is invalid.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing offset -> reset to earliest offset to ensure delivery [check]
Hint: Missing offset resets to earliest message, never skips or blocks [check]
Common Mistakes:
3. A system requires handling 1 million writes per second with guaranteed durability and ordered processing per user session. Which database technology best fits this requirement?
easy
A. Use a distributed log-based system like Kafka partitioned by user session
B. Use a relational database with row-level locking to ensure order and durability
C. Use an in-memory key-value store like Redis without persistence enabled
D. Use a NoSQL document store with eventual consistency and no ordering guarantees

Solution

  1. Step 1: Identify ordering and durability needs

    Relational DBs with row-level locking can't handle 1M writes/sec efficiently; Redis without persistence loses durability; NoSQL eventual consistency lacks ordering guarantees.

  2. Step 2: Recognize Kafka's partitioning for ordering

    Kafka partitions by user session ensure ordered processing and durability at high throughput.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka partitions -> ordered, durable, high-throughput writes [check]
Hint: Kafka partitions preserve order and durability at scale [check]
Common Mistakes:
4. For a high-throughput system requiring SSL termination and HTTP header manipulation, which architecture is more appropriate and why?
medium
A. Use a Reverse Proxy for SSL termination and header manipulation because it operates at the HTTP layer with fine-grained control
B. Use an API Gateway for SSL termination and header manipulation because it is optimized for high throughput
C. Use a Load Balancer for SSL termination and header manipulation because it handles traffic efficiently
D. Use DNS load balancing to distribute SSL termination across multiple servers

Solution

  1. Step 1: Identify component capabilities

    Load Balancers typically operate at transport layer and may support SSL termination but lack detailed HTTP header manipulation; API Gateways add overhead and are less optimized for raw throughput; DNS load balancing does not handle SSL termination.

  2. Step 2: Match requirement to component

    Reverse Proxy operates at HTTP layer, supports SSL termination and header manipulation efficiently, suitable for high throughput.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Reverse Proxy = HTTP layer control + SSL termination [check]
Hint: Reverse Proxy = SSL termination + HTTP header control [check]
Common Mistakes:
5. At 10x scale, which component in a GraphQL API gateway is most likely to become the bottleneck and why?
hard
A. The underlying database, because it cannot handle 10x query volume
B. The GraphQL query parsing and validation layer, due to CPU-intensive query complexity checks on every request
C. The network load balancer, because it cannot distribute traffic evenly at high scale
D. The client SDK, because it caches too much data locally

Solution

  1. Step 1: Identify components stressed by 10x scale

    Database scaling is often handled by sharding or replicas; load balancers scale horizontally; client SDKs are local.

  2. Step 2: Recognize GraphQL parsing and validation cost

    Parsing and complexity validation are CPU-bound and happen per request, becoming CPU bottlenecks at high request rates.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    GraphQL validation CPU cost scales linearly with requests [check]
Hint: GraphQL validation CPU cost limits scale first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard