Bird
Raised Fist0

Circuit Breaker Pattern - Preventing Cascade Failures Across Services

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Circuit Breaker Pattern - Preventing Cascade Failures Across Services
45 min · senior
senior

You are designing a circuit breaker system for a startup with 50 million daily active users (DAU) running a microservices architecture. The system must prevent cascading failures when downstream services become slow or unavailable. The environment requires encrypted communication, minimal latency impact (P99 < 200ms), and must handle high concurrency with dynamic service scaling.

Constraints
  • All inter-service communication must be encrypted using TLS 1.3
  • Circuit breaker state updates must propagate within 100ms to avoid stale decisions
  • P99 latency overhead introduced by circuit breaker must be under 200ms
  • System must support 50M DAU with peak 1M concurrent requests per second
  • Circuit breaker must support dynamic service instance scaling without losing state
📋
Guided Steps
1/5
Step 1
Choosing the Circuit Breaker State Store
What technology will you use to store and propagate circuit breaker states across distributed services?
💡 Hint
Consider a low-latency, strongly consistent store that supports pub/sub for fast state propagation.
Key Decision
Use Apache Cassandra with lightweight transactions (LWT) for consistent circuit breaker state storage and Apache Kafka for state change event propagation within 100ms.
Interviewer Expectation
Mid: Names Cassandra or Redis. Senior: Explains Cassandra LWT for consistency and Kafka for event-driven state sync to avoid stale states.
⚠️Common Wrong Answer
Using Redis without persistence or strong consistency leads to stale or lost circuit breaker states during failover.
Step 2
Circuit Breaker State Machine and Metrics Aggregation
How will you design the circuit breaker state machine and aggregate failure metrics to decide when to open or close the circuit?
💡 Hint
Think about sliding windows, failure thresholds, and how to aggregate metrics from multiple instances efficiently.
Key Decision
Implement a sliding window failure counter using Apache Flink streaming jobs consuming Kafka metrics topics to compute real-time failure rates per service instance.
Interviewer Expectation
Mid: Mentions counters or fixed windows. Senior: Explains Flink streaming for low-latency, scalable aggregation and how sliding windows reduce false positives.
⚠️Common Wrong Answer
Using fixed windows or polling metrics from databases causes delayed or inaccurate circuit breaker decisions.
Step 3
API Design for Circuit Breaker Interaction
Design 3 core API endpoints for services and clients to interact with the circuit breaker system.
💡 Hint
Consider endpoints for reporting call results, querying circuit state, and resetting or retrying after timeout.
Key Decision
Expose RESTful endpoints: POST /circuit-breaker/report (service reports success/failure), GET /circuit-breaker/state?service_id= (client queries state), POST /circuit-breaker/reset (admin or automated retry triggers).
Interviewer Expectation
Mid: Lists endpoints without detailed semantics. Senior: Explains idempotency, security (TLS + auth), and latency constraints for each endpoint.
⚠️Common Wrong Answer
Using synchronous blocking calls for state queries increases latency and risk of cascading failures.
Step 4
Handling Dynamic Scaling and State Consistency
How will your design handle dynamic scaling of service instances without losing circuit breaker state consistency?
💡 Hint
Consider how state is partitioned and synchronized across ephemeral instances.
Key Decision
Partition circuit breaker state by service instance ID in Cassandra with Kafka topics keyed by service instance for event-driven updates, ensuring new instances fetch latest state on startup.
Interviewer Expectation
Mid: Mentions caching or sticky sessions. Senior: Explains partition keys for O(1) lookups and Kafka consumer group rebalancing to maintain state consistency during scaling.
⚠️Common Wrong Answer
Storing state only in local memory causes inconsistent circuit states and incorrect decisions after scaling.
Step 5
Ensuring Low Latency and Security
How will you ensure the circuit breaker adds minimal latency and meets encryption requirements?
💡 Hint
Think about TLS termination points and asynchronous state updates.
Key Decision
Terminate TLS at service mesh (e.g., Istio) for encrypted communication; use asynchronous Kafka event propagation and local cache with TTL to minimize blocking calls and keep P99 latency under 200ms.
Interviewer Expectation
Mid: Mentions TLS and caching. Senior: Explains service mesh benefits, asynchronous updates to avoid blocking, and cache invalidation strategies.
⚠️Common Wrong Answer
Synchronous state fetch on every request adds latency and risks cascading failures.
🔌
API Design Challenge
Sketch 3 core API endpoints for the circuit breaker system that clients and services will use.
Hints
  • How does a service report failure or success to the circuit breaker?
  • How does a client query the circuit breaker state before calling a service?
  • How does the circuit breaker notify or allow recovery after a timeout?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of a strongly consistent distributed store (Cassandra) for circuit breaker state
  • Event-driven state propagation with Kafka to avoid stale decisions
  • Sliding window failure metrics aggregation for accurate circuit state transitions
  • API design covering reporting, querying, and resetting circuit breaker state
  • Encryption of all inter-service communication using TLS 1.3
Good to Have - 30%
  • Service mesh integration for TLS termination and observability
  • Partitioning state by service instance for scalability and consistency
  • Local caching with TTL to reduce latency overhead
  • Asynchronous updates to meet P99 latency constraints
Bonus - 10%
  • Use of Apache Flink for real-time streaming analytics
  • Detailed failure mode analysis and fallback strategies
Anti-patterns to Avoid
Storing circuit breaker state only in local memory causes inconsistent circuit states and incorrect decisions after scaling, leading to cascading failures.
Polling a database synchronously on every request causes high latency and cascading failures due to blocking calls under load.
Using Redis without persistence or strong consistency leads to lost or stale circuit breaker states during failover, causing incorrect circuit decisions.
Blocking synchronous calls for circuit breaker state queries increase tail latency, risking cascading failures under high load.
Ignoring encryption requirements exposes sensitive failure data and risks man-in-the-middle (MITM) attacks compromising system security.
✅
Model Answer
▼ Reveal

The system uses Apache Cassandra with lightweight transactions to store circuit breaker states consistently, combined with Apache Kafka for event-driven state propagation within 100ms. Apache Flink streaming jobs aggregate failure metrics in real-time using sliding windows to accurately determine circuit state transitions. RESTful APIs provide endpoints for reporting call results, querying circuit state, and resetting circuits, all secured with TLS 1.3 and authentication. A service mesh like Istio handles TLS termination and secure communication. Circuit breaker state is partitioned by service instance ID for O(1) lookups and consistent scaling. Local in-memory caching with TTL reduces latency and avoids blocking calls, while asynchronous Kafka event propagation keeps state fresh within the required latency constraints.

Key Decisions
  • Use Apache Cassandra with lightweight transactions to ensure consistent distributed circuit breaker state storage
  • Use Apache Kafka for low-latency, event-driven propagation of circuit breaker state changes within 100ms
  • Implement Apache Flink streaming jobs to perform real-time sliding window failure metric aggregation per service instance
  • Design RESTful APIs with idempotent POST /report for reporting call results, GET /state for querying circuit state, and POST /reset for resetting circuits, all secured with TLS 1.3 and authentication
  • Leverage a service mesh like Istio to terminate TLS and provide encrypted communication between services
  • Partition circuit breaker state by service instance ID to enable O(1) lookups and maintain consistency during dynamic scaling
  • Use local in-memory caching with TTL to minimize blocking calls and keep P99 latency under 200ms
  • Propagate state changes asynchronously via Kafka events to avoid blocking synchronous calls and stale state
  • Apache Cassandra with lightweight transactions for consistent distributed circuit breaker state storage
  • Apache Kafka for low-latency event-driven propagation of circuit breaker state changes
  • Apache Flink streaming jobs for real-time sliding window failure metric aggregation
  • RESTful API endpoints: POST /report, GET /state, POST /reset with TLS 1.3 encryption and authentication
  • Service mesh (e.g., Istio) for TLS termination and secure inter-service communication
  • Partitioning circuit breaker state by service instance ID for O(1) lookups and consistent scaling
  • Local in-memory cache with TTL for circuit breaker state to reduce latency and avoid blocking calls
  • Asynchronous event-driven updates to keep circuit breaker state fresh within 100ms propagation window

Practice

(1/5)
1. In a cache system using LFU eviction, at step 4 the eviction process selects a candidate key for removal. What is the state of the candidate's frequency count at this step?
easy
A. The candidate has the highest frequency count among all keys
B. The candidate's frequency count is ignored and eviction is random
C. The candidate's frequency count is reset to zero before eviction
D. The candidate has the lowest frequency count among all keys

Solution

  1. Step 1: Understand LFU eviction criteria

    LFU evicts the cache entry with the lowest access frequency to keep frequently used items.

  2. Step 2: Analyze eviction candidate state at step 4

    The candidate selected for eviction must have the lowest frequency count; frequency is not reset or ignored.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    LFU evicts lowest frequency count keys [check]
Hint: LFU evicts lowest frequency, not highest or random [OK]
Common Mistakes:
2. In a distributed tracing system, at step 4 the Trace Collector receives a span with a missing parent ID. What is the most likely state of the trace at this point?
easy
A. The trace is complete and ready for visualization
B. The span is treated as a root span, starting a new trace segment
C. The span is discarded because it cannot be linked to any trace
D. The span is buffered indefinitely until the parent span arrives

Solution

  1. Step 1: Understand how tracing handles missing parent IDs

    Spans without parent IDs are treated as root spans, starting new trace trees.

  2. Step 2: Eliminate other options

    Discarding spans loses data unnecessarily; buffering indefinitely risks memory leaks; trace completeness requires all spans, so it's not complete yet.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Missing parent ID -> new root span starts trace segment [check]
Hint: Missing parent ID means new root span [check]
Common Mistakes:
3. Without the distributed metadata service, what exactly breaks in the blob store's object retrieval process?
medium
A. Object retrieval requests block indefinitely waiting for metadata resolution
B. Clients receive errors because object locations and versions cannot be resolved
C. Clients can still retrieve blobs but with stale metadata versions
D. Blobs are retrieved but data integrity checks fail silently

Solution

  1. Step 1: Trace retrieval dependency on metadata service

    The metadata service provides object location and version info; without it, retrieval cannot resolve where to fetch data.

  2. Step 2: Eliminate other options

    Stale metadata retrieval is impossible without metadata service; blocking indefinitely is unlikely as client errors occur; silent integrity failures require data access, which fails first.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Metadata service absence causes unresolved object location errors [check]
Hint: No metadata service -> no object location resolution -> retrieval errors [check]
Common Mistakes:
4. Without Kafka in the messaging pipeline, what exactly breaks when Alice sends a message to Bob?
medium
A. Alice's send blocks until Bob responds; if Bob's server crashes, message is lost with no retry
B. Messages are duplicated multiple times due to lack of deduplication
C. Messages are encrypted incorrectly causing security issues
D. Alice's client immediately shows message delivered even if Bob is offline

Solution

  1. Step 1: Understand Kafka's role in decoupling

    Kafka buffers messages asynchronously, preventing sender blocking. Without Kafka, Chat Server A must synchronously contact Bob's server.

  2. Step 2: Trace failure without Kafka

    If Bob's server is slow or crashes, Alice's send blocks or message is lost with no retry, causing poor UX and data loss.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    No Kafka -> synchronous send blocks and message loss [check]
Hint: Kafka buffers async; without it, sends block and lose messages [check]
Common Mistakes:
5. For the restaurant discovery feature, you must support millions of users searching by location with low latency. Which architecture is best suited given the constraint that restaurant data updates happen frequently (every few minutes)?
medium
A. Use a relational database with geospatial indexes and query directly for each user search
B. Use a distributed cache with geospatial indexing that asynchronously syncs updates from the primary database
C. Use a batch ETL pipeline to rebuild a search index every hour and serve queries from it
D. Use client-side filtering by downloading all restaurants in the city and filtering locally

Solution

  1. Step 1: Analyze read/write pattern for restaurant discovery

    High read volume with frequent writes requires low-latency reads and near-real-time updates.

  2. Step 2: Evaluate architectures

    Relational DB (A) can't handle query volume with low latency; batch ETL (C) is too stale; client filtering (D) is impractical for large data.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Distributed cache with async sync -> low latency + fresh data [check]
Hint: Async cache sync balances freshness and query speed [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard