Bird
Raised Fist0

Bulkhead Pattern - Isolating Failures to Contain Blast Radius

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
API Gateway

Receives client request

Validates and authenticates incoming HTTP request

Protocol:HTTP/HTTPS
BeforeClient request is sent over the internet
AfterRequest is authenticated and ready for routing
Bulkhead Router/LB

Routes request to a specific bulkhead pool

Selects an isolated pool to prevent cascading failures

Protocol:Internal RPC
BeforeRequest is authenticated at API Gateway
AfterRequest assigned to one bulkhead pool instance group
Bulkhead Pools

Forwards request to one microservice instance

Microservice instance processes request independently

Protocol:Internal RPC
BeforeRequest assigned to bulkhead pool
AfterMicroservice instance begins processing request
Microservice Instances

Queries shared data store for required data

Reads or writes persistent data as part of request handling

Protocol:Database Query
BeforeMicroservice instance has request context
AfterData retrieved or updated in shared data store
Shared Data Store

Persists data and responds to microservice

Ensures data durability and consistency

Protocol:Database Query
BeforeData request received from microservice
AfterData returned or updated successfully
Monitoring & Alerting Sys

Collects metrics and logs from microservice and router

Monitors health and triggers alerts on failures

Protocol:Metrics/Logs
BeforeMicroservice and router processing ongoing
AfterSystem health data collected for analysis
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────┐
│  API Gateway  │
└───────┬───────┘
        │ HTTP/HTTPS
        ▼
┌───────────────────────────────┐
│ Bulkhead Router/LB            │
└──────────────┬────────────────┘
               │ Internal RPC
               ▼
      ┌───────────────────┐
      │  Bulkhead Pools   │
      │ ┌───────────────┐ │
      │ │ Microservice  │ │
      │ │ Instances     │ │
      │ └───────────────┘ │
      └─────────┬─────────┘
                │ Database Query
                ▼
      ┌───────────────────┐
      │ Shared Data Store │
      └─────────┬─────────┘
                │ Metrics/Logs
                ▼
  ┌───────────────────────────────┐
  │ Monitoring & Alerting Sys      │
  └───────────────────────────────┘
Design Rationale

The Bulkhead Pattern isolates failures by routing requests through a Bulkhead Router that directs traffic to separate Bulkhead Pools. Each pool contains Microservice Instances that process requests independently, limiting failure impact to a single pool. The API Gateway authenticates requests before routing. Microservices interact with a Shared Data Store for persistence, ensuring data consistency. Meanwhile, a Monitoring & Alerting System collects metrics and logs from components to detect failures early. This architecture prevents cascading failures by isolating faults within pools, maintaining overall system stability and availability.

Primary Focus

✓

Practice

(1/5)
1. A large-scale messaging platform needs to isolate failures to prevent cascading outages during regional network partitions. Which architectural component best limits the blast radius by decoupling service dependencies?
easy
A. Use synchronous REST calls between microservices with retries
B. Use direct TCP socket connections between services for low latency
C. Implement asynchronous message queues with backpressure and dead-letter queues
D. Rely on a single monolithic database to coordinate all service state

Solution

  1. Step 1: Identify failure isolation mechanism

    Asynchronous message queues decouple services, buffering requests and preventing direct cascading failures during outages.

  2. Step 2: Evaluate alternatives

    Synchronous REST calls (A) cause blocking and cascading failures; monolithic DB (C) is a single point of failure; direct TCP (D) lacks buffering and backpressure.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Async queues buffer and isolate failures, limiting blast radius [check]
Hint: Async queues buffer failures, blocking calls cascade [check]
Common Mistakes:
2. In the Snowflake ID generation flow, at Step 3 the sequence number is incremented after the timestamp and machine ID are set. What happens if the sequence number exceeds its maximum value within the same millisecond?
easy
A. The generator blocks and waits until the next millisecond to reset the sequence number.
B. The system resets the sequence to zero and continues generating IDs immediately.
C. The sequence number wraps around causing duplicate IDs within the same millisecond.
D. The machine ID is incremented to avoid sequence overflow.

Solution

  1. Step 1: Understand sequence number role in Snowflake

    The sequence number differentiates IDs generated within the same millisecond on the same machine.

  2. Step 2: Analyze overflow handling

    If the sequence exceeds its max (e.g., 4095), the generator must wait for the next millisecond to avoid duplicates. Resetting immediately (A) risks duplicates; wrapping (C) causes collisions; incrementing machine ID (D) is not part of the design.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sequence overflow -> wait next millisecond to avoid duplicates [check]
Hint: Sequence overflow blocks until next millisecond [OK]
Common Mistakes:
3. If the feature store's ingestion Kafka topic is unavailable, what exactly breaks in the training pipeline's feature freshness guarantees?
medium
A. The training pipeline continues with stale features but logs a warning
B. The feature store falls back to batch recomputation, delaying training by hours
C. Feature ingestion blocks, causing training data to miss recent feature updates until Kafka recovers
D. The training pipeline silently drops missing features, causing inconsistent training data

Solution

  1. Step 1: Understand Kafka's role in feature ingestion

    Kafka streams feature updates to the store, enabling near real-time freshness.

  2. Step 2: Trace failure impact

    If Kafka is down, streaming ingestion halts. The system falls back to batch recomputation to update features, causing delays. Training cannot proceed with partial or dropped features without risking data quality.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Kafka outage triggers batch fallback, delaying freshness [check]
Hint: Kafka down -> batch recomputation fallback delays freshness [check]
Common Mistakes:
4. At 10x scale, which component in a full-text search system is most likely to become the bottleneck first, and why?
hard
A. The inverted index storage nodes, because random disk seeks increase dramatically with index size
B. The distributed log system (Kafka), because partition count limits throughput and causes uneven load
C. The indexing workers, because they must process and merge a growing volume of update events in real-time
D. The query router, because it must maintain stateful TCP connections to all index shards

Solution

  1. Step 1: Analyze query router bottleneck

    Query routers are typically stateless or use connection pooling; they scale horizontally well, so (A) is less likely.

  2. Step 2: Evaluate inverted index storage nodes

    As index size grows 10x, random disk seeks increase non-linearly, causing latency spikes and throughput degradation, making (B) the primary bottleneck.

  3. Step 3: Consider indexing workers and Kafka

    Indexing workers and Kafka scale with partitioning and parallelism, so (C) and (D) are secondary bottlenecks.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Disk seek latency grows fastest, limiting index node throughput [check]
Hint: Disk seeks grow fastest, hitting index storage nodes first [check]
Common Mistakes:
5. At 10x scale, which component in a synchronous chat system is most likely to fail first and why?
hard
A. The database, because it cannot handle the increased write volume synchronously
B. The load balancer, because it cannot distribute synchronous requests evenly at high scale
C. The client devices, because they cannot maintain multiple synchronous connections simultaneously
D. The chat servers, because WebSocket connections are stateful and cannot be easily resharded or migrated

Solution

  1. Step 1: Identify stateful bottleneck in synchronous chat

    Chat servers maintain WebSocket connections statefully, binding clients to specific servers.

  2. Step 2: Analyze scaling challenges

    At 10x scale, resharding or migrating these stateful connections is complex, causing chat servers to fail first.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stateful WebSocket servers limit horizontal scaling, causing early failure [check]
Hint: Stateful WebSocket servers bottleneck synchronous chat scaling [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard