Bird
Raised Fist0

AWS US-EAST-1 Outage - Cascading Failures and Blast Radius Lessons

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
API Gateway (AWS ALB)

Receives client HTTP request

API Gateway accepts and authenticates incoming request

Protocol:HTTP/HTTPS
BeforeClient request is sent over internet to AWS US-EAST-1
AfterRequest is authenticated and forwarded to Kafka cluster
Kafka Cluster (Apache Kafka)

Enqueues request event

Kafka persists request event with partition key for ordering

Protocol:Kafka Protocol
BeforeRequest received from API Gateway, not yet durably stored
AfterRequest event durably stored in Kafka partition
Cassandra Storage Cluster

Writes request data

Cassandra stores request data with eventual consistency

Protocol:CQL Protocol
BeforeRequest event available in Kafka, not yet in DB
AfterRequest data written to Cassandra nodes
Multi-Region Replication Service

Replicates data asynchronously

Service replicates Cassandra data to other AWS regions

Protocol:Replication Protocol
BeforeData written in US-EAST-1 Cassandra cluster only
AfterData asynchronously replicated to secondary regions
Circuit Breaker & Rollback Controller

Monitors system health and triggers rollback

Detects cascading failures, opens circuit to prevent blast radius

Protocol:Control Signals
BeforeReplication lag and error rates rising in US-EAST-1
AfterCircuit breaker trips, rollback initiated to stable state
Observability & Alerting Stack (Prometheus + ELK)

Collects metrics and triggers alerts

Aggregates logs and metrics to detect anomalies and notify ops

Protocol:Metrics/Logs
BeforeSystem metrics streaming normally
AfterAlerts fired for high error rates and replication lag
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│       API Gateway (AWS ALB)   │
└───────────────┬───────────────┘
                │ HTTP/HTTPS
                ▼
┌───────────────────────────────┐
│    Kafka Cluster (Apache Kafka)│
│           on EC2)              │
└───────────────┬───────────────┘
                │ Kafka Protocol
                ▼
┌───────────────────────────────┐
│ Cassandra Storage Cluster      │
│  (NoSQL distributed DB)        │
└───────────────┬───────────────┘
                │ CQL Protocol
                ▼
┌───────────────────────────────┐
│ Multi-Region Replication       │
│ Service                       │
└───────────────┬───────────────┘
                │ Replication Protocol
                ▼
┌───────────────────────────────┐
│ Circuit Breaker & Rollback     │
│ Controller                    │
└───────────────┬───────────────┘
                │ Control Signals
                ▼
┌───────────────────────────────┐
│ Observability & Alerting Stack │
│ (Prometheus + ELK)             │
└───────────────────────────────┘
Design Rationale

This architecture diagram illustrates the flow of a client request through the AWS US-EAST-1 system during an outage scenario. The request enters via the API Gateway, which authenticates and forwards it to the Kafka Cluster for durable event storage. Kafka ensures ordered, persistent queuing before the request data is written to the Cassandra Storage Cluster, a distributed NoSQL database. The Multi-Region Replication Service asynchronously replicates data to other regions, enabling disaster recovery. The Circuit Breaker & Rollback Controller monitors system health metrics and triggers rollbacks to prevent cascading failures and limit blast radius during outages. Observability & Alerting Stack collects logs and metrics, providing real-time alerts to operators. This design highlights how asynchronous replication and circuit breakers help contain failures and maintain system stability under regional outages.

Primary Focus

✓

Practice

(1/5)
1. For the fraud detection system's rule engine, which architecture best supports rapid rule updates without downtime while maintaining low latency scoring?
medium
A. Embed rules directly in the scoring service code and redeploy on changes
B. Use an external rule engine service with hot-reloadable rule definitions
C. Store rules in a database and query them synchronously during scoring
D. Compile rules into scoring service binaries and load at startup only

Solution

  1. Step 1: Evaluate embedding rules in code (A) and binaries (D)

    Both require redeployment or restart on rule changes, causing downtime or latency spikes.

  2. Step 2: Assess querying rules synchronously (C)

    Querying rules synchronously adds latency and risk of inconsistent rule versions during scoring.

  3. Step 3: Consider external rule engine with hot-reload (B)

    This allows dynamic rule updates without downtime and low latency via caching and efficient evaluation.

  4. Final Answer:

    Option B -> Option B

  5. Quick Check:

    External hot-reload rule engine -> dynamic updates, low latency [check]
Hint: Hot-reload rule engine enables live updates without downtime [check]
Common Mistakes:
2. If the Metadata Server crashes before acknowledging a file rename operation, what is the most likely failure observed by the client?
medium
A. Client's write operations to the file silently fail without notification
B. Client receives an error and the file remains under the old name with no partial rename
C. Client sees the file renamed but data blocks lost
D. Client sees the file under both old and new names due to partial metadata update

Solution

  1. Step 1: Understand rename operation atomicity

    Rename involves updating metadata entries; if crash occurs mid-update, partial state may persist.

  2. Step 2: Trace failure effect

    Partial metadata update can cause both old and new filenames to appear, confusing clients.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Partial metadata updates cause duplicate filenames visible to client [check]
Hint: Partial metadata update causes duplicate filenames visible [check]
Common Mistakes:
3. A company must expose internal web services securely to external clients while hiding internal server details and enabling SSL termination. Which proxy architecture is best suited for this?
medium
A. Use a reverse proxy to terminate SSL and hide internal server details
B. Use a transparent proxy to intercept and forward requests without modification
C. Use a forward proxy to route external client requests to internal servers
D. Use a DNS proxy to resolve internal server names for external clients

Solution

  1. Step 1: Identify proxy role for exposing internal services

    A reverse proxy sits in front of internal servers, hides their details, and can terminate SSL for external clients.

  2. Step 2: Eliminate other options

    Forward proxies act on clients, not servers; transparent proxies don't modify requests; DNS proxies only resolve names, not handle SSL or hide servers.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Reverse proxy hides servers and terminates SSL [check]
Hint: Reverse proxy hides servers and handles SSL termination [check]
Common Mistakes:
4. At 10x scale, which component in the distributed cache system is most likely to fail first and why?
hard
A. Cache nodes because consistent hashing rings become imbalanced with many nodes
B. Backing data store because it cannot handle the increased read load
C. Cache nodes because memory pressure from replication and eviction overhead increases exponentially
D. Network because replication traffic saturates bandwidth causing high latency

Solution

  1. Step 1: Analyze consistent hashing at large scale

    With many nodes, virtual nodes and hash ring management become complex, causing imbalance and hotspots.

  2. Step 2: Compare other bottlenecks

    Memory pressure grows but is mitigated by eviction; network saturation happens later; backing store is usually scaled independently.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Consistent hashing imbalance causes cache node hotspots first [check]
Hint: Consistent hashing imbalance causes hotspots at scale [OK]
Common Mistakes:
5. At 10x scale of concurrent users, which component in a typeahead autocomplete system using Elasticsearch and Trie is most likely to become the bottleneck first, and why?
hard
A. The Trie servers, because in-memory Tries are stateful and cannot be easily sharded or distributed across nodes
B. The client-side UI, because rendering large autocomplete result sets slows down user experience
C. The prefix hash map, because storing all prefixes for billions of entries exceeds memory limits quickly
D. The Elasticsearch cluster, because fuzzy queries require heavy CPU and memory, causing query latency spikes

Solution

  1. Step 1: Analyze statefulness and scalability

    Trie servers hold large in-memory state and are difficult to shard due to pointer-based node structures, limiting horizontal scaling.

  2. Step 2: Compare other components

    Elasticsearch scales horizontally with shards; prefix hash map can be partitioned or replaced; client UI is less likely the first bottleneck at backend scale.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Trie statefulness limits sharding -> bottleneck at scale [check]
Hint: Trie stateful, hard to shard -> first bottleneck at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard