Bird
Raised Fist0

AWS US-EAST-1 Outage - Cascading Failures and Blast Radius Lessons

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
🏗️
AWS US-EAST-1 Outage - Cascading Failures and Blast Radius Lessons
Designing resilient distributed systems to prevent cascading failures and minimize blast radius during cloud region outages
hardSystem Failures and Post-MortemsFreq: medium
Prerequisites:Distributed Systems BasicsObservabilityCloud InfrastructureFault Tolerance
⚙️
Functional Requirements
  • Detect and isolate failures rapidly within a cloud region to prevent cascading effects
  • Provide multi-region failover and disaster recovery with minimal downtime
  • Implement fine-grained fault isolation to limit blast radius of failures
  • Maintain service availability and data consistency during partial outages
  • Enable real-time observability and alerting for cascading failure patterns
  • Support automated rollback and circuit breakers to contain faulty components
📊
Non-Functional Requirements
  • 99.99% availability SLA during regional outages
  • Failover recovery time under 5 minutes for critical services
  • Observability data ingestion at 1M events/sec with 100ms P99 latency
  • System must handle 2 billion daily active users globally
  • Data replication latency under 500ms across regions
❓
Clarifying Questions to Ask
  • What is the expected scale of traffic and data volume during outages?
  • Which AWS services or components are most critical to protect?
  • Are we designing for active-active multi-region or active-passive failover?
  • What are the consistency requirements during failover?
  • What observability tools and telemetry data are available?
  • What is the acceptable trade-off between availability and consistency?
Out of Scope
✗ Designing AWS internal infrastructure or control plane
✗ Detailed network routing protocols within AWS backbone
✗ Billing and cost optimization strategies
✗ User-facing application features unrelated to fault tolerance
📌
Assumptions
  • 2 billion daily active users globally
  • Each user generates 40 messages/events per day
  • Average message size is 100 bytes
  • Data retention period is 30 days
  • Replication factor of 3 for durability
⚡
QPS Calculation
write qps2B users × 40 msgs/user/day ÷ 86400 sec/day ≈ 925,926 writes/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2.78M writes/sec
read qpsAssuming 5 reads per write: 925,926 × 5 ≈ 4.63M reads/sec
💾
Storage Estimate
daily storage2B × 40 × 100 bytes = 8 TB/day
retention storage8 TB/day × 30 days = 240 TB
with replication240 TB × 3 replicas = 720 TB total storage
🖥️
Server Estimate
write pathKafka cluster with 100 brokers, each handling ~28K writes/sec
storage nodesCassandra cluster with 200 nodes, each storing ~3.6 TB data
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |   Client Applications |
          +----------+-----------+
                     |
                     v
          +----------------------+
          |   API Gateway (ALB)  |
          +----------+-----------+
                     |
                     v
          +----------------------+
          |   Write Kafka Cluster |
          |  (100 brokers, 3x RF) |
          +----------+-----------+
                     |
                     v
          +----------------------+
          | Cassandra Storage     |
          | (200 nodes, 3x RF)   |
          +----------+-----------+
                     |
                     v
          +----------------------+
          |  Multi-Region Replic. |
          |  (AWS US-WEST-2 etc) |
          +----------+-----------+
                     |
                     v
          +----------------------+
          | Observability & Alert |
          | (Prometheus + ELK)    |
          +----------------------+
🧩
Components
API Gateway (AWS ALB)
Ingress point for client requests with load balancing and TLS termination
Why thisAWS ALB supports high throughput, native integration with AWS services, and automatic scaling
Why notNginx requires manual scaling and lacks native AWS integration, increasing operational overhead
Kafka Cluster (Apache Kafka on EC2)
Durable, high-throughput write buffer for incoming events with replication
Why thisKafka provides partitioned logs with strong durability and horizontal scalability to handle 2.78M writes/sec peak
Why notAWS SQS lacks partitioning and throughput for 2M+ writes/sec; RabbitMQ does not scale linearly at this scale
Cassandra Storage Cluster
Distributed NoSQL store for event persistence with multi-region replication
Why thisCassandra offers linear write scalability, tunable consistency, and multi-datacenter replication for 720TB data
Why notPostgreSQL requires complex manual sharding at this scale; B-tree writes don't scale linearly
Multi-Region Replication Service
Asynchronous replication of data to secondary AWS regions for disaster recovery
Why thisAWS DMS or custom Cassandra replication ensures data durability and availability during US-EAST-1 outages
Why notSynchronous cross-region writes increase latency beyond 500ms SLA, impacting user experience
Observability & Alerting Stack (Prometheus + ELK)
Collects telemetry, detects cascading failure patterns, and triggers alerts
Why thisPrometheus supports high-cardinality metrics ingestion at 1M events/sec with 100ms P99 latency; ELK enables log aggregation and analysis
Why notCloudWatch alone lacks granularity and real-time alerting needed for cascading failure detection
Circuit Breaker & Rollback Controller
Automates failure containment by disabling faulty components and rolling back changes
Why thisHystrix-like pattern implemented via AWS Lambda and Step Functions enables rapid isolation and recovery
Why notManual intervention delays recovery and increases blast radius during cascading failures
🔄
Data Flow
1
Client Applications → API Gateway
Send user events and requests
API Gateway load balances and authenticates incoming traffic
2
API Gateway → Kafka Cluster
Publish events to Kafka topics partitioned by user ID
Kafka buffers writes to handle peak load and provides durability
3
Kafka Cluster → Cassandra Storage
Consume events and persist to Cassandra with replication
Cassandra stores data with tunable consistency and multi-region replication
4
Cassandra Storage → Multi-Region Replication Service
Asynchronously replicate data to secondary AWS regions
Ensures disaster recovery and availability during regional outages
5
Observability & Alerting Stack → Circuit Breaker & Rollback Controller
Detect cascading failure patterns and trigger automated rollback
Rapid failure containment to minimize blast radius

Practice

(1/5)
1. In a system implementing causal consistency, at step 4 the Read Coordinator receives a read request for key K after a write W causally related to a previous write V. What state does the Read Coordinator ensure before returning the value for K?
easy
A. It returns the latest value of K without waiting, since eventual consistency suffices
B. It waits until all writes causally preceding W, including V, are visible to the reader
C. It returns the value from the local replica immediately, ignoring causal dependencies
D. It synchronizes with the leader to enforce linearizability before responding

Solution

  1. Step 1: Trace Read Coordinator behavior at Step 4

    Under causal consistency, the Read Coordinator must ensure that all causally prior writes (like V) are visible before returning W's value to maintain causal order.
  2. Step 2: Eliminate other options

    It returns the latest value of K without waiting, since eventual consistency suffices ignores causal dependencies; It returns the value from the local replica immediately, ignoring causal dependencies breaks causal guarantees by returning stale data; It synchronizes with the leader to enforce linearizability before responding enforces stronger linearizability, not required here.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Read Coordinator waits for causal dependencies before returning [check]
Hint: Causal consistency waits for all prior causally related writes [check]
Common Mistakes:
2. A web service requires encrypted communication with client browsers but must minimize CPU overhead on the server. Which component primarily handles the encryption and decryption workload during connection setup?
easy
A. The TLS handshake process managed by the server's SSL/TLS library
B. The client browser encrypting data before sending HTTP requests
C. The web application server performing manual encryption of HTTP payloads
D. The DNS server resolving domain names to IP addresses

Solution

  1. Step 1: Identify TLS handshake role

    The TLS handshake is responsible for negotiating encryption parameters and performing cryptographic operations to secure the connection.
  2. Step 2: Eliminate other options

    The web application server performing manual encryption of HTTP payloads is incorrect because manual encryption by the app server is inefficient and not standard. The client browser encrypting data before sending HTTP requests is incorrect because the client does not perform the handshake encryption workload; it participates but the server's TLS library handles the cryptographic operations. The DNS server resolving domain names to IP addresses is unrelated to encryption.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    TLS handshake handles encryption setup on server side [check]
Hint: TLS handshake manages encryption setup on server side [check]
Common Mistakes:
3. A public API receives millions of requests per minute from diverse clients. To prevent abuse, which layer should primarily enforce a global rate limit that blocks excessive requests before reaching internal services?
easy
A. Apply rate limiting exclusively inside each microservice to control business logic
B. Implement rate limiting only at the database layer to prevent overload
C. Enforce rate limiting at the API Gateway to block abusive traffic early
D. Use client-side rate limiting to reduce server load

Solution

  1. Step 1: Identify the role of API Gateway

    The API Gateway acts as the first entry point and can block abusive traffic before it reaches internal services or databases.
  2. Step 2: Evaluate other layers

    Database-level rate limiting is too late and risks overload; service-level limits are per-service and don't prevent global abuse; client-side limits are unreliable as clients can be malicious.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    API Gateway blocks abusive requests early [check]
Hint: Block abusive traffic earliest at API Gateway [check]
Common Mistakes:
4. In a system where Step 4 is the Delivery Worker reading from a Kafka topic partitioned by user ID, what is the Delivery Worker's state if it reads a message but the downstream notification service is temporarily unavailable?
easy
A. The message is marked as delivered and removed from Kafka
B. The Delivery Worker skips the message and moves to the next
C. The message is lost because Kafka does not guarantee delivery
D. The message remains uncommitted in Kafka and will be retried

Solution

  1. Step 1: Understand Kafka consumer commit semantics

    Kafka commits offsets only after successful processing. If downstream is unavailable, the message is not committed.
  2. Step 2: Delivery Worker retries uncommitted messages

    Uncommitted messages remain in the topic and will be redelivered on retry, ensuring no message loss.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Kafka offset commit after processing -> retries on failure [check]
Hint: Kafka commits offsets only after successful processing [OK]
Common Mistakes:
5. At 10x scale, which component in the Ticket Server pattern is most likely to become the bottleneck first, and why?
hard
A. Ticket Server, because it handles all block allocations and cannot be sharded easily
B. Clients, because local ID block exhaustion causes frequent server requests
C. Network, because increased traffic saturates bandwidth between clients and server
D. Database storing allocated blocks, because it cannot handle concurrent writes at scale

Solution

  1. Step 1: Identify centralization bottleneck

    The Ticket Server centrally allocates ID blocks and maintains allocation state, limiting horizontal scaling.
  2. Step 2: Compare other components

    Clients scale horizontally; network scales with infrastructure; database load is less critical if Ticket Server caches state.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Ticket Server centralization -> first bottleneck at 10x scale [check]
Hint: Central Ticket Server state limits horizontal scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard