Bird
Raised Fist0

AWS US-EAST-1 Outage - Cascading Failures and Blast Radius Lessons

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
AWS US-EAST-1 Outage - Cascading Failures and Blast Radius Lessons
45 min · senior
senior

You are a senior engineer at a 50M DAU startup running critical services primarily in AWS US-EAST-1. Recently, a major AWS outage in US-EAST-1 caused cascading failures across your services, resulting in prolonged downtime and data loss. Your task is to design a resilient system architecture that minimizes blast radius and prevents cascading failures during such cloud provider outages. The system must maintain P99 latency under 200ms for user-facing APIs and ensure data durability and availability even if an entire AWS region becomes unavailable.

Constraints
  • All data must be encrypted at rest and in transit to comply with security policies.
  • System must tolerate full AWS US-EAST-1 region failure with minimal downtime.
  • P99 latency for user-facing APIs must be under 200ms under normal load.
  • Offline delivery of critical user notifications is required during partial outages.
  • Cross-region replication must be asynchronous to avoid write latency impact.
  • Cost constraints limit active-active multi-region deployments to two regions.
📋
Guided Steps
1/5
Step 1
Designing Multi-Region Data Replication
How would you architect data replication to ensure durability and availability during a full AWS US-EAST-1 outage?
💡 Hint
Consider asynchronous cross-region replication with eventual consistency to avoid write latency spikes.
Key Decision
Use Amazon DynamoDB Global Tables with asynchronous cross-region replication between US-EAST-1 and US-WEST-2 to ensure data durability and availability during region failure.
Interviewer Expectation
Mid: Names DynamoDB Global Tables for multi-region replication. Senior: Explains asynchronous replication avoids write latency impact and eventual consistency trade-offs during failover.
⚠️Common Wrong Answer
Synchronous cross-region writes causing high write latency and cascading failures during region degradation.
Step 2
Mitigating Cascading Failures via Service Isolation
How would you isolate services to prevent cascading failures during an AWS region outage?
💡 Hint
Think about service boundaries, circuit breakers, and fallback mechanisms to contain failures.
Key Decision
Implement service isolation using AWS Lambda with reserved concurrency limits and AWS API Gateway throttling combined with circuit breakers in client SDKs to prevent overload propagation.
Interviewer Expectation
Mid: Mentions circuit breakers and throttling. Senior: Details reserved concurrency limits in Lambda to cap resource usage and explains fallback strategies to degrade gracefully.
⚠️Common Wrong Answer
No isolation causing resource exhaustion and cascading failures across dependent services.
Step 3
Designing Offline Delivery and Sync Mechanisms
How would you design offline delivery of critical notifications and client sync after outages?
💡 Hint
Consider durable message queues and client-side state reconciliation with idempotent APIs.
Key Decision
Use Amazon SQS FIFO queues for durable offline message storage with message deduplication, combined with client sync APIs that use cursor-based reconciliation to ensure exactly-once delivery.
Interviewer Expectation
Mid: Names SQS FIFO for message durability. Senior: Explains deduplication and cursor-based sync to avoid duplicates and ensure consistency after reconnect.
⚠️Common Wrong Answer
Using standard SQS or HTTP polling causing message duplication and high load during reconnect.
Step 4
API Design for Resilient Client Interactions
Sketch three core API endpoints that support resilient client operations during and after an outage.
💡 Hint
How does the client send data ensuring durability despite region failures? How does the client synchronize state or notifications after reconnecting? How does the system acknowledge successful delivery or processing to avoid duplicates?
Key Decision
Design POST /sendMessage with idempotency keys stored in DynamoDB Global Tables, GET /syncMessages with cursor-based pagination, and POST /acknowledgeMessage to confirm delivery and enable safe retries.
Interviewer Expectation
Mid: Names endpoints and basic functionality. Senior: Explains idempotency keys stored in globally replicated DynamoDB, cursor-based sync to minimize data transfer, and acknowledgement to prevent duplicates.
⚠️Common Wrong Answer
Stateless POST without idempotency causing duplicate processing and inconsistent client state.
Step 5
Monitoring and Automated Failover Strategies
How would you monitor system health and automate failover to reduce blast radius during outages?
💡 Hint
Consider AWS CloudWatch alarms, Route 53 health checks, and automated DNS failover.
Key Decision
Use AWS CloudWatch for granular service metrics, Route 53 health checks with weighted routing policies to shift traffic automatically to US-WEST-2 during US-EAST-1 failures.
Interviewer Expectation
Mid: Mentions CloudWatch and Route 53 failover. Senior: Details weighted routing policies, health check thresholds, and gradual traffic shifting to avoid overload.
⚠️Common Wrong Answer
Manual failover causing prolonged downtime and human error.
🔌
API Design Challenge
Sketch 3 core API endpoints to support resilient user operations during and after an AWS US-EAST-1 outage.
Hints
  • How does the client send data ensuring durability despite region failures?
  • How does the client synchronize state or notifications after reconnecting?
  • How does the system acknowledge successful delivery or processing to avoid duplicates?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of asynchronous multi-region replication to avoid write latency spikes and ensure durability.
  • Service isolation with concurrency limits and circuit breakers to prevent cascading failures.
  • Durable offline delivery with exactly-once semantics using SQS FIFO and idempotent APIs.
  • API design supporting idempotency, cursor-based sync, and acknowledgements to maintain consistency.
Good to Have - 30%
  • Automated failover using Route 53 weighted routing and health checks to reduce downtime.
  • Detailed monitoring with CloudWatch for proactive failure detection.
  • Encryption at rest and in transit to satisfy security constraints.
Bonus - 10%
  • Client SDKs implementing exponential backoff and jitter for retry strategies.
  • Use of feature flags to enable gradual rollout of failover mechanisms.
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

The system uses Amazon DynamoDB Global Tables for asynchronous multi-region replication between US-EAST-1 and US-WEST-2 to ensure durability and availability during region failures. Service isolation is achieved via AWS Lambda with reserved concurrency limits and API Gateway throttling, combined with circuit breakers in client SDKs to prevent cascading failures. Offline delivery of critical notifications leverages Amazon SQS FIFO queues with message deduplication to guarantee exactly-once delivery. The API design includes idempotent endpoints with idempotency keys stored in DynamoDB Global Tables, cursor-based pagination for efficient client sync, and acknowledgement endpoints to confirm message processing and enable safe retries. Monitoring and automated failover use AWS CloudWatch for granular metrics and Route 53 weighted routing with health checks to shift traffic gradually during outages. Encryption at rest with AWS KMS and in transit via TLS ensures compliance with security policies.

Key Decisions
  • Use DynamoDB Global Tables for asynchronous cross-region replication to avoid write latency spikes and ensure data durability.
  • Implement service isolation with Lambda reserved concurrency and API Gateway throttling to prevent resource exhaustion and cascading failures.
  • Use SQS FIFO queues with message deduplication for durable offline delivery and exactly-once processing guarantees.
  • Design idempotent APIs with idempotency keys stored in globally replicated DynamoDB to avoid duplicate processing.
  • Employ cursor-based pagination in sync APIs to efficiently reconcile client state after outages.
  • Monitor system health with CloudWatch and automate failover using Route 53 weighted routing and health checks for gradual traffic shifting.
  • Implement circuit breakers in client SDKs to detect overload and prevent failure propagation.
  • Encrypt all data at rest using AWS KMS and in transit using TLS to meet security requirements.
  • Amazon DynamoDB Global Tables for asynchronous multi-region replication ensuring durability and availability during region failure.
  • AWS Lambda with reserved concurrency limits combined with API Gateway throttling to isolate services and prevent cascading failures.
  • Amazon SQS FIFO queues with message deduplication for durable offline delivery and exactly-once processing guarantees.
  • Idempotent REST APIs with idempotency keys stored in DynamoDB Global Tables to avoid duplicate processing during retries.
  • Cursor-based pagination in sync APIs to efficiently reconcile client state after outages with minimal data transfer.
  • AWS CloudWatch for detailed monitoring and Route 53 weighted routing with health checks for automated failover.
  • Circuit breakers implemented in client SDKs to detect and prevent overload propagation during partial outages.
  • Encryption at rest using AWS KMS and in transit via TLS to meet security compliance requirements.

Practice

(1/5)
1. A system ingests millions of IoT sensor readings per second and requires efficient time-based queries with high write throughput. Which storage component is most appropriate to handle this workload?
easy
A. A traditional relational database with row-level locking
B. A time-series database optimized for append-only writes and time-range queries
C. A distributed key-value store without native time indexing
D. A document database storing JSON blobs without time partitioning

Solution

  1. Step 1: Identify workload characteristics

    Millions of writes per second and time-based queries require a storage optimized for high write throughput and efficient time-range scans.

  2. Step 2: Evaluate storage options

    Relational DBs with row-level locking (A) cause write contention and poor scalability. Key-value stores (C) lack native time indexing, making time queries inefficient. Document DBs (D) without time partitioning lead to slow queries and large scans. Time-series DBs (B) are designed for append-only writes and efficient time-based queries.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Time-series DBs optimize for time-based writes and queries [check]
Hint: Time-series DBs excel at high-write, time-range queries [check]
Common Mistakes:
2. For a key-value store that must handle concurrent writes with minimal coordination, which approach is better to resolve conflicts and why?
medium
A. Use vector clocks to detect and merge conflicting versions asynchronously
B. Use synchronous quorum writes to enforce a single authoritative version
C. Use a centralized lock manager to serialize all writes
D. Use consistent hashing to route conflicting writes to the same node

Solution

  1. Step 1: Analyze conflict resolution approaches

    Vector clocks allow detection of concurrent conflicting versions without blocking writes, enabling asynchronous conflict resolution.

  2. Step 2: Eliminate other options

    Centralized locking serializes writes, causing high latency and bottlenecks. Synchronous quorum writes enforce consistency but increase latency and reduce availability. Consistent hashing routes keys but does not resolve conflicts.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Vector clocks enable asynchronous conflict detection and merging [check]
Hint: Vector clocks detect conflicts without blocking writes [OK]
Common Mistakes:
3. For a globally distributed collaborative editor with many offline users, which approach better supports eventual consistency and offline edits, and why?
medium
A. Use Operational Transformation (OT) because it requires a central server to order operations
B. Use a relational database with strict locking to serialize all edits globally
C. Use CRDTs because they allow local operations to be merged without coordination and support offline edits
D. Use a message queue with timestamp ordering to ensure global operation order

Solution

  1. Step 1: Compare OT and CRDT for offline and distributed scenarios

    OT typically requires a central server to order and transform operations, making offline support complex.

  2. Step 2: Understand CRDT advantages

    CRDTs are designed for decentralized environments, allowing local operations to be merged eventually without coordination, ideal for offline edits and global distribution.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    CRDTs enable offline edits and eventual consistency without central coordination [check]
Hint: CRDTs merge offline edits without central coordination [OK]
Common Mistakes:
4. For a mobile app with limited bandwidth and unpredictable network latency, which API design is better to minimize data usage and why?
medium
A. Use REST with multiple endpoints to fetch only needed data per screen
B. Use REST with a single endpoint returning all data to avoid multiple round-trips
C. Use GraphQL subscriptions to push all data updates continuously
D. Use GraphQL to request exactly the fields needed in one query, reducing over-fetching

Solution

  1. Step 1: Analyze bandwidth and latency constraints

    Mobile apps benefit from minimizing data transfer and round-trips due to limited bandwidth and latency.

  2. Step 2: Compare REST and GraphQL trade-offs

    REST multiple endpoints cause multiple round-trips; single endpoint REST over-fetches; GraphQL queries exactly needed fields in one request.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    GraphQL reduces over-fetching and round-trips for mobile [check]
Hint: GraphQL single query cuts data and round-trips [check]
Common Mistakes:
5. At 10x traffic scale, which load balancer component is most likely to become the bottleneck first and why?
hard
A. The IP Hash function, because hashing client IPs becomes computationally expensive at scale
B. The weighted server list, because updating weights dynamically causes synchronization delays
C. The centralized connection tracker, because Least Connections requires global state updates for every connection change
D. The Round Robin pointer, because cycling through servers sequentially cannot handle high concurrency

Solution

  1. Step 1: Identify statefulness in load balancing algorithms

    Least Connections requires tracking active connections per server globally, needing frequent updates.

  2. Step 2: Analyze scaling impact

    At 10x scale, the centralized connection tracker becomes a bottleneck due to synchronization overhead and contention.

  3. Step 3: Evaluate other components

    IP Hash is stateless and fast; weighted lists update less frequently; Round Robin pointer is lightweight and easily sharded.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Global connection tracking bottlenecks Least Connections at high scale [check]
Hint: Global connection tracking bottlenecks Least Connections at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard