Bird
Raised Fist0

Availability - Active-Active vs Active-Passive Failover Patterns

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Availability - Active-Active vs Active-Passive Failover Patterns
Designing highly available systems using Active-Active and Active-Passive failover patterns with scale and failure trade-offs
mediumNon-Functional Requirements FrameworkFreq: common
⚙️
Functional Requirements
  • System must provide continuous availability with minimal downtime during failures.
  • Support automatic failover between data centers or regions.
  • Ensure data consistency or eventual consistency depending on pattern choice.
  • Allow read and write operations to be served with minimal latency.
  • Provide monitoring and alerting for failover events and health status.
  • Support rollback or recovery mechanisms after failover.
📊
Non-Functional Requirements
  • Availability SLA of 99.99% (less than ~52 minutes downtime per year).
  • Support 2 billion daily active users (DAU) with peak write throughput of ~925K writes/sec.
  • Failover detection and recovery within 30 seconds.
  • End-to-end write latency under 100ms P99 during normal operation.
  • Data loss tolerance: zero for critical data, eventual consistency acceptable for non-critical.
❓
Clarifying Questions to Ask
  • What is the acceptable data consistency model during failover?
  • Are writes allowed concurrently in multiple regions or only in one active region?
  • What is the expected RTO (Recovery Time Objective) and RPO (Recovery Point Objective)?
  • Is the system geo-distributed across multiple continents or within a single region?
  • What is the expected read-to-write ratio during peak traffic?
  • Are there any regulatory or compliance constraints affecting data replication?
Out of Scope
✗ Detailed security and encryption mechanisms.
✗ Application-level business logic and domain modeling.
✗ Client SDK design and offline support.
✗ Multi-tenancy and access control.
📌
Assumptions
  • 2 billion DAU
  • 40 messages per user per day
  • 100 bytes average message size
  • Read-to-write ratio of 5:1
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1,851,852 writes/sec
read qpswrite_qps × 5 = 925,926 × 5 ≈ 4,629,630 reads/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024³) ≈ 7,450 GB/day
retention days30
total storage gb7,450 × 30 = 223,500 GB (~223.5 TB)
with replicationAssuming 3x replication: 223.5 TB × 3 = 670.5 TB
🖥️
Server Estimate
write db nodesUsing Cassandra with 50K writes/sec per node: 1,851,852 / 50,000 ≈ 38 nodes
read cache nodesUsing Redis cluster with 100K reads/sec per node: 4,629,630 / 100,000 ≈ 47 nodes
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Client Apps   |
          +-------+--------+
                  |
        +---------+----------+
        | Load Balancer / DNS|
        +---------+----------+
          /                  \
 +--------+--------+   +-----+-------+
 | Active DataCenter|   | Active DataCenter|
 |     Region A     |   |     Region B     |
 +--------+--------+   +-----+-------+
          |                      |
 +--------+--------+   +---------+-------+
 | Write DB Cluster |   | Write DB Cluster |
 |   (Cassandra)    |   |   (Cassandra)    |
 +--------+--------+   +---------+-------+
          |                      |
 +--------+--------+   +---------+-------+
 | Read Cache (Redis)|  | Read Cache (Redis)|
 +------------------+  +-------------------+
          |                      |
          +----------+-----------+
                     |
             +-------+-------+
             | Monitoring &  |
             | Failover Svc |
             +---------------+
🧩
Components
Load Balancer / DNS
Routes client requests to appropriate active data center based on health and latency
Why thisRoute53 with health checks supports geo-DNS failover and latency-based routing
Why notUsing hardware load balancers lacks global failover and geo-routing capabilities
Write DB Cluster (Cassandra)
Stores user data with high write throughput and multi-region replication
Why thisCassandra supports linear write scalability, tunable consistency, and multi-datacenter replication
Why notPostgreSQL requires complex manual sharding and does not scale linearly at 1.8M writes/sec
Read Cache (Redis Cluster)
Serves low-latency read requests with in-memory caching and TTL eviction
Why thisRedis Cluster supports horizontal scaling and sub-millisecond reads with native TTL expiration
Why notMemcached lacks persistence and cluster management at this scale
Monitoring & Failover Service
Detects failures and triggers failover or traffic rerouting automatically
Why thisPrometheus + custom failover orchestrator enables fast detection and automated recovery
Why notManual failover or simple heartbeat scripts have high RTO and risk human error
Client Applications
Generate read/write requests and handle failover transparently
Why thisClients implement retry logic and region fallback to maintain availability
Why notClients without failover logic cause poor user experience during outages
🔄
Data Flow
1
Client Apps → Load Balancer / DNS
Client sends read/write request
DNS routes to nearest healthy active data center
2
Load Balancer / DNS → Write DB Cluster
Write requests routed to Cassandra cluster in active region
Writes are accepted only in active region(s) depending on pattern
3
Write DB Cluster → Read Cache
Data changes asynchronously propagated to Redis cache
Cache updated to serve low-latency reads
4
Client Apps → Read Cache
Read requests served from Redis cache
Reads served locally with sub-100ms latency
5
Write DB Cluster Region A → Write DB Cluster Region B
Data replicated asynchronously or synchronously depending on pattern
Active-Active uses multi-master replication; Active-Passive uses async replication
6
Monitoring & Failover Service → Load Balancer / DNS
On failure detection, triggers failover by updating DNS or LB routing
Failover completes within 30 seconds to meet SLA
7
Monitoring & Failover Service → Client Apps
Sends alerts and health status updates
Clients receive notifications to adjust behavior during failover

Practice

(1/5)
1. For real-time analytics requiring low-latency dashboards and complex ad-hoc queries, which architecture is best suited?
medium
A. Use only Kafka Streams for both aggregation and serving queries
B. Use Flink to write directly to a NoSQL key-value store for dashboards
C. Use batch ETL jobs to load data into a relational database for dashboards
D. Use Flink for streaming aggregation and an OLAP store for dashboard queries

Solution

  1. Step 1: Evaluate streaming aggregation and query needs

    Flink excels at complex streaming aggregations; OLAP stores support fast ad-hoc queries and low-latency dashboards.

  2. Step 2: Eliminate other options

    Kafka Streams lacks complex query capabilities; batch ETL adds latency unsuitable for real-time; NoSQL key-value stores lack rich query support for dashboards.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Flink + OLAP store balances streaming and query complexity [check]
Hint: Flink streams, OLAP queries for real-time + ad-hoc [check]
Common Mistakes:
2. For global video delivery, which architecture better handles sudden traffic spikes in a specific region with minimal latency increase?
medium
A. Hierarchical CDN with regional edge caches and origin fallback
B. Centralized origin server cluster with global load balancer
C. Peer-to-peer streaming where clients share video chunks directly
D. Multiple independent origin servers in each region without caching

Solution

  1. Step 1: Evaluate architectures for traffic spike handling

    Centralized origin (Centralized origin server cluster with global load balancer) risks overload and latency spikes. Peer-to-peer (Peer-to-peer streaming where clients share video chunks directly) depends on client availability and is complex for global scale. Independent origins without caching (Multiple independent origin servers in each region without caching) lack scalability and increase origin load.

  2. Step 2: Assess hierarchical CDN benefits

    Regional edge caches (Hierarchical CDN with regional edge caches and origin fallback) absorb traffic spikes locally, reducing origin load and latency, providing scalable and resilient delivery.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Hierarchical CDN absorbs regional spikes with low latency [check]
Hint: Regional caches absorb spikes, origin fallback reduces load [check]
Common Mistakes:
3. For a system requiring per-user rate limits with minimal latency, which architecture is better and why?
medium
A. Service-layer counters with no database persistence for simplicity
B. Centralized database counters updated synchronously on every request for accuracy
C. Client-side counters with server verification on every request for trustworthiness
D. Distributed in-memory counters at the API Gateway with periodic async database sync for scalability

Solution

  1. Step 1: Compare latency and scalability

    Synchronous DB updates cause high latency and DB bottlenecks; client-side counters are untrustworthy; service-layer without persistence risks data loss.

  2. Step 2: Evaluate distributed in-memory counters

    API Gateway counters provide low latency and scalability; async DB sync ensures eventual consistency and durability.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Distributed in-memory counters balance latency and scalability [check]
Hint: Async DB sync + in-memory counters = low latency + scalability [check]
Common Mistakes:
4. For a microservices architecture with frequent service instance churn and strict latency requirements, which service discovery approach is best and why?
medium
A. Server-side discovery with centralized load balancer to simplify client logic
B. Client-side discovery with local caching to reduce latency and handle churn gracefully
C. DNS-based discovery relying on TTL for updates to minimize infrastructure complexity
D. Static configuration files updated periodically to avoid runtime discovery overhead

Solution

  1. Step 1: Analyze churn and latency constraints

    Frequent churn requires fast updates; strict latency demands minimal network hops.

  2. Step 2: Evaluate discovery approaches

    Client-side discovery with caching reduces latency and adapts quickly; server-side adds network hop; DNS TTL delays updates; static configs are stale.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Client caches endpoints locally for low latency and churn handling [check]
Hint: Client caches endpoints for fast updates and low latency [check]
Common Mistakes:
5. At 10x scale of global traffic, which component in the BGP-based routing infrastructure is most likely to fail first, and why?
hard
A. Edge routers, because their CPU cannot handle the increased packet forwarding rate
B. Route reflectors, because they become overwhelmed maintaining full routing tables and propagating updates
C. Network monitoring systems, because data volume exceeds processing capacity causing delayed alerts
D. Client devices, because they cannot process rapid route changes at scale

Solution

  1. Step 1: Identify bottleneck at scale

    Edge routers handle packet forwarding at line rate; at 10x traffic, their CPU and forwarding ASICs are stressed first.

  2. Step 2: Evaluate other components

    Route reflectors (A) scale better with hierarchical design; monitoring systems (C) can be scaled horizontally; client devices (D) are downstream and less critical for routing stability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Edge routers' CPU overload causes first failure at 10x traffic [check]
Hint: Edge routers hit CPU limits first at massive scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard