Bird
Raised Fist0

Read Replicas and CQRS Pattern for Read-Heavy Systems

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Read Replicas and CQRS Pattern for Read-Heavy Systems
Designing scalable read-heavy systems using read replicas and CQRS to separate read/write workloads
mediumDatabases - SQL and TransactionsFreq: common
⚙️
Functional Requirements
  • Support high throughput read queries with low latency (sub-100ms P99).
  • Support consistent writes with transactional guarantees on the primary database.
  • Separate read and write workloads using CQRS pattern to optimize performance.
  • Provide eventual consistency between write and read stores with configurable lag tolerance.
  • Support horizontal scaling of read replicas to handle increasing read traffic.
  • Allow failover and recovery of read replicas without impacting write availability.
📊
Non-Functional Requirements
  • System must support 2 billion daily active users (DAU).
  • Write throughput estimated at 925K writes per second at peak.
  • Read throughput estimated at 10x write throughput (≈9.25M reads/sec peak).
  • Read latency P99 must be under 100ms.
  • System availability SLA of 99.99% for both reads and writes.
❓
Clarifying Questions to Ask
  • What is the acceptable consistency model between writes and reads? Is eventual consistency acceptable?
  • What is the expected read-to-write ratio? Is 10:1 a good assumption?
  • Are there any specific SLAs for failover time of read replicas?
  • What is the average size of a data record or message?
  • Is the system expected to support complex read queries or mostly key-value lookups?
  • What is the retention period for data stored in the system?
Out of Scope
✗ Designing the client-side caching or CDN layer.
✗ Handling multi-region global replication and geo-distribution.
✗ Designing the write-side database sharding strategy in detail.
✗ Security and authentication mechanisms.
📌
Assumptions
  • 2 billion DAU generating 40 write messages per user per day.
  • Average message size is 100 bytes.
  • Read-to-write ratio is 10:1 based on system usage patterns.
  • Retention period for data is 30 days.
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1.85M writes/sec
read qpsWrite QPS × 10 = 1.85M × 10 = 18.5M reads/sec peak
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024³) ≈ 7.45 TB/day
retention storage gb7.45 TB × 30 days = 223.5 TB
with replicationAssuming 3x replication for durability: 223.5 TB × 3 = 670.5 TB total
🖥️
Server Estimate
write db nodesAssuming each write node handles 50K writes/sec, need 1.85M / 50K = 37 nodes
read replica nodesEach read replica handles 500K reads/sec, need 18.5M / 500K = 37 replicas
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |   Client Apps  |
          +-------+--------+
                  |
          +-------v--------+
          |  API Gateway   |
          +-------+--------+
                  |
          +-------v--------+
          | Command Service|
          +-------+--------+
                  |
          +-------v--------+
          | Primary DB (SQL|
          |  - PostgreSQL) |
          +-------+--------+
                  |
          +-------v--------+
          |  Kafka Topic   |
          +-------+--------+
                  |
          +-------v--------+
          | Read Replica   |
          |  DB Cluster    |
          | (PostgreSQL)   |
          +-------+--------+
                  |
          +-------v--------+
          | Query Service  |
          +----------------+
🧩
Components
Primary DB (PostgreSQL)
Handles all write transactions with strong consistency and durability.
Why thisPostgreSQL supports ACID transactions and strong consistency needed for writes.
Why notMySQL with master-slave replication has higher replication lag and weaker consistency guarantees at scale.
Kafka Topic
Acts as an event log to asynchronously propagate write events to read replicas.
Why thisKafka provides durable, ordered, and scalable event streaming with high throughput and low latency.
Why notRabbitMQ lacks Kafka's throughput and partitioning scalability for 1.85M+ events/sec.
Read Replica DB Cluster (PostgreSQL)
Serves read queries from replicas asynchronously updated via Kafka consumers.
Why thisPostgreSQL read replicas provide familiar SQL interface and can scale horizontally for reads.
Why notElasticsearch is optimized for search, not transactional reads, and eventual consistency is harder to tune.
Command Service
Receives client write requests, validates, and writes to Primary DB synchronously.
Why thisStateless microservice layer enables scaling writes and enforcing business logic before DB writes.
Why notDirect DB writes from clients risk inconsistent validation and lack of centralized control.
Query Service
Handles read requests by querying read replicas, providing low-latency responses.
Why thisDecouples read workload from writes, enabling independent scaling and caching strategies.
Why notDirect client queries to primary DB would overload it and increase write latency.
🔄
Data Flow
1
Client Apps → API Gateway
Send read or write requests
API Gateway routes requests to Command or Query services
2
API Gateway → Command Service
Forward write requests
Command Service validates and processes writes
3
Command Service → Primary DB
Execute transactional writes
Ensures strong consistency and durability
4
Primary DB → Kafka Topic
Publish write events asynchronously
Decouples write and read workloads
5
Kafka Topic → Read Replica DB Cluster
Consume events and update read replicas
Enables eventual consistency for reads
6
Client Apps → API Gateway
Send read requests
API Gateway routes to Query Service
7
API Gateway → Query Service
Query read replicas
Read replicas serve low-latency read queries
8
Query Service → Client Apps
Return read responses
Low latency responses served from read replicas

Practice

(1/5)
1. A web application needs to handle millions of concurrent users sending requests simultaneously. Which component is primarily responsible for managing client connections and distributing requests to backend services?
easy
A. Database server, because it stores all user data and handles queries
B. Load balancer, because it manages incoming client connections and routes requests
C. Cache server, because it speeds up data retrieval for clients
D. Message queue, because it buffers requests for asynchronous processing

Solution

  1. Step 1: Identify the role of the load balancer

    The load balancer manages client connections and distributes incoming requests evenly across backend servers to prevent overload.

  2. Step 2: Eliminate other components

    The database server handles data storage but does not manage client connections; cache servers speed up data retrieval but don't handle connections; message queues buffer asynchronous tasks but don't manage live client connections.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Load balancer routes client requests to backend servers [check]
Hint: Load balancer directs traffic, not stores or caches data [check]
Common Mistakes:
2. If the Kafka cluster is unavailable, what exact failure occurs in the notification system's delivery path?
medium
A. Notifications are lost silently because the delivery workers cannot fetch messages
B. Notification producers block synchronously, causing upstream request latency spikes and potential timeouts
C. Delivery workers switch to a backup queue automatically with no message loss
D. The system falls back to synchronous HTTP calls to devices, causing slow delivery

Solution

  1. Step 1: Trace failure impact of Kafka unavailability

    Kafka unavailability blocks producers or causes retries; delivery workers cannot fetch messages from Kafka.

  2. Step 2: Evaluate fallback mechanisms

    In a robust design, a backup queue or dead-letter queue is used to avoid message loss; synchronous HTTP fallback (D) is not automatic; silent loss (A) is unacceptable.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Backup queue prevents message loss during Kafka downtime [check]
Hint: Backup queue prevents loss when Kafka is down [check]
Common Mistakes:
3. At 10x transaction volume, which component in the digital wallet system is most likely to become the first bottleneck, and why?
hard
A. The event log (Kafka), because partition count limits throughput and rebalancing partitions causes downtime
B. The reconciliation service, because it performs expensive cross-checks on all transactions synchronously
C. The API gateway, because it handles all client requests and cannot horizontally scale due to session affinity
D. The ledger database, because it requires strong consistency and cannot shard entries without breaking double-entry guarantees

Solution

  1. Step 1: Analyze ledger database constraints

    Ledger DB enforces strong consistency and atomic double-entry transactions, limiting sharding options.

  2. Step 2: Compare other components

    Reconciliation (A) can be scaled asynchronously; API gateway (C) can be scaled with stateless design; Kafka (D) scales with partitioning but ledger DB consistency is harder to scale.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Ledger DB consistency limits sharding -> first bottleneck [check]
Hint: Ledger DB consistency limits sharding -> first bottleneck [OK]
Common Mistakes:
4. At 10x scale, which component in the multi-tenant SaaS platform is most likely to fail first, and why?
hard
A. Feature Flag Service cache, because stale flag data causes widespread inconsistent behavior
B. Billing database, due to write amplification from synchronous updates per tenant request
C. Tenant Metadata Service, as it becomes a centralized bottleneck for tenant context lookups
D. API Gateway, because it maintains stateful WebSocket connections that cannot be easily sharded

Solution

  1. Step 1: Analyze scaling bottlenecks

    Billing DB (A) can be scaled with sharding; Tenant Metadata (C) can be cached; API Gateway (D) handles stateless HTTP mostly.

  2. Step 2: Evaluate Feature Flag Service cache at scale

    Cache invalidation and stale data grow exponentially, causing inconsistent feature exposure and client errors.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Feature flag cache stale -> inconsistent behavior at scale [check]
Hint: Feature flag cache stale -> first scale failure [check]
Common Mistakes:
5. At 10x scale increase for a system using Redis as a cache in front of DynamoDB, which component is most likely to fail first and why?
hard
A. Redis cache servers because memory pressure causes eviction storms and latency spikes
B. Application servers because CPU saturates handling cache misses
C. Network load balancers because they cannot handle increased connection counts
D. DynamoDB because partition throughput limits are exceeded causing throttling

Solution

  1. Step 1: Analyze Redis cache behavior under 10x load

    Increased load causes Redis memory pressure, triggering eviction storms and latency spikes.

  2. Step 2: Compare Redis failure risk to DynamoDB and other components

    DynamoDB scales well with partitioning; network and app servers scale horizontally; Redis memory is stateful and harder to scale quickly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Redis memory pressure -> eviction storms -> latency spikes first at scale [check]
Hint: Redis memory pressure -> eviction storms cause first failure at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard