Read replicas in HLD - System Design Exercise

Design: Read Replica System for Database Scaling

Design focuses on read replica architecture, replication mechanism, failover strategy, and load balancing for read queries. Write path design and application logic are out of scope.

Functional Requirements

FR1: Support high read throughput by distributing read requests across multiple replicas

FR2: Ensure data consistency between primary database and replicas with minimal lag

FR3: Allow failover to replicas in case the primary database fails

FR4: Support read queries with p99 latency under 100ms

FR5: Handle up to 100,000 read requests per second

FR6: Maintain 99.9% availability for read operations

Non-Functional Requirements

NFR1: Replication lag should be less than 2 seconds

NFR2: Writes must go to the primary database only

NFR3: System should be horizontally scalable by adding more read replicas

NFR4: Failover should not cause more than 5 seconds downtime

NFR5: Use industry-standard replication technologies

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Primary database server

Read replica database servers

Replication mechanism (e.g., asynchronous or semi-synchronous replication)

Load balancer for read traffic

Monitoring and alerting system for replication lag and failover

Failover manager or orchestrator

Design Patterns

Master-slave replication pattern

Read-write splitting

Failover and recovery pattern

Load balancing with health checks

Eventual consistency vs strong consistency trade-offs

Reference Architecture

          +-------------------+
          |   Application     |
          +---------+---------+
                    |
          +---------v---------+
          |   Read Load       |
          |   Balancer        |
          +----+--------+-----+
               |        |
      +--------v+      +-v---------+
      | Read Replica 1| Read Replica 2|
      +--------------+--------------+
               \        /
                \      /
             +----v----v----+
             | Primary DB   |
             +-------------+

Components

Primary Database

Relational DBMS with replication support (e.g., PostgreSQL, MySQL)

Handles all write operations and propagates changes to read replicas

Read Replicas

Same DBMS as primary configured as replicas

Serve read-only queries to reduce load on primary and improve read throughput

Replication Mechanism

Asynchronous or semi-synchronous replication

Keep replicas updated with changes from primary with minimal lag

Read Load Balancer

Software or hardware load balancer (e.g., HAProxy, Envoy)

Distribute read requests evenly across healthy replicas

Monitoring System

Monitoring tools (e.g., Prometheus, Grafana)

Track replication lag, replica health, and trigger alerts

Failover Manager

Automated failover tool or orchestrator (e.g., Patroni, MHA)

Detect primary failure and promote a replica to primary

Request Flow

1. 1. Application sends write requests to the primary database.

2. 2. Primary database processes writes and asynchronously replicates changes to read replicas.

3. 3. Application sends read requests to the read load balancer.

4. 4. Load balancer routes read requests to one of the healthy read replicas.

5. 5. Read replica serves the read query and returns the result to the application.

6. 6. Monitoring system continuously checks replication lag and replica health.

7. 7. If primary database fails, failover manager promotes a read replica to primary.

8. 8. Application updates configuration to send writes to the new primary.

Database Schema

Entities remain the same as primary database schema. Replication is at the data storage level, so no schema changes are required. Relationships and constraints are preserved on replicas to ensure data integrity.

Scaling Discussion

Bottlenecks

Replication lag increases with high write volume causing stale reads

Primary database becomes a write bottleneck under heavy write load

Load balancer becomes a single point of failure or bottleneck

Failover delay causes downtime impacting availability

Network bandwidth limits replication speed

Solutions

Use semi-synchronous replication to reduce lag at cost of slight write latency increase

Scale writes by sharding or partitioning data across multiple primaries

Deploy multiple load balancers with health checks and failover

Automate failover with fast detection and promotion tools

Use compression and efficient protocols for replication traffic

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 15 minutes designing architecture and data flow, 10 minutes discussing scaling and failover strategies, and 10 minutes answering questions.

Explain the difference between primary and read replicas and their roles

Discuss replication methods and trade-offs between consistency and latency

Describe how load balancing improves read throughput and availability

Highlight monitoring and failover mechanisms to maintain uptime

Address scaling challenges and realistic solutions

Practice

1. What is the primary purpose of using read replicas in a system design?

easy

A. To backup data by creating copies on different servers

B. To increase write throughput by adding more write nodes

C. To improve read scalability by offloading read queries from the primary database

D. To encrypt data for security purposes

Solution

1. Step 1: Understand the role of read replicas

   Read replicas are copies of the primary database used mainly to handle read requests.

2. Step 2: Identify the benefit in system design

   By directing read queries to replicas, the primary database is less loaded, improving read scalability.

3. Final Answer:

   To improve read scalability by offloading read queries from the primary database -> Option C

4. Quick Check:

   Read replicas = improve read scalability [OK]

Hint: Read replicas handle reads, not writes, to reduce primary load [OK]

Common Mistakes:

• Thinking read replicas improve write performance
• Confusing read replicas with backup systems
• Assuming read replicas encrypt data

2. Which of the following is the correct way to describe replication in read replicas?

easy

A. Replication is synchronous, so replicas always have the latest data instantly.

B. Replication duplicates only write queries, not read queries.

C. Replication happens only when the primary database restarts.

D. Replication is asynchronous, so replicas may lag behind the primary database.

Solution

1. Step 1: Understand replication types

   Read replicas usually use asynchronous replication, meaning data updates are sent with some delay.

2. Step 2: Identify the correct replication behavior

   Because replication is asynchronous, replicas may not always have the latest data instantly.

3. Final Answer:

   Replication is asynchronous, so replicas may lag behind the primary database. -> Option D

4. Quick Check:

   Replication = asynchronous lag possible [OK]

Hint: Replication delay means replicas can be slightly outdated [OK]

Common Mistakes:

• Assuming replication is always synchronous
• Thinking replication triggers only on restart
• Confusing replication of queries with replication of data

3. Consider a system with one primary database and two read replicas. If the primary receives 100 write requests and 300 read requests per second, how should the read requests be handled to optimize performance?

medium

A. Send all 300 read requests to the read replicas and 100 writes to the primary

B. Send all 400 requests to the primary database

C. Send half of the read requests to the primary and half to the replicas

D. Send all requests randomly to any database

Solution

1. Step 1: Separate reads and writes

   Writes must go to the primary database to maintain data consistency.

2. Step 2: Offload reads to replicas

   All read requests should be directed to read replicas to reduce load on the primary and improve read response.

3. Final Answer:

   Send all 300 read requests to the read replicas and 100 writes to the primary -> Option A

4. Quick Check:

   Reads to replicas, writes to primary = best practice [OK]

Hint: Direct reads to replicas, writes to primary for best performance [OK]

Common Mistakes:

• Sending writes to replicas causing data inconsistency
• Overloading primary with all requests
• Randomly distributing requests without separation

4. A system uses read replicas but users report seeing stale data sometimes. What is the most likely cause and how can it be fixed?

medium

A. Primary database is overloaded; fix by adding more replicas

B. Replication is asynchronous causing delay; fix by using synchronous replication

C. Read replicas are down; fix by restarting replicas

D. Network latency is high; fix by reducing read requests

Solution

1. Step 1: Identify cause of stale data

   Stale data usually happens because asynchronous replication delays updates to replicas.

2. Step 2: Fix by changing replication method

   Using synchronous replication ensures replicas have the latest data but may impact performance.

3. Final Answer:

   Replication is asynchronous causing delay; fix by using synchronous replication -> Option B

4. Quick Check:

   Stale data = async replication delay [OK]

Hint: Stale reads? Check replication delay type [OK]

Common Mistakes:

• Restarting replicas without checking replication
• Adding replicas without fixing replication lag
• Reducing reads instead of addressing replication

5. You are designing a high-traffic e-commerce platform. To handle millions of read requests per second, you plan to use read replicas. Which combination of strategies best ensures scalability and data freshness?

hard

A. Use multiple asynchronous read replicas with eventual consistency and implement cache invalidation on writes

B. Use a single synchronous read replica to guarantee data freshness but risk read bottlenecks

C. Send all reads and writes to the primary database to avoid replication delays

D. Use asynchronous replicas without any cache or consistency strategy

Solution

1. Step 1: Understand scalability needs

   Millions of reads require multiple replicas to distribute load effectively.

2. Step 2: Balance data freshness and performance

   Asynchronous replicas scale well but have lag; cache invalidation helps keep data fresh for users.

3. Step 3: Evaluate other options

   Single synchronous replica limits scalability; sending all to primary causes bottlenecks; no cache hurts freshness.

4. Final Answer:

   Use multiple asynchronous read replicas with eventual consistency and implement cache invalidation on writes -> Option A

5. Quick Check:

   Multiple async replicas + cache = scalable & fresh [OK]

Hint: Combine async replicas with cache for scale and freshness [OK]

Common Mistakes:

• Choosing single synchronous replica causing bottlenecks
• Ignoring cache leading to stale reads
• Sending all traffic to primary causing overload