Read replicas in HLD - Architecture Diagram

System Overview - Read replicas

This system uses read replicas to improve database read scalability and availability. It handles many read requests by distributing them across multiple replicas, while writes go to the primary database.

Key requirements include fast read response times, data consistency with some replication delay, and fault tolerance for database failures.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  +-------------------+
  |                   |
  v                   v
Primary DB          Read Replica 1
  |                   |
  |                   v
  |               Read Replica 2
  |
  v
Cache

Components

User

client

Sends requests to the system

Load Balancer

load_balancer

Distributes incoming requests evenly to API Gateway instances

API Gateway

api_gateway

Routes requests to appropriate services and manages read/write separation

Primary DB

database_primary

Handles all write operations and replicates data to read replicas

Read Replica 1

database_replica

Handles read requests to reduce load on primary database

Read Replica 2

database_replica

Additional read replica for scalability and availability

Cache

cache

Stores frequently accessed data to speed up read responses

Request Flow - 9 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Cache

Cache → API Gateway

API Gateway → Read Replica 1 or 2

Read Replica 1 or 2 → API Gateway

API Gateway → User

API Gateway → Primary DB

Primary DB → Read Replica 1 and Read Replica 2

Failure Scenario

Component Fails:Primary DB

Impact:Writes fail and data cannot be updated; reads from replicas may be stale

Mitigation:Promote a read replica to primary; use replication lag monitoring to ensure data freshness

Architecture Quiz - 3 Questions

Which component handles write operations in this architecture?

A Primary DB

B Read Replica 1

C Cache

D Load Balancer

Design Principle

This architecture uses read replicas to separate read and write workloads, improving read scalability and availability. Caches reduce database load further. Replication delay introduces eventual consistency, which is acceptable for many read-heavy applications.

Practice

1. What is the primary purpose of using read replicas in a system design?

easy

A. To backup data by creating copies on different servers

B. To increase write throughput by adding more write nodes

C. To improve read scalability by offloading read queries from the primary database

D. To encrypt data for security purposes

Solution

1. Step 1: Understand the role of read replicas

   Read replicas are copies of the primary database used mainly to handle read requests.

2. Step 2: Identify the benefit in system design

   By directing read queries to replicas, the primary database is less loaded, improving read scalability.

3. Final Answer:

   To improve read scalability by offloading read queries from the primary database -> Option C

4. Quick Check:

   Read replicas = improve read scalability [OK]

Hint: Read replicas handle reads, not writes, to reduce primary load [OK]

Common Mistakes:

• Thinking read replicas improve write performance
• Confusing read replicas with backup systems
• Assuming read replicas encrypt data

2. Which of the following is the correct way to describe replication in read replicas?

easy

A. Replication is synchronous, so replicas always have the latest data instantly.

B. Replication duplicates only write queries, not read queries.

C. Replication happens only when the primary database restarts.

D. Replication is asynchronous, so replicas may lag behind the primary database.

Solution

1. Step 1: Understand replication types

   Read replicas usually use asynchronous replication, meaning data updates are sent with some delay.

2. Step 2: Identify the correct replication behavior

   Because replication is asynchronous, replicas may not always have the latest data instantly.

3. Final Answer:

   Replication is asynchronous, so replicas may lag behind the primary database. -> Option D

4. Quick Check:

   Replication = asynchronous lag possible [OK]

Hint: Replication delay means replicas can be slightly outdated [OK]

Common Mistakes:

• Assuming replication is always synchronous
• Thinking replication triggers only on restart
• Confusing replication of queries with replication of data

3. Consider a system with one primary database and two read replicas. If the primary receives 100 write requests and 300 read requests per second, how should the read requests be handled to optimize performance?

medium

A. Send all 300 read requests to the read replicas and 100 writes to the primary

B. Send all 400 requests to the primary database

C. Send half of the read requests to the primary and half to the replicas

D. Send all requests randomly to any database

Solution

1. Step 1: Separate reads and writes

   Writes must go to the primary database to maintain data consistency.

2. Step 2: Offload reads to replicas

   All read requests should be directed to read replicas to reduce load on the primary and improve read response.

3. Final Answer:

   Send all 300 read requests to the read replicas and 100 writes to the primary -> Option A

4. Quick Check:

   Reads to replicas, writes to primary = best practice [OK]

Hint: Direct reads to replicas, writes to primary for best performance [OK]

Common Mistakes:

• Sending writes to replicas causing data inconsistency
• Overloading primary with all requests
• Randomly distributing requests without separation

4. A system uses read replicas but users report seeing stale data sometimes. What is the most likely cause and how can it be fixed?

medium

A. Primary database is overloaded; fix by adding more replicas

B. Replication is asynchronous causing delay; fix by using synchronous replication

C. Read replicas are down; fix by restarting replicas

D. Network latency is high; fix by reducing read requests

Solution

1. Step 1: Identify cause of stale data

   Stale data usually happens because asynchronous replication delays updates to replicas.

2. Step 2: Fix by changing replication method

   Using synchronous replication ensures replicas have the latest data but may impact performance.

3. Final Answer:

   Replication is asynchronous causing delay; fix by using synchronous replication -> Option B

4. Quick Check:

   Stale data = async replication delay [OK]

Hint: Stale reads? Check replication delay type [OK]

Common Mistakes:

• Restarting replicas without checking replication
• Adding replicas without fixing replication lag
• Reducing reads instead of addressing replication

5. You are designing a high-traffic e-commerce platform. To handle millions of read requests per second, you plan to use read replicas. Which combination of strategies best ensures scalability and data freshness?

hard

A. Use multiple asynchronous read replicas with eventual consistency and implement cache invalidation on writes

B. Use a single synchronous read replica to guarantee data freshness but risk read bottlenecks

C. Send all reads and writes to the primary database to avoid replication delays

D. Use asynchronous replicas without any cache or consistency strategy

Solution

1. Step 1: Understand scalability needs

   Millions of reads require multiple replicas to distribute load effectively.

2. Step 2: Balance data freshness and performance

   Asynchronous replicas scale well but have lag; cache invalidation helps keep data fresh for users.

3. Step 3: Evaluate other options

   Single synchronous replica limits scalability; sending all to primary causes bottlenecks; no cache hurts freshness.

4. Final Answer:

   Use multiple asynchronous read replicas with eventual consistency and implement cache invalidation on writes -> Option A

5. Quick Check:

   Multiple async replicas + cache = scalable & fresh [OK]

Hint: Combine async replicas with cache for scale and freshness [OK]

Common Mistakes:

• Choosing single synchronous replica causing bottlenecks
• Ignoring cache leading to stale reads
• Sending all traffic to primary causing overload