Read replicas in HLD - System Design Guide

Problem Statement

When a database receives too many read requests, a single server can become overwhelmed, causing slow responses and potential downtime. This bottleneck limits the system's ability to serve users quickly and reliably, especially during traffic spikes.

Solution

Read replicas solve this by creating copies of the primary database that handle read requests. The primary database processes all writes and updates, then asynchronously replicates data to the read replicas. This spreads the read load across multiple servers, improving performance and availability.

Architecture

Primary DB

(Writes)

→Read Replica 1

↓

→Read Replica 2

This diagram shows a primary database handling all write operations and asynchronously replicating data to multiple read replicas that serve read requests.

Trade-offs

✓ Pros

→

Distributes read traffic to multiple servers, reducing load on the primary database.

→

Improves read query performance and reduces latency for users.

→

Increases system availability by allowing reads to continue if the primary is busy or down.

✗ Cons

→

Data replication is asynchronous, so read replicas may serve slightly stale data.

→

Adds complexity in managing replication lag and consistency.

→

Write operations still depend on the primary database, which can remain a bottleneck.

Use read replicas when read traffic exceeds 1,000 requests per second or when read latency impacts user experience significantly.

Avoid read replicas if your application requires strict read-after-write consistency or if read traffic is low (under 1,000 req/sec), where replication overhead outweighs benefits.

Real World Examples

Amazon

Amazon uses read replicas in their DynamoDB and RDS services to handle massive read workloads while keeping write operations consistent on the primary.

Netflix

Netflix employs read replicas to serve user data and streaming metadata quickly, reducing load on the primary database during peak hours.

Airbnb

Airbnb uses read replicas to scale their search and listing read queries, ensuring fast response times without affecting write performance.

Alternatives

Caching

Caching stores frequently accessed data in fast storage like memory, reducing database reads, while read replicas duplicate the entire database for reads.

Use when: Choose caching when data can be stale for short periods and you want ultra-fast access to specific queries.

Sharding

Sharding splits the database horizontally into smaller parts, each handling a subset of data, unlike read replicas which copy the full dataset.

Use when: Choose sharding when your dataset is too large for a single database and you need to scale writes and reads horizontally.

Summary

Read replicas improve system performance by distributing read traffic across multiple database copies.

They help reduce latency and increase availability but may serve slightly stale data due to asynchronous replication.

Use read replicas when read demand is high and strict immediate consistency is not required.

Practice

1. What is the primary purpose of using read replicas in a system design?

easy

A. To backup data by creating copies on different servers

B. To increase write throughput by adding more write nodes

C. To improve read scalability by offloading read queries from the primary database

D. To encrypt data for security purposes

Solution

1. Step 1: Understand the role of read replicas

   Read replicas are copies of the primary database used mainly to handle read requests.

2. Step 2: Identify the benefit in system design

   By directing read queries to replicas, the primary database is less loaded, improving read scalability.

3. Final Answer:

   To improve read scalability by offloading read queries from the primary database -> Option C

4. Quick Check:

   Read replicas = improve read scalability [OK]

Hint: Read replicas handle reads, not writes, to reduce primary load [OK]

Common Mistakes:

• Thinking read replicas improve write performance
• Confusing read replicas with backup systems
• Assuming read replicas encrypt data

2. Which of the following is the correct way to describe replication in read replicas?

easy

A. Replication is synchronous, so replicas always have the latest data instantly.

B. Replication duplicates only write queries, not read queries.

C. Replication happens only when the primary database restarts.

D. Replication is asynchronous, so replicas may lag behind the primary database.

Solution

1. Step 1: Understand replication types

   Read replicas usually use asynchronous replication, meaning data updates are sent with some delay.

2. Step 2: Identify the correct replication behavior

   Because replication is asynchronous, replicas may not always have the latest data instantly.

3. Final Answer:

   Replication is asynchronous, so replicas may lag behind the primary database. -> Option D

4. Quick Check:

   Replication = asynchronous lag possible [OK]

Hint: Replication delay means replicas can be slightly outdated [OK]

Common Mistakes:

• Assuming replication is always synchronous
• Thinking replication triggers only on restart
• Confusing replication of queries with replication of data

3. Consider a system with one primary database and two read replicas. If the primary receives 100 write requests and 300 read requests per second, how should the read requests be handled to optimize performance?

medium

A. Send all 300 read requests to the read replicas and 100 writes to the primary

B. Send all 400 requests to the primary database

C. Send half of the read requests to the primary and half to the replicas

D. Send all requests randomly to any database

Solution

1. Step 1: Separate reads and writes

   Writes must go to the primary database to maintain data consistency.

2. Step 2: Offload reads to replicas

   All read requests should be directed to read replicas to reduce load on the primary and improve read response.

3. Final Answer:

   Send all 300 read requests to the read replicas and 100 writes to the primary -> Option A

4. Quick Check:

   Reads to replicas, writes to primary = best practice [OK]

Hint: Direct reads to replicas, writes to primary for best performance [OK]

Common Mistakes:

• Sending writes to replicas causing data inconsistency
• Overloading primary with all requests
• Randomly distributing requests without separation

4. A system uses read replicas but users report seeing stale data sometimes. What is the most likely cause and how can it be fixed?

medium

A. Primary database is overloaded; fix by adding more replicas

B. Replication is asynchronous causing delay; fix by using synchronous replication

C. Read replicas are down; fix by restarting replicas

D. Network latency is high; fix by reducing read requests

Solution

1. Step 1: Identify cause of stale data

   Stale data usually happens because asynchronous replication delays updates to replicas.

2. Step 2: Fix by changing replication method

   Using synchronous replication ensures replicas have the latest data but may impact performance.

3. Final Answer:

   Replication is asynchronous causing delay; fix by using synchronous replication -> Option B

4. Quick Check:

   Stale data = async replication delay [OK]

Hint: Stale reads? Check replication delay type [OK]

Common Mistakes:

• Restarting replicas without checking replication
• Adding replicas without fixing replication lag
• Reducing reads instead of addressing replication

5. You are designing a high-traffic e-commerce platform. To handle millions of read requests per second, you plan to use read replicas. Which combination of strategies best ensures scalability and data freshness?

hard

A. Use multiple asynchronous read replicas with eventual consistency and implement cache invalidation on writes

B. Use a single synchronous read replica to guarantee data freshness but risk read bottlenecks

C. Send all reads and writes to the primary database to avoid replication delays

D. Use asynchronous replicas without any cache or consistency strategy

Solution

1. Step 1: Understand scalability needs

   Millions of reads require multiple replicas to distribute load effectively.

2. Step 2: Balance data freshness and performance

   Asynchronous replicas scale well but have lag; cache invalidation helps keep data fresh for users.

3. Step 3: Evaluate other options

   Single synchronous replica limits scalability; sending all to primary causes bottlenecks; no cache hurts freshness.

4. Final Answer:

   Use multiple asynchronous read replicas with eventual consistency and implement cache invalidation on writes -> Option A

5. Quick Check:

   Multiple async replicas + cache = scalable & fresh [OK]

Hint: Combine async replicas with cache for scale and freshness [OK]

Common Mistakes:

• Choosing single synchronous replica causing bottlenecks
• Ignoring cache leading to stale reads
• Sending all traffic to primary causing overload