Read replicas in HLD - Scalability & System Analysis

Scalability Analysis - Read replicas

Growth Table: Read Replicas Scaling

Users	Traffic Characteristics	Database Load	Read Replica Usage	Notes
100 users	Low read/write requests	Single primary DB handles all	Not needed	Simple setup, no replicas
10,000 users	Moderate reads, writes increasing	Primary DB under moderate load	1-2 read replicas for read scaling	Offload read queries to replicas
1,000,000 users	High read volume, writes steady	Primary DB write bottleneck possible	Multiple read replicas distributed geographically	Use replicas for read-heavy traffic, reduce latency
100,000,000 users	Very high read/write traffic	Primary DB write bottleneck, network limits	Sharded replicas, geo-distributed clusters	Advanced replication, caching, and sharding needed

First Bottleneck

At small to medium scale, the primary database becomes the first bottleneck because it handles all write operations and read queries if no replicas exist. As user count grows, the primary server's CPU, disk I/O, and network bandwidth limit its ability to serve all requests quickly.

Without read replicas, read-heavy workloads overwhelm the primary DB, causing slow responses and timeouts.

Scaling Solutions

• Read Replicas: Create one or more read-only copies of the primary database to handle read queries, reducing load on the primary.
• Load Balancing: Distribute read requests evenly across replicas to maximize throughput.
• Geographical Distribution: Place replicas closer to users to reduce latency.
• Write Optimization: Keep writes on the primary; optimize with batching or queueing.
• Caching: Use in-memory caches (e.g., Redis) to reduce database reads further.
• Sharding: For very large scale, split data across multiple primary-replica clusters by key ranges.

Back-of-Envelope Cost Analysis

• Assuming 10,000 users, each making 1 request per second: 10,000 QPS total.
• Primary DB can handle ~5,000 QPS; read replicas needed to handle remaining reads.
• Each read replica can handle ~5,000 QPS; 2 replicas cover 10,000 QPS reads.
• Storage: Replicas require same storage as primary; plan for data growth.
• Network bandwidth: 1 Gbps (~125 MB/s) can support ~10,000 QPS with small query sizes.

Interview Tip

When discussing read replicas in an interview, start by explaining the primary database bottleneck under read-heavy workloads. Then describe how read replicas offload read queries to improve scalability and reduce latency. Mention trade-offs like replication lag and eventual consistency. Finally, discuss how to scale replicas horizontally and combine with caching and sharding for very large systems.

Self Check

Your database handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add read replicas to offload read queries from the primary database. This reduces load on the primary and allows the system to handle increased read traffic without immediate changes to the primary server.

Key Result

Read replicas help scale read-heavy workloads by offloading read queries from the primary database, improving throughput and reducing latency. The primary database remains the write bottleneck and requires further strategies at very large scale.

Practice

1. What is the primary purpose of using read replicas in a system design?

easy

A. To backup data by creating copies on different servers

B. To increase write throughput by adding more write nodes

C. To improve read scalability by offloading read queries from the primary database

D. To encrypt data for security purposes

Solution

1. Step 1: Understand the role of read replicas

   Read replicas are copies of the primary database used mainly to handle read requests.

2. Step 2: Identify the benefit in system design

   By directing read queries to replicas, the primary database is less loaded, improving read scalability.

3. Final Answer:

   To improve read scalability by offloading read queries from the primary database -> Option C

4. Quick Check:

   Read replicas = improve read scalability [OK]

Hint: Read replicas handle reads, not writes, to reduce primary load [OK]

Common Mistakes:

• Thinking read replicas improve write performance
• Confusing read replicas with backup systems
• Assuming read replicas encrypt data

2. Which of the following is the correct way to describe replication in read replicas?

easy

A. Replication is synchronous, so replicas always have the latest data instantly.

B. Replication duplicates only write queries, not read queries.

C. Replication happens only when the primary database restarts.

D. Replication is asynchronous, so replicas may lag behind the primary database.

Solution

1. Step 1: Understand replication types

   Read replicas usually use asynchronous replication, meaning data updates are sent with some delay.

2. Step 2: Identify the correct replication behavior

   Because replication is asynchronous, replicas may not always have the latest data instantly.

3. Final Answer:

   Replication is asynchronous, so replicas may lag behind the primary database. -> Option D

4. Quick Check:

   Replication = asynchronous lag possible [OK]

Hint: Replication delay means replicas can be slightly outdated [OK]

Common Mistakes:

• Assuming replication is always synchronous
• Thinking replication triggers only on restart
• Confusing replication of queries with replication of data

3. Consider a system with one primary database and two read replicas. If the primary receives 100 write requests and 300 read requests per second, how should the read requests be handled to optimize performance?

medium

A. Send all 300 read requests to the read replicas and 100 writes to the primary

B. Send all 400 requests to the primary database

C. Send half of the read requests to the primary and half to the replicas

D. Send all requests randomly to any database

Solution

1. Step 1: Separate reads and writes

   Writes must go to the primary database to maintain data consistency.

2. Step 2: Offload reads to replicas

   All read requests should be directed to read replicas to reduce load on the primary and improve read response.

3. Final Answer:

   Send all 300 read requests to the read replicas and 100 writes to the primary -> Option A

4. Quick Check:

   Reads to replicas, writes to primary = best practice [OK]

Hint: Direct reads to replicas, writes to primary for best performance [OK]

Common Mistakes:

• Sending writes to replicas causing data inconsistency
• Overloading primary with all requests
• Randomly distributing requests without separation

4. A system uses read replicas but users report seeing stale data sometimes. What is the most likely cause and how can it be fixed?

medium

A. Primary database is overloaded; fix by adding more replicas

B. Replication is asynchronous causing delay; fix by using synchronous replication

C. Read replicas are down; fix by restarting replicas

D. Network latency is high; fix by reducing read requests

Solution

1. Step 1: Identify cause of stale data

   Stale data usually happens because asynchronous replication delays updates to replicas.

2. Step 2: Fix by changing replication method

   Using synchronous replication ensures replicas have the latest data but may impact performance.

3. Final Answer:

   Replication is asynchronous causing delay; fix by using synchronous replication -> Option B

4. Quick Check:

   Stale data = async replication delay [OK]

Hint: Stale reads? Check replication delay type [OK]

Common Mistakes:

• Restarting replicas without checking replication
• Adding replicas without fixing replication lag
• Reducing reads instead of addressing replication

5. You are designing a high-traffic e-commerce platform. To handle millions of read requests per second, you plan to use read replicas. Which combination of strategies best ensures scalability and data freshness?

hard

A. Use multiple asynchronous read replicas with eventual consistency and implement cache invalidation on writes

B. Use a single synchronous read replica to guarantee data freshness but risk read bottlenecks

C. Send all reads and writes to the primary database to avoid replication delays

D. Use asynchronous replicas without any cache or consistency strategy

Solution

1. Step 1: Understand scalability needs

   Millions of reads require multiple replicas to distribute load effectively.

2. Step 2: Balance data freshness and performance

   Asynchronous replicas scale well but have lag; cache invalidation helps keep data fresh for users.

3. Step 3: Evaluate other options

   Single synchronous replica limits scalability; sending all to primary causes bottlenecks; no cache hurts freshness.

4. Final Answer:

   Use multiple asynchronous read replicas with eventual consistency and implement cache invalidation on writes -> Option A

5. Quick Check:

   Multiple async replicas + cache = scalable & fresh [OK]

Hint: Combine async replicas with cache for scale and freshness [OK]

Common Mistakes:

• Choosing single synchronous replica causing bottlenecks
• Ignoring cache leading to stale reads
• Sending all traffic to primary causing overload