Database replication (master-slave) in HLD - Scalability & System Analysis

Scalability Analysis - Database replication (master-slave)

Growth Table: Database Replication (Master-Slave)

Users / Traffic	Writes per second	Reads per second	Replication Lag	Number of Slaves	Notes
100 users	10	50	Negligible	1	Single master, one slave for read scaling
10,000 users	1,000	5,000	Small, under 1 second	3-5	More slaves added to handle read traffic
1,000,000 users	50,000	200,000	Noticeable, seconds delay	10-20	Replication lag increases, master CPU high
100,000,000 users	1,000,000+	5,000,000+	High, seconds to minutes	50+	Master bottleneck, replication lag critical

First Bottleneck

The master database server is the first bottleneck because it handles all write operations. As traffic grows, the master CPU and disk I/O become overwhelmed. Replication lag to slaves increases, causing stale reads. Network bandwidth between master and slaves can also limit replication speed.

Scaling Solutions

• Read Scaling: Add more slave replicas to distribute read queries.
• Write Scaling: Use sharding to split data across multiple masters.
• Connection Pooling: Reduce overhead by reusing database connections.
• Caching: Use in-memory caches (e.g., Redis) to reduce read load on slaves.
• Asynchronous Replication: Accept some lag to improve master throughput.
• Monitoring: Track replication lag and master load to trigger scaling actions.

Back-of-Envelope Cost Analysis

Assuming 1 master and 5 slaves at 10,000 users:

• Writes: ~1,000 QPS on master (CPU and disk intensive)
• Reads: ~5,000 QPS distributed across slaves (~1,000 QPS each)
• Replication bandwidth: ~10 MB/s between master and slaves (depends on data size)
• Storage: Master and slaves store full dataset; storage grows with data size
• Network: Master needs high bandwidth to push changes to all slaves

Interview Tip

Start by explaining the master-slave model and its purpose (write master, read slaves). Discuss bottlenecks focusing on the master for writes and replication lag. Then propose scaling strategies like adding slaves, sharding, and caching. Always mention trade-offs such as eventual consistency and lag. Use numbers to justify your points.

Self Check Question

Your database master handles 1,000 QPS writes. Traffic grows 10x to 10,000 QPS writes. What do you do first and why?

Answer: The master is overwhelmed. First, consider sharding the database to split write load across multiple masters. This reduces load per master and improves write scalability. Adding more slaves won't help write bottleneck.

Key Result

Master database becomes the first bottleneck as write traffic grows; scaling requires sharding for writes and adding slaves for reads.

Practice

1. What is the primary role of the master in a master-slave database replication setup?

easy

A. Handle all write operations and propagate changes to slaves

B. Serve read requests only

C. Backup data from slaves

D. Monitor network traffic between nodes

Solution

1. Step 1: Understand master responsibilities

   The master database is responsible for handling all write operations to ensure data consistency.

2. Step 2: Understand slave responsibilities

   Slaves replicate data from the master and typically handle read requests to reduce load on the master.

3. Final Answer:

   Handle all write operations and propagate changes to slaves -> Option A

4. Quick Check:

   Master = writes + replication [OK]

Hint: Master handles writes; slaves handle reads [OK]

Common Mistakes:

• Thinking slaves handle writes
• Confusing backup role with master
• Assuming master only monitors

2. Which of the following is the correct sequence for setting up master-slave replication?

easy

A. Configure slaves only, master is optional

B. Configure slaves, then configure master, then start replication

C. Start replication, then configure master and slaves

D. Configure master, then configure slaves, then start replication

Solution

1. Step 1: Configure master first

   The master must be configured first to accept writes and log changes for replication.

2. Step 2: Configure slaves after master

   Slaves are then configured to connect to the master and replicate its data.

3. Final Answer:

   Configure master, then configure slaves, then start replication -> Option D

4. Quick Check:

   Master setup before slaves [OK]

Hint: Master setup precedes slaves setup [OK]

Common Mistakes:

• Configuring slaves before master
• Starting replication before configuration
• Ignoring master configuration

3. Consider a master-slave setup where the master receives 100 writes per second and each slave replicates with a 2-second lag. If a read request is sent to a slave immediately after a write, what is the likely result?

medium

A. The slave returns the latest data including the write

B. The slave crashes due to replication lag

C. The slave returns stale data missing the latest write

D. The slave rejects the read request

Solution

1. Step 1: Understand replication lag effect

   Replication lag means slaves are behind the master by 2 seconds, so recent writes are not yet applied.

2. Step 2: Analyze read request timing

   A read immediately after a write will hit the slave before replication applies the change, so stale data is returned.

3. Final Answer:

   The slave returns stale data missing the latest write -> Option C

4. Quick Check:

   Replication lag causes stale reads [OK]

Hint: Replication lag means slaves have delayed data [OK]

Common Mistakes:

• Assuming slaves always have latest data
• Thinking slaves reject reads during lag
• Believing slaves crash due to lag

4. A system uses master-slave replication but experiences inconsistent reads from slaves. Which change can fix this issue?

medium

A. Allow writes on slaves to sync data faster

B. Increase monitoring and reduce replication lag

C. Disable replication and use only master for reads

D. Add more slaves without changing replication

Solution

1. Step 1: Identify cause of inconsistent reads

   Inconsistent reads usually happen due to replication lag causing slaves to have outdated data.

2. Step 2: Fix by reducing replication lag

   Improving monitoring and optimizing replication reduces lag, making slave reads more consistent.

3. Final Answer:

   Increase monitoring and reduce replication lag -> Option B

4. Quick Check:

   Reduce lag to fix inconsistent reads [OK]

Hint: Reduce lag to fix stale slave reads [OK]

Common Mistakes:

• Allowing writes on slaves breaks consistency
• Disabling replication loses read scaling
• Adding slaves without lag fix doesn't help

5. You are designing a high-availability system using master-slave replication. Which approach best ensures minimal downtime and data loss during master failure?

hard

A. Automatically promote a slave to master and redirect writes

B. Stop all writes until the master is manually restored

C. Allow writes on slaves during master downtime

D. Use only one master without slaves to avoid complexity

Solution

1. Step 1: Understand master failure impact

   If the master fails, writes cannot proceed unless a new master is available.

2. Step 2: Use automatic failover

   Promoting a slave to master automatically and redirecting writes ensures availability and minimal data loss.

3. Step 3: Evaluate other options

   Stopping writes causes downtime; writing on slaves breaks consistency; single master lacks redundancy.

4. Final Answer:

   Automatically promote a slave to master and redirect writes -> Option A

5. Quick Check:

   Automatic failover ensures availability [OK]

Hint: Auto-promote slave for failover [OK]

Common Mistakes:

• Stopping writes causes downtime
• Writing on slaves breaks data integrity
• Ignoring failover leads to outages