Why database scaling handles data growth in HLD - Scalability Evidence

Scalability Analysis - Why database scaling handles data growth

Growth Table: Data Growth Impact on Database

Users / Data Size	100 Users	10K Users	1M Users	100M Users
Data Volume	Small (MBs)	Medium (GBs)	Large (TBs)	Very Large (PBs)
Database Load	Low QPS (queries/sec)	Moderate QPS	High QPS	Very High QPS
Response Time	Fast (ms)	Acceptable (tens ms)	Slower (hundreds ms)	Slow (seconds)
Storage Needs	Minimal	Growing	Large Disk Arrays	Distributed Storage Systems
Backup & Recovery	Simple	Scheduled	Complex	Highly Automated

First Bottleneck: Database Storage and Query Performance

As data grows, the database storage and query speed become the first limits. Large data slows down searches and updates. Disk space fills up. Indexes become less efficient. This causes slower responses and higher load on the database server.

Scaling Solutions for Database Data Growth

• Vertical Scaling: Upgrade to bigger servers with more CPU, RAM, and faster disks to handle more data and queries.
• Read Replicas: Create copies of the database to spread read queries and reduce load on the main database.
• Sharding: Split data horizontally across multiple database servers by user ID or other keys to distribute storage and queries.
• Caching: Use in-memory caches like Redis to store frequent query results and reduce database hits.
• Archival: Move old or less-used data to cheaper storage to keep the main database smaller and faster.

Back-of-Envelope Cost Analysis

Assuming 1M users generate 10 QPS each, total 10M QPS is unrealistic for one DB. A single PostgreSQL instance handles ~5K QPS. So, 10M QPS requires ~2000 DB instances or sharding.

Storage: 1M users with 1GB each = 1PB data. This needs distributed storage solutions.

Bandwidth: 1 Gbps network supports ~125 MB/s. Large data transfers require multiple network interfaces or data centers.

Interview Tip: Structuring Database Scaling Discussion

Start by explaining how data growth affects storage and query speed. Identify the first bottleneck (usually DB storage or query load). Then discuss vertical scaling, read replicas, sharding, and caching as solutions. Mention trade-offs and complexity added by each.

Self Check Question

Your database handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first and why?

Key Result

Database scaling handles data growth by addressing storage limits and query performance bottlenecks through vertical scaling, read replicas, sharding, and caching to maintain fast responses as data and traffic increase.

Practice

1. Why do systems use database scaling when data grows larger?

easy

A. To keep data access fast and reliable

B. To reduce the amount of data stored

C. To make data harder to access

D. To delete old data automatically

Solution

1. Step 1: Understand the purpose of scaling

   Database scaling is used to handle more data without slowing down or losing reliability.

2. Step 2: Identify the correct goal

   The main goal is to keep data access fast and reliable as data grows.

3. Final Answer:

   To keep data access fast and reliable -> Option A

4. Quick Check:

   Scaling improves speed and reliability [OK]

Hint: Scaling keeps databases fast despite growth [OK]

Common Mistakes:

• Thinking scaling reduces data size
• Believing scaling deletes old data
• Assuming scaling makes data harder to access

2. Which of the following is a correct example of vertical scaling in databases?

easy

A. Adding more servers to share the load

B. Upgrading a server with a faster CPU and more RAM

C. Splitting data into smaller parts across servers

D. Copying data to multiple servers for backup

Solution

1. Step 1: Define vertical scaling

   Vertical scaling means improving one server's hardware like CPU or RAM.

2. Step 2: Match the option to vertical scaling

   Upgrading a server with better CPU and RAM fits vertical scaling.

3. Final Answer:

   Upgrading a server with a faster CPU and more RAM -> Option B

4. Quick Check:

   Vertical scaling = better single server [OK]

Hint: Vertical scaling upgrades one server's hardware [OK]

Common Mistakes:

• Confusing vertical with horizontal scaling
• Thinking adding servers is vertical scaling
• Mixing sharding with vertical scaling

3. Given a database system that uses sharding, what happens when data grows beyond one shard's capacity?

medium

A. The system slows down because all data is on one server

B. Replication copies data to the same shard

C. Data is deleted to keep the shard size small

D. New shards are added to distribute the data load

Solution

1. Step 1: Understand sharding

   Sharding splits data across multiple servers or shards to spread load.

2. Step 2: What happens when one shard is full?

   New shards are added to hold more data and keep performance.

3. Final Answer:

   New shards are added to distribute the data load -> Option D

4. Quick Check:

   Sharding adds shards as data grows [OK]

Hint: Sharding adds servers to handle more data [OK]

Common Mistakes:

• Thinking data is deleted when shard is full
• Believing replication happens on same shard
• Assuming system slows down without adding shards

4. A database uses replication to improve read speed and backup. If replication is misconfigured, what problem might occur?

medium

A. Data inconsistency between primary and replicas

B. Faster writes on the primary database

C. Automatic deletion of all replicas

D. Sharding data across replicas

Solution

1. Step 1: Understand replication purpose

   Replication copies data to replicas for speed and backup.

2. Step 2: Identify misconfiguration effects

   If replication is wrong, replicas may not match primary, causing inconsistency.

3. Final Answer:

   Data inconsistency between primary and replicas -> Option A

4. Quick Check:

   Replication errors cause inconsistency [OK]

Hint: Misconfigured replication causes data mismatch [OK]

Common Mistakes:

• Thinking replication speeds up writes
• Assuming replicas get deleted automatically
• Confusing sharding with replication

5. A company expects its user data to grow rapidly. Which combination of scaling techniques best handles this growth while ensuring fast reads and fault tolerance?

hard

A. Vertical scaling only, upgrading server hardware

B. Replication only, copying data to multiple servers

C. Horizontal scaling with sharding and replication

D. Deleting old data regularly to reduce size

Solution

1. Step 1: Analyze growth and requirements

   Rapid data growth needs spreading data and fast access with backups.

2. Step 2: Match techniques to needs

   Horizontal scaling adds servers; sharding splits data; replication copies data for speed and fault tolerance.

3. Step 3: Evaluate other options

   Vertical scaling alone can't handle rapid growth well; replication alone doesn't split data; deleting data loses information.

4. Final Answer:

   Horizontal scaling with sharding and replication -> Option C

5. Quick Check:

   Sharding + replication = scalable and reliable [OK]

Hint: Combine sharding and replication for growth and speed [OK]

Common Mistakes:

• Relying only on vertical scaling for big growth
• Using replication without sharding for large data
• Deleting data instead of scaling