Why database scaling handles data growth in HLD - Why This Architecture

Problem Statement

When a database grows too large for a single server, it slows down queries and updates. The server can run out of memory, CPU, or disk space, causing delays or crashes. This makes the system unreliable and unable to handle more users or data.

Solution

Database scaling spreads data and workload across multiple servers or resources. This reduces the load on any single server, allowing the system to handle more data and users smoothly. It can be done by adding more power to one server or by distributing data across many servers.

Architecture

Client App

→Load Balancer

↓

Database

Database

↓

Database

Database

This diagram shows a client sending requests through a load balancer that distributes queries across multiple database servers to handle growing data and traffic.

Trade-offs

✓ Pros

→

Improves performance by distributing load across servers.

→

Increases storage capacity by adding more servers.

→

Enhances fault tolerance; if one server fails, others continue working.

✗ Cons

→

Adds complexity in managing data consistency across servers.

→

Requires more infrastructure and operational effort.

→

Can introduce latency due to network communication between servers.

Use when database size or traffic exceeds the capacity of a single server, typically above tens of thousands of queries per second or terabytes of data.

Avoid if your database workload is small (under a few thousand queries per second) or data size fits comfortably on one server, as scaling adds unnecessary complexity.

Real World Examples

Amazon

Amazon uses database sharding to split customer data across servers, enabling fast order processing despite massive data growth.

Twitter

Twitter scales its databases horizontally to handle billions of tweets and user interactions daily without slowing down.

Uber

Uber distributes ride and user data across multiple databases to maintain low latency and high availability as demand grows.

Alternatives

Vertical Scaling

Increases capacity by upgrading a single server's hardware instead of adding more servers.

Use when: Choose when workload growth is moderate and upgrading hardware is simpler and cost-effective.

Caching

Stores frequently accessed data in fast storage to reduce database load without changing database size.

Use when: Choose when read traffic is high but data size is manageable on a single server.

Summary

Database scaling prevents slowdowns and crashes caused by data growth by spreading load across resources.

It improves performance and capacity but adds complexity in management and consistency.

Choosing the right scaling method depends on workload size, growth rate, and system requirements.

Practice

1. Why do systems use database scaling when data grows larger?

easy

A. To keep data access fast and reliable

B. To reduce the amount of data stored

C. To make data harder to access

D. To delete old data automatically

Solution

1. Step 1: Understand the purpose of scaling

   Database scaling is used to handle more data without slowing down or losing reliability.

2. Step 2: Identify the correct goal

   The main goal is to keep data access fast and reliable as data grows.

3. Final Answer:

   To keep data access fast and reliable -> Option A

4. Quick Check:

   Scaling improves speed and reliability [OK]

Hint: Scaling keeps databases fast despite growth [OK]

Common Mistakes:

• Thinking scaling reduces data size
• Believing scaling deletes old data
• Assuming scaling makes data harder to access

2. Which of the following is a correct example of vertical scaling in databases?

easy

A. Adding more servers to share the load

B. Upgrading a server with a faster CPU and more RAM

C. Splitting data into smaller parts across servers

D. Copying data to multiple servers for backup

Solution

1. Step 1: Define vertical scaling

   Vertical scaling means improving one server's hardware like CPU or RAM.

2. Step 2: Match the option to vertical scaling

   Upgrading a server with better CPU and RAM fits vertical scaling.

3. Final Answer:

   Upgrading a server with a faster CPU and more RAM -> Option B

4. Quick Check:

   Vertical scaling = better single server [OK]

Hint: Vertical scaling upgrades one server's hardware [OK]

Common Mistakes:

• Confusing vertical with horizontal scaling
• Thinking adding servers is vertical scaling
• Mixing sharding with vertical scaling

3. Given a database system that uses sharding, what happens when data grows beyond one shard's capacity?

medium

A. The system slows down because all data is on one server

B. Replication copies data to the same shard

C. Data is deleted to keep the shard size small

D. New shards are added to distribute the data load

Solution

1. Step 1: Understand sharding

   Sharding splits data across multiple servers or shards to spread load.

2. Step 2: What happens when one shard is full?

   New shards are added to hold more data and keep performance.

3. Final Answer:

   New shards are added to distribute the data load -> Option D

4. Quick Check:

   Sharding adds shards as data grows [OK]

Hint: Sharding adds servers to handle more data [OK]

Common Mistakes:

• Thinking data is deleted when shard is full
• Believing replication happens on same shard
• Assuming system slows down without adding shards

4. A database uses replication to improve read speed and backup. If replication is misconfigured, what problem might occur?

medium

A. Data inconsistency between primary and replicas

B. Faster writes on the primary database

C. Automatic deletion of all replicas

D. Sharding data across replicas

Solution

1. Step 1: Understand replication purpose

   Replication copies data to replicas for speed and backup.

2. Step 2: Identify misconfiguration effects

   If replication is wrong, replicas may not match primary, causing inconsistency.

3. Final Answer:

   Data inconsistency between primary and replicas -> Option A

4. Quick Check:

   Replication errors cause inconsistency [OK]

Hint: Misconfigured replication causes data mismatch [OK]

Common Mistakes:

• Thinking replication speeds up writes
• Assuming replicas get deleted automatically
• Confusing sharding with replication

5. A company expects its user data to grow rapidly. Which combination of scaling techniques best handles this growth while ensuring fast reads and fault tolerance?

hard

A. Vertical scaling only, upgrading server hardware

B. Replication only, copying data to multiple servers

C. Horizontal scaling with sharding and replication

D. Deleting old data regularly to reduce size

Solution

1. Step 1: Analyze growth and requirements

   Rapid data growth needs spreading data and fast access with backups.

2. Step 2: Match techniques to needs

   Horizontal scaling adds servers; sharding splits data; replication copies data for speed and fault tolerance.

3. Step 3: Evaluate other options

   Vertical scaling alone can't handle rapid growth well; replication alone doesn't split data; deleting data loses information.

4. Final Answer:

   Horizontal scaling with sharding and replication -> Option C

5. Quick Check:

   Sharding + replication = scalable and reliable [OK]

Hint: Combine sharding and replication for growth and speed [OK]

Common Mistakes:

• Relying only on vertical scaling for big growth
• Using replication without sharding for large data
• Deleting data instead of scaling