Problem Statement
When a single database grows too large, it becomes slow to query and update. This causes long wait times for users and risks total failure if the database crashes, since all data is in one place.
0
0
Database sharding strategies in HLD - System Design Guide
Solution
Database sharding splits data into smaller parts called shards, each stored on different servers. Requests are routed to the correct shard based on a key, so queries and writes happen in parallel, improving speed and reliability.
Architecture
Client App
→Shard Router
Shard 2 DB
Shard 2 DB
Shard 3 DB
Shard 3 DB
This diagram shows a client sending requests to a shard router, which directs queries to the correct database shard based on the data key.
Trade-offs
✓ Pros
Improves database performance by distributing load across multiple servers.
Increases system availability since failure of one shard doesn't affect others.
Enables horizontal scaling by adding more shards as data grows.
Reduces contention and lock conflicts by isolating data subsets.
✗ Cons
Adds complexity in routing queries to the correct shard.
Cross-shard queries are difficult and often inefficient.
Requires careful shard key selection to avoid uneven data distribution.
Data rebalancing when adding or removing shards can be complex and costly.
Use when database size or traffic exceeds the capacity of a single server, typically above millions of records or thousands of queries per second.
Avoid if your database is small or traffic is low (under 1,000 queries per second), as sharding complexity outweighs benefits.
Real World Examples
Twitter
Twitter shards user data by user ID to distribute load and reduce latency for timeline queries.
Amazon
Amazon shards product catalog data by product category to enable parallel processing and faster search.
Uber
Uber shards trip data by geographic region to localize data and improve query speed for rides in specific cities.
Alternatives
Vertical partitioning
Splits database tables by columns instead of rows, storing different columns on different servers.
Use when:
Choose when different parts of data have very different access patterns or sizes.
Replication
Copies entire database to multiple servers for read scaling and failover, without splitting data.
Use when:
Choose when read traffic is high but write traffic is low or can be handled by a single master.
Federated database
Combines multiple independent databases under a unified interface without physically splitting data.
Use when:
Choose when integrating legacy or heterogeneous databases without centralizing data.
Summary
Database sharding splits large databases into smaller parts to improve performance and reliability.
It requires careful shard key selection and adds complexity in query routing and data management.
Sharding is best for very large or high-traffic databases but not suitable for small-scale systems.