Shard key selection in HLD - System Design Guide

Problem Statement

When a database grows too large, a single server cannot handle all the data or requests. Without a good way to split data, some servers get overloaded while others stay idle, causing slow responses and failures.

Solution

Shard key selection divides data into parts based on a chosen key so each server stores only a portion. This key decides where data goes, balancing load and making queries efficient by directing them to the right server.

Architecture

Client App

Client App

↓

Shard Router

Shard Router

↓

Shard1

This diagram shows a client sending requests to a shard router, which uses the shard key to direct each request to the correct shard server holding the relevant data.

Trade-offs

✓ Pros

→

Distributes data evenly to avoid hotspots and overload on any single server.

→

Improves query speed by routing requests directly to the shard containing the data.

→

Enables horizontal scaling by adding more shards as data grows.

→

Reduces single points of failure by isolating data partitions.

✗ Cons

→

Choosing a poor shard key can cause uneven data distribution and hotspots.

→

Changing shard keys later is complex and may require data reshuffling.

→

Cross-shard queries become more complicated and slower.

→

Requires careful planning to balance load and query patterns.

Use when data size or request volume exceeds the capacity of a single database server, typically above millions of records or thousands of requests per second.

Avoid if data size is small or query patterns require frequent cross-shard joins, as complexity and overhead outweigh benefits.

Real World Examples

Twitter

Twitter uses user ID as a shard key to distribute tweets across multiple servers, ensuring balanced load and fast access to user timelines.

Amazon

Amazon shards product catalog data by product category to distribute storage and query load efficiently across servers.

Uber

Uber shards ride data by geographic region to localize data access and reduce latency for regional queries.

Alternatives

Range-based sharding

Splits data by continuous ranges of shard key values rather than hashing them.

Use when: Choose when queries often target specific ranges and data distribution is predictable.

Directory-based sharding

Uses a lookup table to map data keys to shards instead of computing shard location from the key.

Use when: Choose when shard key distribution is uneven or dynamic and requires flexible mapping.

No sharding (single database)

Stores all data on one server without partitioning.

Use when: Choose when data size and traffic are low and simplicity is preferred.

Summary

Shard key selection decides how data is split across multiple servers to balance load and improve performance.

A good shard key evenly distributes data and aligns with common query patterns to avoid hotspots.

Poor shard key choices cause uneven load, complex queries, and difficult scaling.

Practice

1. What is the main purpose of selecting a shard key in a distributed database system?

easy

A. To compress data to save storage space

B. To encrypt data for security purposes

C. To backup data automatically

D. To split data across servers for better load distribution and faster access

Solution

1. Step 1: Understand shard key role

   The shard key determines how data is divided across multiple servers or shards.

2. Step 2: Identify main benefit

   Splitting data helps balance load and speeds up data retrieval by parallelizing queries.

3. Final Answer:

   To split data across servers for better load distribution and faster access -> Option D

4. Quick Check:

   Shard key purpose = split data load [OK]

Hint: Shard key splits data to spread load and speed up access [OK]

Common Mistakes:

• Confusing shard key with encryption key
• Thinking shard key compresses data
• Assuming shard key is for backup

2. Which of the following is a good characteristic of a shard key?

easy

A. Has very few unique values

B. Has many unique values and distributes data evenly

C. Is always a timestamp

D. Is a large text field

Solution

1. Step 1: Identify shard key traits

   A good shard key should have many unique values to avoid data concentration.

2. Step 2: Understand distribution importance

   Even distribution prevents hotspots and balances load across shards.

3. Final Answer:

   Has many unique values and distributes data evenly -> Option B

4. Quick Check:

   Good shard key = unique + even distribution [OK]

Hint: Choose shard keys with many unique values and even spread [OK]

Common Mistakes:

• Picking keys with few unique values causing hotspots
• Assuming timestamp is always best shard key
• Using large text fields that slow down indexing

3. Consider a user database sharded by country_code which has only 5 unique values. What is the likely problem with this shard key?

medium

A. Data will be evenly distributed across shards

B. It will encrypt user data automatically

C. Some shards will be overloaded due to uneven data distribution

D. Queries will be faster because of fewer shards

Solution

1. Step 1: Analyze unique values count

   Only 5 unique values means few shards and many users per shard.

2. Step 2: Understand load distribution

   Popular countries will have more users, causing some shards to be overloaded.

3. Final Answer:

   Some shards will be overloaded due to uneven data distribution -> Option C

4. Quick Check:

   Few unique keys = uneven shard load [OK]

Hint: Few unique shard keys cause uneven load and hotspots [OK]

Common Mistakes:

• Assuming fewer shards always improve speed
• Thinking shard key encrypts data
• Ignoring uneven data distribution effects

4. A system uses user_id % 10 as a shard key but notices uneven shard sizes. What is the likely cause?

medium

A. User IDs are not uniformly distributed modulo 10

B. Shard key is too unique

C. Shard key is a string instead of number

D. System is encrypting shard keys

Solution

1. Step 1: Understand modulo shard key

   Modulo operation expects uniform distribution of user IDs.

2. Step 2: Identify distribution problem

   If user IDs cluster in certain ranges, modulo results cluster causing uneven shards.

3. Final Answer:

   User IDs are not uniformly distributed modulo 10 -> Option A

4. Quick Check:

   Non-uniform IDs cause uneven modulo shard keys [OK]

Hint: Modulo shard keys need uniform input distribution [OK]

Common Mistakes:

• Assuming modulo always balances shards
• Confusing shard key type with encryption
• Ignoring data distribution patterns

5. You design a sharded system for an e-commerce app. Users query orders mostly by user_id and order_date. Which shard key choice best supports fast queries and balanced load?

hard

A. Shard by a composite key of user_id and order_date

B. Shard by user_id only

C. Shard by order_date only

D. Shard by a random UUID

Solution

1. Step 1: Match shard key to query patterns

   Users query by both user_id and order_date, so shard key should support both.

2. Step 2: Evaluate shard key options

   Sharding by user_id alone may cause hotspots; order_date alone may cause uneven load by time.

3. Step 3: Use composite key benefits

   Composite key balances load and supports queries filtering by either field efficiently.

4. Final Answer:

   Shard by a composite key of user_id and order_date -> Option A

5. Quick Check:

   Composite shard key matches queries and balances load [OK]

Hint: Composite keys balance load and match query patterns best [OK]

Common Mistakes:

• Choosing single field shard keys ignoring query patterns
• Using random UUID causing inefficient queries
• Ignoring load balancing in shard key choice