Shard sizing strategy in Elasticsearch - Step-by-Step Execution

Concept Flow - Shard sizing strategy

Estimate total data size

↓

Decide shard size target

↓

Calculate number of shards needed

↓

Create index with calculated shards

↓

Monitor shard performance & adjust if needed

This flow shows how to plan shard sizes by estimating data, choosing shard size, calculating shards, creating index, and monitoring.

Execution Sample

Elasticsearch

total_data_size_gb = 100
shard_size_gb = 30
num_shards = (total_data_size_gb + shard_size_gb - 1) // shard_size_gb
print(f"Number of shards needed: {num_shards}")

Calculate how many shards are needed for 100GB data if each shard targets 30GB.

Execution Table

Step	Variable	Value	Calculation/Condition	Action/Output
1	total_data_size_gb	100	Given total data size	Store 100GB
2	shard_size_gb	30	Target shard size	Store 30GB
3	num_shards	4	(100 + 30 - 1) // 30 = 129 // 30	Calculate shards needed
4	print	Number of shards needed: 4	Output result	Show number of shards
5	-	-	-	Execution ends

💡 Calculation completes when number of shards is determined and printed.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
total_data_size_gb	undefined	100	100	100	100
shard_size_gb	undefined	undefined	30	30	30
num_shards	undefined	undefined	undefined	4	4

Key Moments - 2 Insights

Why do we add shard_size_gb - 1 before dividing?

Adding shard_size_gb - 1 ensures we round up the division to get enough shards, as shown in step 3 of the execution_table.

Can shard size be too small or too large?

Yes, too small shards cause overhead, too large shards slow queries. The target shard size balances performance and resource use, as chosen in step 2.

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of num_shards after step 3?

A 3

B 4

C 5

D 30

💡 Hint

Check the 'num_shards' value in row 3 of the execution_table.

At which step is the shard size defined?

A Step 2

B Step 1

C Step 3

D Step 4

💡 Hint

Look for 'shard_size_gb' assignment in the execution_table.

If total_data_size_gb was 90 instead of 100, how many shards would be needed?

A 2

B 4

C 3

D 5

💡 Hint

Use the formula from step 3 in execution_table with 90GB data.

Concept Snapshot

Shard sizing strategy:
- Estimate total data size (e.g., 100GB)
- Choose target shard size (e.g., 30GB)
- Calculate shards: (total + target -1) // target
- Create index with calculated shards
- Monitor and adjust shard size if needed

Full Transcript

Shard sizing strategy in Elasticsearch involves estimating the total data size you expect to store. Then, you pick a target shard size that balances performance and resource use, often around 20-50GB. Next, calculate how many shards you need by dividing total data size by shard size, rounding up to ensure enough shards. Finally, create your index with that number of shards and watch performance to adjust if necessary. For example, if you have 100GB data and want 30GB shards, you calculate (100 + 30 - 1) // 30 = 4 shards. This method helps keep shards efficient and manageable.