Index settings (shards, replicas) in Elasticsearch - Deep Dive

Overview - Index settings (shards, replicas)

What is it?

Index settings in Elasticsearch control how data is stored and managed. Two key settings are shards and replicas. Shards split the data into parts to spread across servers, while replicas are copies of these parts for safety and speed. These settings help Elasticsearch handle large amounts of data efficiently and reliably.

Why it matters

Without shards and replicas, Elasticsearch would struggle to manage big data or recover from failures. Shards allow data to be divided and searched quickly, while replicas protect against data loss and improve search speed. Without these, searches would be slow, and data could be lost if a server fails.

Where it fits

Before learning index settings, you should understand what an Elasticsearch index is and basic cluster concepts. After this, you can learn about query optimization and cluster scaling. Index settings are a foundation for managing data distribution and reliability in Elasticsearch.

Mental Model

Core Idea

Shards split data into pieces for distribution, and replicas copy those pieces for safety and faster access.

Think of it like...

Imagine a big book split into chapters (shards) stored in different libraries. Each chapter has backup copies (replicas) in other libraries to prevent loss and allow more readers to access the book quickly.

┌───────────────┐
│ Elasticsearch  │
│    Index      │
├───────────────┤
│ Shard 1       │───┐
│ Shard 2       │   │  Distributed across servers
│ Shard 3       │───┘
├───────────────┤
│ Replica 1 of 1│
│ Replica 1 of 2│  Copies for safety and speed
│ Replica 1 of 3│

Build-Up - 7 Steps

1

Foundation

What is an Elasticsearch index

Concept: Introduce the basic concept of an index as a collection of documents.

An Elasticsearch index is like a database table that stores documents. Each document is a piece of data with fields. The index organizes and stores these documents so you can search them quickly.

Result

You understand that an index holds data and is the main unit you work with in Elasticsearch.

Knowing what an index is helps you see why splitting and copying data matters for performance and safety.

2

Foundation

Understanding shards as data partitions

Concept: Explain shards as parts that split the index data for distribution.

A shard is a smaller piece of an index. Elasticsearch breaks an index into shards to spread data across multiple servers. This lets Elasticsearch handle more data and search faster by working on shards in parallel.

Result

You see that shards divide data to make storage and searching scalable.

Understanding shards reveals how Elasticsearch manages big data by splitting it into manageable parts.

3

Intermediate

Role of replicas for safety and speed

🤔Before reading on: do you think replicas only protect data or also improve search speed? Commit to your answer.

Concept: Introduce replicas as copies of shards that provide fault tolerance and improve search performance.

Replicas are exact copies of shards stored on different servers. They protect data if a server fails and allow Elasticsearch to handle more search requests by querying replicas in parallel.

Result

You understand replicas serve two purposes: data safety and faster searches.

Knowing replicas improve both reliability and speed helps you design resilient and efficient clusters.

4

Intermediate

Configuring shard and replica counts

🤔Before reading on: do you think you can change shard count after index creation? Commit to your answer.

Concept: Explain how to set the number of shards and replicas when creating an index and the limitations.

When creating an index, you choose how many primary shards it has and how many replicas per shard. Shard count is fixed after creation, but replica count can be changed anytime. Choosing the right numbers balances performance, storage, and fault tolerance.

Result

You learn how to configure shards and replicas and the impact of these choices.

Understanding configuration limits prevents costly mistakes and helps plan for growth.

5

Intermediate

How shards and replicas affect search and indexing

Concept: Show how queries and data writes interact with shards and replicas.

When you index data, Elasticsearch writes it to primary shards first, then copies to replicas. When searching, queries go to all shards and replicas to find results quickly. This parallelism speeds up searches and keeps data safe.

Result

You see the flow of data and queries through shards and replicas.

Knowing this flow clarifies why shards and replicas are essential for Elasticsearch's speed and reliability.

6

Advanced

Shard allocation and balancing in clusters

🤔Before reading on: do you think Elasticsearch places shards randomly or with a strategy? Commit to your answer.

Concept: Explain how Elasticsearch decides where to place shards and replicas across servers.

Elasticsearch uses allocation rules to spread shards and replicas evenly across nodes. It avoids placing a replica on the same node as its primary shard to prevent data loss if a node fails. It also balances load to optimize performance.

Result

You understand shard placement strategies that keep data safe and cluster balanced.

Knowing allocation strategies helps you troubleshoot cluster issues and optimize resource use.

7

Expert

Impact of shard size and count on cluster health

🤔Before reading on: do you think more shards always mean better performance? Commit to your answer.

Concept: Discuss how shard size and number affect cluster performance and resource use.

Too many small shards increase overhead and slow down the cluster, while too few large shards can cause bottlenecks. Experts balance shard size (often 10-50GB) and count to optimize speed, memory use, and recovery times.

Result

You learn the trade-offs in shard sizing and how it affects cluster health.

Understanding shard sizing prevents common performance problems and guides scaling decisions.

Under the Hood

Elasticsearch stores data in Lucene segments inside shards. Each shard is a self-contained index. When data is indexed, it writes to the primary shard's segments, then replicates to replica shards asynchronously. The cluster state tracks shard locations and health. Queries fan out to all shards and replicas, merging results before returning.

Why designed this way?

Sharding was designed to scale Elasticsearch horizontally, allowing it to handle huge data volumes by splitting work. Replicas provide fault tolerance and load balancing. This design balances speed, reliability, and scalability, unlike monolithic indexes that can't scale well or recover quickly.

┌───────────────┐
│ Cluster State │
└──────┬────────┘
       │ Tracks shard locations
┌──────▼───────┐       ┌───────────────┐
│ Primary Shard│──────▶│ Replica Shard │
│ (Lucene idx) │       │ (Copy of data)│
└──────────────┘       └───────────────┘
       ▲                      ▲
       │                      │
   Writes data           Replicates data
       │                      │
┌──────▼───────┐       ┌───────────────┐
│ Search Query │──────▶│ Search Query  │
│ fan out to   │       │ fan out to    │
│ all shards   │       │ all replicas  │
└──────────────┘       └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Can you change the number of primary shards after index creation? Commit to yes or no.

Common Belief: You can change the number of primary shards anytime to scale your index.

Reality: Primary shard count is fixed at index creation and cannot be changed later.

Why it matters: Trying to change shard count later leads to confusion and forces costly reindexing or data migration.

Quick: Do replicas only exist for backup and never improve search speed? Commit to yes or no.

Common Belief: Replicas are only for data safety and do not affect search performance.

Reality: Replicas also serve search requests, improving query throughput and reducing latency.

Why it matters: Ignoring replicas' role in search can lead to underutilized resources and slower query response.

Quick: Does having more shards always make your cluster faster? Commit to yes or no.

Common Belief: More shards always mean better performance because data is split more.

Reality: Too many shards cause overhead and slow down the cluster due to resource strain.

Why it matters: Over-sharding wastes memory and CPU, causing slower searches and cluster instability.

Quick: Are replicas stored on the same server as their primary shards? Commit to yes or no.

Common Belief: Replicas can be on the same server as primary shards since they are just copies.

Reality: Replicas are placed on different servers to prevent data loss if one server fails.

Why it matters: Placing replicas on the same server defeats fault tolerance and risks data loss.

Expert Zone

1

Shard count affects cluster state size and recovery time, not just data distribution.

2

Replica shards can be promoted to primary automatically if a primary shard fails, ensuring high availability.

3

Shard allocation awareness can be customized with node attributes to optimize hardware usage and data locality.

When NOT to use

Avoid using too many shards for small datasets; instead, use fewer shards or a single shard index. For very large datasets, consider index lifecycle management and rollover instead of huge shards. If you need real-time analytics, consider specialized data stores alongside Elasticsearch.

Production Patterns

In production, teams often start with a small shard count and increase replicas for fault tolerance. They monitor shard sizes to keep them balanced around 20-40GB. Rolling upgrades and shard rebalancing are used to maintain cluster health without downtime.

Connections

Distributed Systems

Builds-on

Understanding shards and replicas deepens knowledge of data partitioning and replication, core ideas in distributed computing.

RAID Storage Arrays

Similar pattern

Replicas in Elasticsearch are like RAID mirrors, providing redundancy and improving read speed by duplicating data.

Library Book Management

Analogy to real-world system

Managing shards and replicas is like organizing books across libraries with backups, helping grasp data distribution and fault tolerance.

Common Pitfalls

#1 Setting too many shards for a small dataset.

Wrong approach:

PUT /myindex
{
"settings": {
"number_of_shards": 50,
"number_of_replicas": 1
}
}

Correct approach:

PUT /myindex
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1
}
}

Root cause:

Misunderstanding that more shards always improve performance leads to excessive overhead.

#2 Trying to change primary shard count after index creation.

Wrong approach:

PUT /myindex/_settings
{
"number_of_shards": 10
}

Correct approach:

Create a new index with desired shard count and reindex data:
PUT /newindex
{
"settings": {"number_of_shards": 10}
}
POST /_reindex
{
"source": {"index": "myindex"},
"dest": {"index": "newindex"}
}

Root cause:

Not knowing shard count is immutable causes attempts to change it directly.

#3 Placing replicas on the same node as primary shards.

Wrong approach:

Manually assigning shards without awareness, causing primary and replica on same node.

Correct approach:

Use default allocation awareness or configure node attributes to ensure replicas are on different nodes.

Root cause:

Ignoring shard allocation rules risks data loss and reduces fault tolerance.

Key Takeaways

Shards split an Elasticsearch index into parts to distribute data and speed up searches.

Replicas are copies of shards that protect data and improve search performance by handling queries.

Shard count is fixed at index creation, but replica count can be changed anytime to adjust fault tolerance.

Proper shard sizing and allocation are critical for cluster health, performance, and reliability.

Misconfiguring shards or replicas can cause slow searches, data loss risk, and cluster instability.

Practice

1. What is the main purpose of setting number_of_shards in an Elasticsearch index?

easy

A. To define the size of the index

B. To create backup copies of data

C. To split data into parts for faster searching

D. To set the maximum number of documents

Solution

1. Step 1: Understand shards role

   Shards divide the index data into smaller parts to allow parallel searching.

2. Step 2: Differentiate shards from replicas

   Replicas are copies for safety, not for splitting data.

3. Final Answer:

   To split data into parts for faster searching -> Option C

4. Quick Check:

   Shards = split data for speed [OK]

Hint: Shards split data, replicas copy data [OK]

Common Mistakes:

• Confusing shards with replicas
• Thinking shards create backups
• Assuming shards set index size

2. Which of the following is the correct way to set 3 shards and 2 replicas in an Elasticsearch index creation JSON?

easy

A. { "settings": { "number_of_shards": 3, "number_of_replicas": 2 } }

B. { "settings": { "shards": 3, "replicas": 2 } }

C. { "settings": { "shard_count": 3, "replica_count": 2 } }

D. { "settings": { "num_shards": 3, "num_replicas": 2 } }

Solution

1. Step 1: Recall correct setting keys

   The official keys are number_of_shards and number_of_replicas.

2. Step 2: Match keys with options

   Only { "settings": { "number_of_shards": 3, "number_of_replicas": 2 } } uses the exact correct keys and valid JSON format.

3. Final Answer:

   { "settings": { "number_of_shards": 3, "number_of_replicas": 2 } } -> Option A

4. Quick Check:

   Correct keys = number_of_shards, number_of_replicas [OK]

Hint: Use full keys: number_of_shards and number_of_replicas [OK]

Common Mistakes:

• Using shorthand keys like shards or replicas
• Wrong JSON syntax
• Mixing shard and replica counts

3. Given an index with number_of_shards set to 4 and number_of_replicas set to 1, how many total shard copies exist in the cluster?

medium

A. 4

B. 6

C. 5

D. 8

Solution

1. Step 1: Calculate primary shards

   There are 4 primary shards as set.

2. Step 2: Calculate replica shards

   Each primary shard has 1 replica, so 4 replicas total.

3. Step 3: Add primary and replicas

   Total shards = 4 primary + 4 replicas = 8 shards.

4. Final Answer:

   8 -> Option D

5. Quick Check:

   Total shards = primary + replicas = 8 [OK]

Hint: Total shards = shards x (1 + replicas) [OK]

Common Mistakes:

• Counting only primary shards
• Adding shards and replicas incorrectly
• Confusing replicas with shards

4. You tried to create an index with this setting: { "settings": { "number_of_shards": 2, "number_of_replicas": -1 } }. What is the problem?

medium

A. Negative replicas value is invalid

B. Number of shards cannot be 2

C. Missing required field for replicas

D. JSON syntax error

Solution

1. Step 1: Check shards value

   Number of shards is 2, which is valid.

2. Step 2: Check replicas value

   Replicas cannot be negative; -1 is invalid.

3. Final Answer:

   Negative replicas value is invalid -> Option A

4. Quick Check:

   Replicas must be zero or positive [OK]

Hint: Replicas cannot be negative numbers [OK]

Common Mistakes:

• Assuming negative replicas mean no replicas
• Thinking shards must be > 3
• Ignoring JSON correctness

5. You want to create an index optimized for high availability with 5 shards. Which setting combination is best to ensure data safety and fast search?

hard

A. { "number_of_shards": 5, "number_of_replicas": 0 }

B. { "number_of_shards": 5, "number_of_replicas": 1 }

C. { "number_of_shards": 1, "number_of_replicas": 5 }

D. { "number_of_shards": 10, "number_of_replicas": 1 }

Solution

1. Step 1: Understand shards and replicas roles

   Shards split data for speed; replicas provide copies for safety.

2. Step 2: Evaluate options for availability and speed

   { "number_of_shards": 5, "number_of_replicas": 1 } keeps 5 shards for speed and 1 replica for safety, balancing both.

3. Step 3: Reject other options

   The option with zero replicas has no safety, the option with 1 shard is slow for search, the option with 10 shards has overhead.

4. Final Answer:

   { "number_of_shards": 5, "number_of_replicas": 1 } -> Option B

5. Quick Check:

   Balance shards and replicas for speed and safety [OK]

Hint: Balance shards for speed and replicas for safety [OK]

Common Mistakes:

• Setting zero replicas for safety
• Using too few shards for speed
• Over-sharding causing overhead