Index settings (shards, replicas) in Elasticsearch - Deep Dive

Overview - Index settings (shards, replicas)

What is it?

Index settings in Elasticsearch control how data is stored and managed. Two key settings are shards and replicas. Shards split the data into parts to spread across servers, while replicas are copies of these parts for safety and speed. These settings help Elasticsearch handle large amounts of data efficiently and reliably.

Why it matters

Without shards and replicas, Elasticsearch would struggle to manage big data or recover from failures. Shards allow data to be divided and searched quickly, while replicas protect against data loss and improve search speed. Without these, searches would be slow, and data could be lost if a server fails.

Where it fits

Before learning index settings, you should understand what an Elasticsearch index is and basic cluster concepts. After this, you can learn about query optimization and cluster scaling. Index settings are a foundation for managing data distribution and reliability in Elasticsearch.

Mental Model

Core Idea

Shards split data into pieces for distribution, and replicas copy those pieces for safety and faster access.

Think of it like...

Imagine a big book split into chapters (shards) stored in different libraries. Each chapter has backup copies (replicas) in other libraries to prevent loss and allow more readers to access the book quickly.

┌───────────────┐
│ Elasticsearch  │
│    Index      │
├───────────────┤
│ Shard 1       │───┐
│ Shard 2       │   │  Distributed across servers
│ Shard 3       │───┘
├───────────────┤
│ Replica 1 of 1│
│ Replica 1 of 2│  Copies for safety and speed
│ Replica 1 of 3│

Build-Up - 7 Steps

1

Foundation

What is an Elasticsearch index

Concept: Introduce the basic concept of an index as a collection of documents.

An Elasticsearch index is like a database table that stores documents. Each document is a piece of data with fields. The index organizes and stores these documents so you can search them quickly.

Result

You understand that an index holds data and is the main unit you work with in Elasticsearch.

Knowing what an index is helps you see why splitting and copying data matters for performance and safety.

2

Foundation

Understanding shards as data partitions

Concept: Explain shards as parts that split the index data for distribution.

A shard is a smaller piece of an index. Elasticsearch breaks an index into shards to spread data across multiple servers. This lets Elasticsearch handle more data and search faster by working on shards in parallel.

Result

You see that shards divide data to make storage and searching scalable.

Understanding shards reveals how Elasticsearch manages big data by splitting it into manageable parts.

3

Intermediate

Role of replicas for safety and speed

🤔Before reading on: do you think replicas only protect data or also improve search speed? Commit to your answer.

Concept: Introduce replicas as copies of shards that provide fault tolerance and improve search performance.

Replicas are exact copies of shards stored on different servers. They protect data if a server fails and allow Elasticsearch to handle more search requests by querying replicas in parallel.

Result

You understand replicas serve two purposes: data safety and faster searches.

Knowing replicas improve both reliability and speed helps you design resilient and efficient clusters.

4

Intermediate

Configuring shard and replica counts

🤔Before reading on: do you think you can change shard count after index creation? Commit to your answer.

Concept: Explain how to set the number of shards and replicas when creating an index and the limitations.

When creating an index, you choose how many primary shards it has and how many replicas per shard. Shard count is fixed after creation, but replica count can be changed anytime. Choosing the right numbers balances performance, storage, and fault tolerance.

Result

You learn how to configure shards and replicas and the impact of these choices.

Understanding configuration limits prevents costly mistakes and helps plan for growth.

5

Intermediate

How shards and replicas affect search and indexing

Concept: Show how queries and data writes interact with shards and replicas.

When you index data, Elasticsearch writes it to primary shards first, then copies to replicas. When searching, queries go to all shards and replicas to find results quickly. This parallelism speeds up searches and keeps data safe.

Result

You see the flow of data and queries through shards and replicas.

Knowing this flow clarifies why shards and replicas are essential for Elasticsearch's speed and reliability.

6

Advanced

Shard allocation and balancing in clusters

🤔Before reading on: do you think Elasticsearch places shards randomly or with a strategy? Commit to your answer.

Concept: Explain how Elasticsearch decides where to place shards and replicas across servers.

Elasticsearch uses allocation rules to spread shards and replicas evenly across nodes. It avoids placing a replica on the same node as its primary shard to prevent data loss if a node fails. It also balances load to optimize performance.

Result

You understand shard placement strategies that keep data safe and cluster balanced.

Knowing allocation strategies helps you troubleshoot cluster issues and optimize resource use.

7

Expert

Impact of shard size and count on cluster health

🤔Before reading on: do you think more shards always mean better performance? Commit to your answer.

Concept: Discuss how shard size and number affect cluster performance and resource use.

Too many small shards increase overhead and slow down the cluster, while too few large shards can cause bottlenecks. Experts balance shard size (often 10-50GB) and count to optimize speed, memory use, and recovery times.

Result

You learn the trade-offs in shard sizing and how it affects cluster health.

Understanding shard sizing prevents common performance problems and guides scaling decisions.

Under the Hood

Elasticsearch stores data in Lucene segments inside shards. Each shard is a self-contained index. When data is indexed, it writes to the primary shard's segments, then replicates to replica shards asynchronously. The cluster state tracks shard locations and health. Queries fan out to all shards and replicas, merging results before returning.

Why designed this way?

Sharding was designed to scale Elasticsearch horizontally, allowing it to handle huge data volumes by splitting work. Replicas provide fault tolerance and load balancing. This design balances speed, reliability, and scalability, unlike monolithic indexes that can't scale well or recover quickly.

┌───────────────┐
│ Cluster State │
└──────┬────────┘
       │ Tracks shard locations
┌──────▼───────┐       ┌───────────────┐
│ Primary Shard│──────▶│ Replica Shard │
│ (Lucene idx) │       │ (Copy of data)│
└──────────────┘       └───────────────┘
       ▲                      ▲
       │                      │
   Writes data           Replicates data
       │                      │
┌──────▼───────┐       ┌───────────────┐
│ Search Query │──────▶│ Search Query  │
│ fan out to   │       │ fan out to    │
│ all shards   │       │ all replicas  │
└──────────────┘       └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Can you change the number of primary shards after index creation? Commit to yes or no.

Common Belief: You can change the number of primary shards anytime to scale your index.

Reality: Primary shard count is fixed at index creation and cannot be changed later.

Why it matters: Trying to change shard count later leads to confusion and forces costly reindexing or data migration.

Quick: Do replicas only exist for backup and never improve search speed? Commit to yes or no.

Common Belief: Replicas are only for data safety and do not affect search performance.

Reality: Replicas also serve search requests, improving query throughput and reducing latency.

Why it matters: Ignoring replicas' role in search can lead to underutilized resources and slower query response.

Quick: Does having more shards always make your cluster faster? Commit to yes or no.

Common Belief: More shards always mean better performance because data is split more.

Reality: Too many shards cause overhead and slow down the cluster due to resource strain.

Why it matters: Over-sharding wastes memory and CPU, causing slower searches and cluster instability.

Quick: Are replicas stored on the same server as their primary shards? Commit to yes or no.

Common Belief: Replicas can be on the same server as primary shards since they are just copies.

Reality: Replicas are placed on different servers to prevent data loss if one server fails.

Why it matters: Placing replicas on the same server defeats fault tolerance and risks data loss.

Expert Zone

1

Shard count affects cluster state size and recovery time, not just data distribution.

2

Replica shards can be promoted to primary automatically if a primary shard fails, ensuring high availability.

3

Shard allocation awareness can be customized with node attributes to optimize hardware usage and data locality.

When NOT to use

Avoid using too many shards for small datasets; instead, use fewer shards or a single shard index. For very large datasets, consider index lifecycle management and rollover instead of huge shards. If you need real-time analytics, consider specialized data stores alongside Elasticsearch.

Production Patterns

In production, teams often start with a small shard count and increase replicas for fault tolerance. They monitor shard sizes to keep them balanced around 20-40GB. Rolling upgrades and shard rebalancing are used to maintain cluster health without downtime.

Connections

Distributed Systems

Builds-on

Understanding shards and replicas deepens knowledge of data partitioning and replication, core ideas in distributed computing.

RAID Storage Arrays

Similar pattern

Replicas in Elasticsearch are like RAID mirrors, providing redundancy and improving read speed by duplicating data.

Library Book Management

Analogy to real-world system

Managing shards and replicas is like organizing books across libraries with backups, helping grasp data distribution and fault tolerance.

Common Pitfalls

#1 Setting too many shards for a small dataset.

Wrong approach:

PUT /myindex
{
"settings": {
"number_of_shards": 50,
"number_of_replicas": 1
}
}

Correct approach:

PUT /myindex
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1
}
}

Root cause:

Misunderstanding that more shards always improve performance leads to excessive overhead.

#2 Trying to change primary shard count after index creation.

Wrong approach:

PUT /myindex/_settings
{
"number_of_shards": 10
}

Correct approach:

Create a new index with desired shard count and reindex data:
PUT /newindex
{
"settings": {"number_of_shards": 10}
}
POST /_reindex
{
"source": {"index": "myindex"},
"dest": {"index": "newindex"}
}

Root cause:

Not knowing shard count is immutable causes attempts to change it directly.

#3 Placing replicas on the same node as primary shards.

Wrong approach:

Manually assigning shards without awareness, causing primary and replica on same node.

Correct approach:

Use default allocation awareness or configure node attributes to ensure replicas are on different nodes.

Root cause:

Ignoring shard allocation rules risks data loss and reduces fault tolerance.

Key Takeaways

Shards split an Elasticsearch index into parts to distribute data and speed up searches.

Replicas are copies of shards that protect data and improve search performance by handling queries.

Shard count is fixed at index creation, but replica count can be changed anytime to adjust fault tolerance.

Proper shard sizing and allocation are critical for cluster health, performance, and reliability.

Misconfiguring shards or replicas can cause slow searches, data loss risk, and cluster instability.