Index settings (shards, replicas) in Elasticsearch - Mini Project: Build & Apply

Configure Elasticsearch Index Settings for Shards and Replicas

📖 Scenario: You are setting up an Elasticsearch index for a small online bookstore. To ensure good performance and data safety, you need to configure the number of shards and replicas for the index.

🎯 Goal: Create an Elasticsearch index configuration that sets the number of primary shards and replica shards correctly.

📋 What You'll Learn

Create an index settings dictionary with the exact key settings

Inside settings, set number_of_shards to 3

Inside settings, set number_of_replicas to 2

Use the exact key names and values as specified

💡 Why This Matters

🌍 Real World

Elasticsearch indexes need shard and replica settings to manage data distribution and fault tolerance in real applications like search engines and logging systems.

💼 Career

Understanding and configuring index settings is essential for roles like DevOps engineers, backend developers, and data engineers working with Elasticsearch.

1

Create the base index settings dictionary

Create a dictionary called index_settings with a key settings that is an empty dictionary.

Elasticsearch

index_settings = {
    "settings": {
        
    }
}
# Your code here

Hint

Start by creating a dictionary named index_settings with a key settings that holds another dictionary.

2

Add number_of_shards setting

Inside the settings dictionary of index_settings, add the key number_of_shards with the value 3.

Elasticsearch

index_settings = {
    "settings": {
        "number_of_shards": 3
    }
}
# Your code here

Hint

Add the key number_of_shards with value 3 inside the settings dictionary.

3

Add number_of_replicas setting

Inside the settings dictionary of index_settings, add the key number_of_replicas with the value 2 alongside number_of_shards.

Elasticsearch

index_settings = {
    "settings": {
        "number_of_shards": 3,
        "number_of_replicas": 2
    }
}
# Your code here

Hint

Add the key number_of_replicas with value 2 inside the settings dictionary, after number_of_shards.

4

Complete the index settings configuration

Ensure the index_settings dictionary has the settings key with number_of_shards set to 3 and number_of_replicas set to 2. This completes the index configuration.

Elasticsearch

index_settings = {
    "settings": {
        "number_of_shards": 3,
        "number_of_replicas": 2
    }
}
# Your code here

Hint

Double-check that the dictionary has both keys with the correct values to finish the setup.

Practice

1. What is the main purpose of setting number_of_shards in an Elasticsearch index?

easy

A. To define the size of the index

B. To create backup copies of data

C. To split data into parts for faster searching

D. To set the maximum number of documents

Solution

1. Step 1: Understand shards role

   Shards divide the index data into smaller parts to allow parallel searching.

2. Step 2: Differentiate shards from replicas

   Replicas are copies for safety, not for splitting data.

3. Final Answer:

   To split data into parts for faster searching -> Option C

4. Quick Check:

   Shards = split data for speed [OK]

Hint: Shards split data, replicas copy data [OK]

Common Mistakes:

• Confusing shards with replicas
• Thinking shards create backups
• Assuming shards set index size

2. Which of the following is the correct way to set 3 shards and 2 replicas in an Elasticsearch index creation JSON?

easy

A. { "settings": { "number_of_shards": 3, "number_of_replicas": 2 } }

B. { "settings": { "shards": 3, "replicas": 2 } }

C. { "settings": { "shard_count": 3, "replica_count": 2 } }

D. { "settings": { "num_shards": 3, "num_replicas": 2 } }

Solution

1. Step 1: Recall correct setting keys

   The official keys are number_of_shards and number_of_replicas.

2. Step 2: Match keys with options

   Only { "settings": { "number_of_shards": 3, "number_of_replicas": 2 } } uses the exact correct keys and valid JSON format.

3. Final Answer:

   { "settings": { "number_of_shards": 3, "number_of_replicas": 2 } } -> Option A

4. Quick Check:

   Correct keys = number_of_shards, number_of_replicas [OK]

Hint: Use full keys: number_of_shards and number_of_replicas [OK]

Common Mistakes:

• Using shorthand keys like shards or replicas
• Wrong JSON syntax
• Mixing shard and replica counts

3. Given an index with number_of_shards set to 4 and number_of_replicas set to 1, how many total shard copies exist in the cluster?

medium

A. 4

B. 6

C. 5

D. 8

Solution

1. Step 1: Calculate primary shards

   There are 4 primary shards as set.

2. Step 2: Calculate replica shards

   Each primary shard has 1 replica, so 4 replicas total.

3. Step 3: Add primary and replicas

   Total shards = 4 primary + 4 replicas = 8 shards.

4. Final Answer:

   8 -> Option D

5. Quick Check:

   Total shards = primary + replicas = 8 [OK]

Hint: Total shards = shards x (1 + replicas) [OK]

Common Mistakes:

• Counting only primary shards
• Adding shards and replicas incorrectly
• Confusing replicas with shards

4. You tried to create an index with this setting: { "settings": { "number_of_shards": 2, "number_of_replicas": -1 } }. What is the problem?

medium

A. Negative replicas value is invalid

B. Number of shards cannot be 2

C. Missing required field for replicas

D. JSON syntax error

Solution

1. Step 1: Check shards value

   Number of shards is 2, which is valid.

2. Step 2: Check replicas value

   Replicas cannot be negative; -1 is invalid.

3. Final Answer:

   Negative replicas value is invalid -> Option A

4. Quick Check:

   Replicas must be zero or positive [OK]

Hint: Replicas cannot be negative numbers [OK]

Common Mistakes:

• Assuming negative replicas mean no replicas
• Thinking shards must be > 3
• Ignoring JSON correctness

5. You want to create an index optimized for high availability with 5 shards. Which setting combination is best to ensure data safety and fast search?

hard

A. { "number_of_shards": 5, "number_of_replicas": 0 }

B. { "number_of_shards": 5, "number_of_replicas": 1 }

C. { "number_of_shards": 1, "number_of_replicas": 5 }

D. { "number_of_shards": 10, "number_of_replicas": 1 }

Solution

1. Step 1: Understand shards and replicas roles

   Shards split data for speed; replicas provide copies for safety.

2. Step 2: Evaluate options for availability and speed

   { "number_of_shards": 5, "number_of_replicas": 1 } keeps 5 shards for speed and 1 replica for safety, balancing both.

3. Step 3: Reject other options

   The option with zero replicas has no safety, the option with 1 shard is slow for search, the option with 10 shards has overhead.

4. Final Answer:

   { "number_of_shards": 5, "number_of_replicas": 1 } -> Option B

5. Quick Check:

   Balance shards and replicas for speed and safety [OK]

Hint: Balance shards for speed and replicas for safety [OK]

Common Mistakes:

• Setting zero replicas for safety
• Using too few shards for speed
• Over-sharding causing overhead