Shard allocation awareness in Elasticsearch - Mini Project: Build & Apply

Shard allocation awareness

📖 Scenario: You are managing an Elasticsearch cluster that stores data across multiple nodes. To improve fault tolerance, you want to control how shards are allocated based on node attributes like rack or zone. This helps ensure that copies of data are spread across different physical locations.

🎯 Goal: Learn how to configure shard allocation awareness in Elasticsearch by setting cluster and index settings. You will create a cluster setting for awareness attributes, then configure an index to use shard allocation awareness to distribute shards across nodes with different attributes.

📋 What You'll Learn

Create a cluster setting to enable shard allocation awareness on the attribute rack

Create an index called my-index with 3 primary shards and 1 replica

Configure the index to use shard allocation awareness on the rack attribute

Print the final index settings to verify shard allocation awareness is set

💡 Why This Matters

🌍 Real World

Shard allocation awareness helps keep Elasticsearch data safe by spreading copies across different physical locations or racks, reducing risk of data loss if one location fails.

💼 Career

Understanding shard allocation awareness is important for Elasticsearch administrators and DevOps engineers to design resilient and high-availability search clusters.

1

Set cluster awareness attribute

Use the Elasticsearch cluster update settings API to set cluster.routing.allocation.awareness.attributes to rack.

Elasticsearch

# Use the cluster update settings API to set awareness attribute
# Your code here

Need a hint?

Use the PUT /_cluster/settings API with persistent settings to set cluster.routing.allocation.awareness.attributes to rack.

2

Create index with shards and replicas

Create an index called my-index with 3 primary shards and 1 replica using the index creation API.

Elasticsearch

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "rack"
  }
}

# Create index my-index with 3 primary shards and 1 replica
# Your code here

Need a hint?

Use PUT /my-index with settings to specify number_of_shards and number_of_replicas.

3

Configure index shard allocation awareness

Update the index my-index settings to add index.routing.allocation.awareness.attributes set to rack to enable shard allocation awareness on the index level.

Elasticsearch

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "rack"
  }
}

PUT /my-index
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

# Update my-index settings to add shard allocation awareness on rack
# Your code here

Need a hint?

Use PUT /my-index/_settings to add index.routing.allocation.awareness.attributes with value rack.

4

Verify index settings output

Use the index get settings API to print the settings of my-index and verify that index.routing.allocation.awareness.attributes is set to rack.

Elasticsearch

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "rack"
  }
}

PUT /my-index
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

PUT /my-index/_settings
{
  "index.routing.allocation.awareness.attributes": "rack"
}

# Get and print my-index settings
# Your code here

Need a hint?

Use GET /my-index/_settings to retrieve and verify the index settings.