Shard allocation awareness in Elasticsearch

Introduction

Shard allocation awareness helps Elasticsearch keep data copies on different machines or racks. This makes sure your data stays safe even if one machine or rack fails.

You want to keep copies of your data on different physical racks to avoid losing data if a rack goes down.

You run Elasticsearch on multiple data centers and want to spread shards across them for safety.

You want to make sure shards are not all on the same machine to avoid single points of failure.

You want better control over where Elasticsearch places shards based on custom labels like zone or room.

Syntax

Elasticsearch

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "rack_id"
  }
}

PUT /my_index
{
  "settings": {
    "index.routing.allocation.awareness.include": {
      "rack_id": "rack1,rack2"
    }
  }
}

The cluster.routing.allocation.awareness.attributes setting tells Elasticsearch which node attribute to use for awareness (like rack_id).

The index.routing.allocation.awareness.include setting controls which attribute values shards should be allocated to.

Examples

This sets the awareness attribute to zone, so Elasticsearch will try to spread shards across different zones.

Elasticsearch

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone"
  }
}

This tells Elasticsearch to allocate shards only to nodes in zone1 or zone2.

Elasticsearch

PUT /my_index
{
  "settings": {
    "index.routing.allocation.awareness.include": {
      "zone": "zone1,zone2"
    }
  }
}

You can use multiple attributes like rack and zone for more detailed shard spreading.

Elasticsearch

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "rack,zone"
  }
}

Sample Program

This example sets the cluster to use rack_id for shard allocation awareness. Then it creates an index my_index that only allocates shards to nodes with rack1 or rack2. Finally, it shows where shards are placed.

Elasticsearch

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "rack_id"
  }
}

PUT /my_index
{
  "settings": {
    "index.routing.allocation.awareness.include": {
      "rack_id": "rack1,rack2"
    },
    "number_of_shards": 2,
    "number_of_replicas": 1
  }
}

GET /_cat/shards/my_index?v

Important Notes

Make sure your Elasticsearch nodes have the attribute (like rack_id) set in their elasticsearch.yml file under node.attr.rack_id.

Shard allocation awareness helps prevent data loss by spreading shards, but it does not guarantee perfect balance if nodes are uneven.

Use the GET /_cat/shards API to check where shards are allocated.

Summary

Shard allocation awareness spreads data copies across different physical locations.

Set cluster awareness attributes and index allocation rules to control shard placement.

This improves data safety and availability in case of machine or rack failures.