Hot-warm-cold architecture in Elasticsearch - Mini Project: Build & Apply

Implement Hot-Warm-Cold Architecture in Elasticsearch

📖 Scenario: You are managing a large Elasticsearch cluster that stores logs from a web application. To optimize storage costs and performance, you want to organize your data using the hot-warm-cold architecture. This means recent data is stored on fast nodes (hot), older data on less expensive nodes (warm), and the oldest data on the cheapest nodes (cold).

🎯 Goal: Build an Elasticsearch index lifecycle management (ILM) policy and apply it to an index template that moves data through hot, warm, and cold phases automatically.

📋 What You'll Learn

Create an ILM policy named hot-warm-cold-policy with hot, warm, and cold phases

In the hot phase, rollover the index when it reaches 1GB or 1 day old

In the warm phase, allocate the index to warm nodes and reduce replicas to 1

In the cold phase, allocate the index to cold nodes and set the index to read-only

Create an index template named logs-template that applies the ILM policy to indices starting with logs-

💡 Why This Matters

🌍 Real World

Hot-warm-cold architecture helps manage large volumes of time-series data like logs by optimizing performance and cost.

💼 Career

Understanding ILM policies and data tiering is essential for Elasticsearch administrators and DevOps engineers managing scalable search and analytics clusters.

1

Create the ILM policy with hot, warm, and cold phases

Create an ILM policy called hot-warm-cold-policy with three phases: hot, warm, and cold. In the hot phase, set rollover conditions to max_size of 1gb and max_age of 1d. Leave warm and cold phases empty for now.

Elasticsearch

# Your code here to create the ILM policy with hot phase rollover conditions

Need a hint?

Use the Elasticsearch ILM API to define the policy JSON with phases and rollover actions.

2

Add warm phase actions to the ILM policy

Add to the warm phase in the hot-warm-cold-policy ILM policy these actions: allocate the index to nodes with attribute data=warm and set the number of replicas to 1.

Elasticsearch

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "actions": {
          // Your code here to allocate to warm nodes and set replicas
        }
      },
      "cold": {}
    }
  }
}

Need a hint?

Use allocate action with require to specify node attribute, and replica_count to set replicas.

3

Add cold phase actions to the ILM policy

Add to the cold phase in the hot-warm-cold-policy ILM policy these actions: allocate the index to nodes with attribute data=cold and set the index to read-only by adding the readonly action.

Elasticsearch

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "actions": {
          "allocate": {
            "require": {
              "data": "warm"
            }
          },
          "set_priority": {
            "priority": 50
          },
          "replica_count": {
            "number_of_replicas": 1
          }
        }
      },
      "cold": {
        "actions": {
          // Your code here to allocate to cold nodes and set readonly
        }
      }
    }
  }
}

Need a hint?

Use allocate with require for cold nodes and add readonly action.

4

Create an index template applying the ILM policy

Create an index template named logs-template that matches index patterns starting with logs-. In the template settings, apply the ILM policy hot-warm-cold-policy by setting index.lifecycle.name to hot-warm-cold-policy and index.lifecycle.rollover_alias to logs-alias.

Elasticsearch

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "actions": {
          "allocate": {
            "require": {
              "data": "warm"
            }
          },
          "set_priority": {
            "priority": 50
          },
          "replica_count": {
            "number_of_replicas": 1
          }
        }
      },
      "cold": {
        "actions": {
          "allocate": {
            "require": {
              "data": "cold"
            }
          },
          "readonly": {}
        }
      }
    }
  }
}

// Your code here to create the index template applying the ILM policy

Need a hint?

Define an index template JSON with index_patterns, settings for ILM, and an alias matching the rollover alias.