Hot-warm-cold architecture in Elasticsearch

Introduction

This architecture helps manage data efficiently by storing recent data on fast machines and older data on slower, cheaper machines.

You want to keep recent logs quickly accessible for analysis.

You need to save storage costs by moving old data to cheaper hardware.

You want to improve search speed for recent data while still keeping older data available.

You have large amounts of time-series data like logs or metrics.

You want to automate data lifecycle management in Elasticsearch.

Syntax

Elasticsearch

PUT /_ilm/policy/hot-warm-cold-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "require": { "data": "warm" }
          },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "allocate": {
            "require": { "data": "cold" }
          },
          "set_priority": { "priority": 0 },
          "freeze": {}
        }
      }
    }
  }
}

The policy defines phases: hot, warm, and cold with actions for each.

Use allocate to move data to nodes with matching attributes.

Examples

This example shows a simple hot-warm policy with rollover after 7 days and moving data to warm nodes after 7 days.

Elasticsearch

PUT /_ilm/policy/simple-hot-warm
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "7d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": { "require": { "data": "warm" } },
          "set_priority": { "priority": 50 }
        }
      }
    }
  }
}

This template applies the hot-warm-cold policy to indices matching 'logs-*' and sets rollover alias.

Elasticsearch

PUT /_template/logs_template
{
  "index_patterns": ["logs-*"],
  "settings": {
    "index.lifecycle.name": "hot-warm-cold-policy",
    "index.lifecycle.rollover_alias": "logs-alias"
  }
}

Sample Program

This program creates a hot-warm-cold lifecycle policy, sets an index template to use it, and creates the first index with a rollover alias.

Elasticsearch

PUT /_ilm/policy/hot-warm-cold-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "require": { "data": "warm" }
          },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "allocate": {
            "require": { "data": "cold" }
          },
          "set_priority": { "priority": 0 },
          "freeze": {}
        }
      }
    }
  }
}

PUT /_template/logs_template
{
  "index_patterns": ["logs-*"],
  "settings": {
    "index.lifecycle.name": "hot-warm-cold-policy",
    "index.lifecycle.rollover_alias": "logs-alias"
  }
}

PUT /logs-000001
{
  "aliases": {
    "logs-alias": {
      "is_write_index": true
    }
  }
}

Important Notes

Make sure your Elasticsearch nodes have attributes like node.attr.data: warm or cold to allocate data properly.

Rollover helps create new indices automatically when size or age limits are reached.

Freezing cold indices reduces resource use but makes searches slower.

Summary

Hot-warm-cold architecture stores recent data on fast nodes and older data on slower, cheaper nodes.

Use index lifecycle management (ILM) policies to automate data movement between phases.

Rollover and allocation actions help manage data size and location efficiently.