Hot-warm-cold architecture in Elasticsearch - Step-by-Step Execution

Concept Flow - Hot-warm-cold architecture

Data Ingested

↓

Hot Tier: Fast Storage & Indexing

↓

Warm Tier: Slower Storage, Read-Only

↓

Cold Tier: Cheapest Storage, Rare Access

↓

Data Archived or Deleted

Data flows from fast hot storage for new data, to slower warm storage for less active data, then to cold storage for rare access or archiving.

Execution Sample

Elasticsearch

PUT /_ilm/policy/hot-warm-cold-policy
{
  "policy": {
    "phases": {
      "hot": {"actions": {"rollover": {"max_age": "7d"}}},
      "warm": {"min_age": "7d", "actions": {"allocate": {"require": {"data": "warm"}}, "set_priority": {"priority": 50}}},
      "cold": {"min_age": "30d", "actions": {"allocate": {"require": {"data": "cold"}}, "set_priority": {"priority": 0}}},
      "delete": {"min_age": "90d", "actions": {"delete": {}}}
    }
  }
}

Defines an index lifecycle policy moving data through hot, warm, and cold phases based on age.

Execution Table

Step	Phase	Condition	Action	Result
1	Hot	Index age < 7 days	Indexing on fast nodes	Data stored on hot tier
2	Hot to Warm	Index age >= 7 days	Rollover and move to warm nodes	Data moved to warm tier, read-only
3	Warm to Cold	Index age >= 30 days	Move to cold nodes	Data moved to cold tier, cheaper storage
4	Cold	Index age >= 90 days	Optional delete or archive	Data archived or deleted
5	-	No further action	Stop lifecycle	Data lifecycle complete

💡 Data reaches end of lifecycle or is deleted after cold phase

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
Index Age	0 days	5 days	10 days	35 days	95 days
Data Location	Hot Tier	Hot Tier	Warm Tier	Cold Tier	Archived/Deleted
Index State	Writeable	Writeable	Read-only	Read-only	Deleted

Key Moments - 3 Insights

Why does data move from hot to warm tier after 7 days?

Because the policy rollover action triggers at 7 days (see execution_table row 2), moving data to warm tier to save cost while keeping it accessible.

Is data still writable in the warm tier?

No, data becomes read-only in warm tier as shown in variable_tracker under Index State after Step 2.

What happens to data after 90 days in cold tier?

It can be archived or deleted as per policy (execution_table row 4), completing the lifecycle.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does data become read-only?

A Step 2

B Step 1

C Step 3

D Step 4

💡 Hint

Check the 'Index State' in variable_tracker after Step 2 and the 'Action' column in execution_table row 2.

According to variable_tracker, what is the data location after 35 days?

A Hot Tier

B Warm Tier

C Cold Tier

D Archived

💡 Hint

Look at 'Data Location' column after Step 3 in variable_tracker.

If the rollover max_age in hot phase changed to 10 days, when would data move to warm tier?

A At 7 days

B At 10 days

C At 30 days

D Immediately

💡 Hint

Refer to execution_table row 2 where rollover triggers the move based on max_age.

Concept Snapshot

Hot-warm-cold architecture in Elasticsearch:
- Hot tier: fast, writeable storage for new data
- Warm tier: slower, read-only storage for older data
- Cold tier: cheapest storage for rarely accessed data
- Data moves through phases based on age via ILM policies
- Helps balance performance and cost

Full Transcript

The hot-warm-cold architecture in Elasticsearch organizes data storage by age and usage. New data is stored in the hot tier for fast indexing and searching. After a set time (e.g., 7 days), data moves to the warm tier where it becomes read-only and stored on slower, cheaper nodes. Later, after more time (e.g., 30 days), data moves to the cold tier for long-term storage at minimal cost. Finally, data can be archived or deleted after it is no longer needed. This flow is controlled by index lifecycle management policies that automate moving data through these phases based on age. Variables like index age, data location, and index state change step-by-step as data progresses through the lifecycle.