Hot-warm-cold architecture in Elasticsearch - Mini Project: Build & Apply

Implement Hot-Warm-Cold Architecture in Elasticsearch

📖 Scenario: You are managing a large Elasticsearch cluster that stores logs from a web application. To optimize storage costs and performance, you want to organize your data using the hot-warm-cold architecture. This means recent data is stored on fast nodes (hot), older data on less expensive nodes (warm), and the oldest data on the cheapest nodes (cold).

🎯 Goal: Build an Elasticsearch index lifecycle management (ILM) policy and apply it to an index template that moves data through hot, warm, and cold phases automatically.

📋 What You'll Learn

Create an ILM policy named hot-warm-cold-policy with hot, warm, and cold phases

In the hot phase, rollover the index when it reaches 1GB or 1 day old

In the warm phase, allocate the index to warm nodes and reduce replicas to 1

In the cold phase, allocate the index to cold nodes and set the index to read-only

Create an index template named logs-template that applies the ILM policy to indices starting with logs-

💡 Why This Matters

🌍 Real World

Hot-warm-cold architecture helps manage large volumes of time-series data like logs by optimizing performance and cost.

💼 Career

Understanding ILM policies and data tiering is essential for Elasticsearch administrators and DevOps engineers managing scalable search and analytics clusters.

1

Create the ILM policy with hot, warm, and cold phases

Create an ILM policy called hot-warm-cold-policy with three phases: hot, warm, and cold. In the hot phase, set rollover conditions to max_size of 1gb and max_age of 1d. Leave warm and cold phases empty for now.

Elasticsearch

# Your code here to create the ILM policy with hot phase rollover conditions

Hint

Use the Elasticsearch ILM API to define the policy JSON with phases and rollover actions.

2

Add warm phase actions to the ILM policy

Add to the warm phase in the hot-warm-cold-policy ILM policy these actions: allocate the index to nodes with attribute data=warm and set the number of replicas to 1.

Elasticsearch

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "actions": {
          // Your code here to allocate to warm nodes and set replicas
        }
      },
      "cold": {}
    }
  }
}

Hint

Use allocate action with require to specify node attribute, and replica_count to set replicas.

3

Add cold phase actions to the ILM policy

Add to the cold phase in the hot-warm-cold-policy ILM policy these actions: allocate the index to nodes with attribute data=cold and set the index to read-only by adding the readonly action.

Elasticsearch

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "actions": {
          "allocate": {
            "require": {
              "data": "warm"
            }
          },
          "set_priority": {
            "priority": 50
          },
          "replica_count": {
            "number_of_replicas": 1
          }
        }
      },
      "cold": {
        "actions": {
          // Your code here to allocate to cold nodes and set readonly
        }
      }
    }
  }
}

Hint

Use allocate with require for cold nodes and add readonly action.

4

Create an index template applying the ILM policy

Create an index template named logs-template that matches index patterns starting with logs-. In the template settings, apply the ILM policy hot-warm-cold-policy by setting index.lifecycle.name to hot-warm-cold-policy and index.lifecycle.rollover_alias to logs-alias.

Elasticsearch

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "actions": {
          "allocate": {
            "require": {
              "data": "warm"
            }
          },
          "set_priority": {
            "priority": 50
          },
          "replica_count": {
            "number_of_replicas": 1
          }
        }
      },
      "cold": {
        "actions": {
          "allocate": {
            "require": {
              "data": "cold"
            }
          },
          "readonly": {}
        }
      }
    }
  }
}

// Your code here to create the index template applying the ILM policy

Hint

Define an index template JSON with index_patterns, settings for ILM, and an alias matching the rollover alias.

Practice

1. What is the main purpose of the hot-warm-cold architecture in Elasticsearch?

easy

A. To encrypt data at rest and in transit

B. To store recent data on fast nodes and older data on slower, cheaper nodes

C. To backup data to external storage automatically

D. To replicate data across multiple clusters for high availability

Solution

1. Step 1: Understand the architecture purpose

   The hot-warm-cold architecture is designed to optimize storage costs and performance by placing recent data on fast nodes and older data on slower, cheaper nodes.

2. Step 2: Match the purpose to options

   To store recent data on fast nodes and older data on slower, cheaper nodes correctly describes this purpose, while other options describe different Elasticsearch features.

3. Final Answer:

   To store recent data on fast nodes and older data on slower, cheaper nodes -> Option B

4. Quick Check:

   Hot-warm-cold architecture = store data by age and speed [OK]

Hint: Remember: hot = fast recent, cold = slow old data [OK]

Common Mistakes:

• Confusing hot-warm-cold with backup or replication
• Thinking it encrypts data automatically
• Assuming it manages cluster replication

2. Which Elasticsearch feature is used to automate moving data between hot, warm, and cold phases?

easy

A. Snapshot and Restore

B. Document Level Security

C. Index Lifecycle Management (ILM)

D. Cross-cluster Search

Solution

1. Step 1: Identify automation for data phase movement

   Index Lifecycle Management (ILM) automates moving indices through hot, warm, and cold phases based on policies.

2. Step 2: Compare other features

   Snapshot and Restore handles backups, Cross-cluster Search queries multiple clusters, and Document Level Security controls access, so they don't automate data movement.

3. Final Answer:

   Index Lifecycle Management (ILM) -> Option C

4. Quick Check:

   ILM automates data phase transitions [OK]

Hint: ILM = automates index phase changes [OK]

Common Mistakes:

• Choosing Snapshot instead of ILM
• Confusing security features with lifecycle management
• Thinking cross-cluster search manages data phases

3. Given this ILM policy snippet, what phase will the index move to after 30 days?

{
  "phases": {
    "hot": {"min_age": "0d"},
    "warm": {"min_age": "7d"},
    "cold": {"min_age": "30d"}
  }
}

medium

A. Cold phase

B. Warm phase

C. Hot phase

D. Delete phase

Solution

1. Step 1: Analyze min_age values for phases

   The policy defines hot from 0 days, warm from 7 days, and cold from 30 days.

2. Step 2: Determine phase after 30 days

   After 30 days, the index reaches the cold phase because its min_age is 30 days, which is the threshold for cold.

3. Final Answer:

   Cold phase -> Option A

4. Quick Check:

   30 days = cold phase start [OK]

Hint: Check min_age values to find current phase [OK]

Common Mistakes:

• Choosing warm phase after 30 days
• Confusing delete phase with cold phase
• Ignoring min_age thresholds

4. You wrote this ILM policy but your index never moves to the warm phase:

{
  "phases": {
    "hot": {"min_age": "0d"},
    "warm": {"min_age": "10d"}
  }
}

What is the likely problem?

medium

A. The index size is too small to trigger rollover

B. The warm phase min_age is too low

C. The warm phase is missing an allocation action

D. The policy lacks a cold phase

Solution

1. Step 1: Understand ILM phase transition requirements

   For an index to move from hot to warm, rollover conditions like size or age must be met.

2. Step 2: Identify missing trigger

   If the index size is too small, rollover won't happen, so the index stays in hot phase and never moves to warm.

3. Final Answer:

   The index size is too small to trigger rollover -> Option A

4. Quick Check:

   Small index size blocks rollover and phase move [OK]

Hint: Check rollover conditions to enable phase change [OK]

Common Mistakes:

• Assuming missing allocation causes no move
• Thinking warm phase min_age is too low
• Believing cold phase is required to move to warm

5. You want to optimize storage costs by moving indices older than 60 days to cold nodes and delete indices older than 90 days. Which ILM policy snippet correctly implements this?

hard

A. { "phases": { "hot": {"min_age": "0d"}, "warm": {"min_age": "30d"}, "cold": {"min_age": "90d"}, "delete": {"min_age": "90d"} } }

B. { "phases": { "hot": {"min_age": "0d"}, "warm": {"min_age": "30d"}, "delete": {"min_age": "60d"} } }

C. { "phases": { "hot": {"min_age": "0d"}, "warm": {"min_age": "60d"}, "cold": {"min_age": "90d"}, "delete": {"min_age": "120d"} } }

D. { "phases": { "hot": {"min_age": "0d"}, "cold": {"min_age": "60d"}, "delete": {"min_age": "90d"} } }

Solution

1. Step 1: Identify required phase ages

   Indices older than 60 days should move to cold, and older than 90 days should be deleted.

2. Step 2: Match policy phases to requirements

   { "phases": { "hot": {"min_age": "0d"}, "cold": {"min_age": "60d"}, "delete": {"min_age": "90d"} } } has hot at 0d, cold at 60d, and delete at 90d, matching the requirements exactly.

3. Final Answer:

   { "phases": { "hot": {"min_age": "0d"}, "cold": {"min_age": "60d"}, "delete": {"min_age": "90d"} } } -> Option D

4. Quick Check:

   60d cold and 90d delete phases match [OK]

Hint: Match min_age exactly to your data lifecycle needs [OK]

Common Mistakes:

• Adding unnecessary warm phase with wrong min_age
• Setting delete phase too early
• Skipping cold phase before delete