Index lifecycle management in Elasticsearch - Mini Project: Build & Apply

Index Lifecycle Management in Elasticsearch

📖 Scenario: You are managing logs for a web application. To save space and keep the system fast, you want to automatically delete old logs after some time.

🎯 Goal: Create an index lifecycle policy in Elasticsearch that automatically deletes indices older than 30 days.

📋 What You'll Learn

Create an index lifecycle policy named log_policy with a delete phase

Set the delete phase to trigger after 30 days

Apply the policy to an index template named log_template

The index template should match indices named logs-*

💡 Why This Matters

🌍 Real World

Managing log data efficiently by automatically deleting old data to save storage and improve search performance.

💼 Career

Index lifecycle management is a key skill for Elasticsearch administrators and DevOps engineers to maintain healthy and cost-effective search clusters.

1

Create the index lifecycle policy

Create an index lifecycle policy called log_policy with a delete phase that deletes indices after 30d.

Elasticsearch

# Your code here to create the index lifecycle policy named log_policy with a delete phase after 30 days

Hint

Use the delete phase with min_age set to 30d and an empty delete action.

2

Create an index template with the lifecycle policy

Create an index template called log_template that matches indices named logs-* and applies the lifecycle policy log_policy.

Elasticsearch

{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

# Your code here to create the index template named log_template with pattern logs-* and lifecycle policy log_policy

Hint

Set index_patterns to ["logs-*"] and in settings add index.lifecycle.name with value log_policy.

3

Apply the lifecycle policy to an existing index

Add the lifecycle policy log_policy to an existing index named logs-2024-06 by updating its settings.

Elasticsearch

{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

{
  "index_patterns": ["logs-*"] ,
  "template": {
    "settings": {
      "index.lifecycle.name": "log_policy"
    }
  },
  "name": "log_template"
}

# Your code here to update the index logs-2024-06 with lifecycle policy log_policy

Hint

Update the index settings to include index.lifecycle.name with value log_policy.

4

Verify the lifecycle policy is active

Write a query to get the lifecycle policy status for the index logs-2024-06 to verify the policy log_policy is applied.

Elasticsearch

{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

{
  "index_patterns": ["logs-*"] ,
  "template": {
    "settings": {
      "index.lifecycle.name": "log_policy"
    }
  },
  "name": "log_template"
}

{
  "settings": {
    "index.lifecycle.name": "log_policy"
  }
}

# Your code here to get the lifecycle status of index logs-2024-06

Hint

Use the _ilm/explain API on the index logs-2024-06 to check lifecycle status.

Practice

1. What is the main purpose of Index Lifecycle Management (ILM) in Elasticsearch?

easy

A. To automate index handling and improve cluster performance

B. To manually create indexes for each data type

C. To backup data to external storage

D. To encrypt index data for security

Solution

1. Step 1: Understand ILM's role

   ILM automates managing indexes through their lifecycle phases.

2. Step 2: Identify the benefit

   Automation helps save space and improves cluster health and performance.

3. Final Answer:

   To automate index handling and improve cluster performance -> Option A

4. Quick Check:

   ILM automates index management = A [OK]

Hint: ILM automates index tasks to keep Elasticsearch healthy [OK]

Common Mistakes:

• Thinking ILM is for manual index creation
• Confusing ILM with backup or encryption
• Assuming ILM only deletes indexes

2. Which of the following is the correct way to define a rollover action in an ILM policy JSON?

easy

A. "rollover": { "max_size": "50gb", "max_age": "30d" }

B. "rollover": "true"

C. "rollover": { "size": "50gb" }

D. "rollover": { "max_documents": 1000 }

Solution

1. Step 1: Check rollover action syntax

   Rollover requires conditions like max_size and max_age in a JSON object.

2. Step 2: Validate options

   "rollover": { "max_size": "50gb", "max_age": "30d" } correctly uses max_size and max_age keys with proper values.

3. Final Answer:

   "rollover": { "max_size": "50gb", "max_age": "30d" } -> Option A

4. Quick Check:

   Rollover needs max_size and max_age keys = A [OK]

Hint: Rollover needs max_size and max_age keys in JSON [OK]

Common Mistakes:

• Using string 'true' instead of object for rollover
• Missing max_age or max_size keys
• Using incorrect key names like 'size'

3. Given this ILM policy snippet:

{"phases": {"hot": {"actions": {"rollover": {"max_size": "10gb"}}}, "delete": {"min_age": "90d", "actions": {"delete": {}}}}}

What happens to an index after it reaches 10GB?

medium

A. It archives data to external storage

B. It rolls over to a new index and continues indexing

C. It stops accepting new data but stays

D. It is immediately deleted

Solution

1. Step 1: Analyze rollover action in hot phase

   The policy rolls over the index when it reaches 10GB.

2. Step 2: Understand delete phase timing

   Deletion happens only after 90 days, so not immediate.

3. Final Answer:

   It rolls over to a new index and continues indexing -> Option B

4. Quick Check:

   Rollover at 10GB, delete after 90d = D [OK]

Hint: Rollover triggers at size, delete waits min_age [OK]

Common Mistakes:

• Confusing rollover with immediate deletion
• Ignoring min_age delay for delete phase
• Assuming data stops indexing after rollover

4. You have this ILM policy JSON:

{"phases": {"hot": {"actions": {"rollover": {"max_size": "20gb"}}}, "warm": {"min_age": "10d", "actions": {"shrink": {"number_of_shards": 1}}}}}

After applying, the shrink action never runs. What is the likely cause?

medium

A. Rollover max_size is too small

B. Shrink action requires delete phase

C. Shrink action must be in hot phase

D. The index has not reached 10 days old yet

Solution

1. Step 1: Check warm phase min_age

   Shrink runs only after index is at least 10 days old.

2. Step 2: Confirm index age

   If index age is less than 10 days, shrink won't trigger yet.

3. Final Answer:

   The index has not reached 10 days old yet -> Option D

4. Quick Check:

   min_age controls action timing = C [OK]

Hint: Check min_age before expecting warm phase actions [OK]

Common Mistakes:

• Assuming shrink needs delete phase
• Changing rollover size unrelated to shrink
• Placing shrink in wrong phase

5. You want to create an ILM policy that:
- Rolls over an index at 30GB or 15 days
- Moves the index to warm phase after rollover
- Deletes the index 60 days after rollover

Which policy configuration correctly implements this?

hard

A. {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb"}}}, "warm": {"min_age": "30d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

B. {"phases": {"hot": {"actions": {"rollover": {"max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {"shrink": {}}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

C. {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb", "max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

D. {"phases": {"hot": {"actions": {"rollover": {"max_size": "15gb", "max_age": "30d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

Solution

1. Step 1: Verify rollover conditions

   {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb", "max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}} uses max_size 30gb and max_age 15d, matching requirements.

2. Step 2: Check warm and delete phases

   Warm phase starts at 15d (after rollover), delete at 60d, as required.

3. Step 3: Confirm other options

   Other options have wrong sizes, ages, or missing actions.

4. Final Answer:

   {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb", "max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}} -> Option C

5. Quick Check:

   Rollover at 30gb/15d, warm at 15d, delete at 60d = B [OK]

Hint: Match rollover size and age, then warm and delete min_age [OK]

Common Mistakes:

• Swapping max_size and max_age values
• Setting warm min_age too late
• Missing delete action or wrong min_age