Index lifecycle management in Elasticsearch

Introduction

Index lifecycle management helps you automatically manage your data as it grows old or becomes less important. It saves space and keeps your search fast.

You want to delete old logs after a certain time to save storage.

You want to move data to cheaper storage when it is not used often.

You want to keep recent data on fast disks for quick searching.

You want to automatically create new indexes for new data.

You want to reduce manual work managing many indexes.

Syntax

Elasticsearch

PUT _ilm/policy/<policy_name>
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

The policy defines phases like hot, warm, cold, and delete.

Each phase has actions like rollover, shrink, or delete to manage indexes automatically.

Examples

This policy rolls over the index after 7 days and deletes it after 30 days.

Elasticsearch

PUT _ilm/policy/log_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This policy moves data to warm nodes after 30 days and freezes it after 60 days to save resources.

Elasticsearch

PUT _ilm/policy/archive_policy
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "freeze": {}
        }
      }
    }
  }
}

Sample Program

This example creates a policy that rolls over indexes daily or after 10GB and deletes them after 7 days. Then it retrieves the policy to confirm.

Elasticsearch

PUT _ilm/policy/sample_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "10gb",
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

GET _ilm/policy/sample_policy

Important Notes

Make sure your index template uses the lifecycle policy by setting index.lifecycle.name.

Rollover requires a write alias to work properly.

Phases run automatically based on index age and size.

Summary

Index lifecycle management automates index handling to save space and improve performance.

Policies define phases with actions like rollover and delete.

Use ILM to reduce manual work and keep your Elasticsearch cluster healthy.

Practice

1. What is the main purpose of Index Lifecycle Management (ILM) in Elasticsearch?

easy

A. To automate index handling and improve cluster performance

B. To manually create indexes for each data type

C. To backup data to external storage

D. To encrypt index data for security

Solution

1. Step 1: Understand ILM's role

   ILM automates managing indexes through their lifecycle phases.

2. Step 2: Identify the benefit

   Automation helps save space and improves cluster health and performance.

3. Final Answer:

   To automate index handling and improve cluster performance -> Option A

4. Quick Check:

   ILM automates index management = A [OK]

Hint: ILM automates index tasks to keep Elasticsearch healthy [OK]

Common Mistakes:

• Thinking ILM is for manual index creation
• Confusing ILM with backup or encryption
• Assuming ILM only deletes indexes

2. Which of the following is the correct way to define a rollover action in an ILM policy JSON?

easy

A. "rollover": { "max_size": "50gb", "max_age": "30d" }

B. "rollover": "true"

C. "rollover": { "size": "50gb" }

D. "rollover": { "max_documents": 1000 }

Solution

1. Step 1: Check rollover action syntax

   Rollover requires conditions like max_size and max_age in a JSON object.

2. Step 2: Validate options

   "rollover": { "max_size": "50gb", "max_age": "30d" } correctly uses max_size and max_age keys with proper values.

3. Final Answer:

   "rollover": { "max_size": "50gb", "max_age": "30d" } -> Option A

4. Quick Check:

   Rollover needs max_size and max_age keys = A [OK]

Hint: Rollover needs max_size and max_age keys in JSON [OK]

Common Mistakes:

• Using string 'true' instead of object for rollover
• Missing max_age or max_size keys
• Using incorrect key names like 'size'

3. Given this ILM policy snippet:

{"phases": {"hot": {"actions": {"rollover": {"max_size": "10gb"}}}, "delete": {"min_age": "90d", "actions": {"delete": {}}}}}

What happens to an index after it reaches 10GB?

medium

A. It archives data to external storage

B. It rolls over to a new index and continues indexing

C. It stops accepting new data but stays

D. It is immediately deleted

Solution

1. Step 1: Analyze rollover action in hot phase

   The policy rolls over the index when it reaches 10GB.

2. Step 2: Understand delete phase timing

   Deletion happens only after 90 days, so not immediate.

3. Final Answer:

   It rolls over to a new index and continues indexing -> Option B

4. Quick Check:

   Rollover at 10GB, delete after 90d = D [OK]

Hint: Rollover triggers at size, delete waits min_age [OK]

Common Mistakes:

• Confusing rollover with immediate deletion
• Ignoring min_age delay for delete phase
• Assuming data stops indexing after rollover

4. You have this ILM policy JSON:

{"phases": {"hot": {"actions": {"rollover": {"max_size": "20gb"}}}, "warm": {"min_age": "10d", "actions": {"shrink": {"number_of_shards": 1}}}}}

After applying, the shrink action never runs. What is the likely cause?

medium

A. Rollover max_size is too small

B. Shrink action requires delete phase

C. Shrink action must be in hot phase

D. The index has not reached 10 days old yet

Solution

1. Step 1: Check warm phase min_age

   Shrink runs only after index is at least 10 days old.

2. Step 2: Confirm index age

   If index age is less than 10 days, shrink won't trigger yet.

3. Final Answer:

   The index has not reached 10 days old yet -> Option D

4. Quick Check:

   min_age controls action timing = C [OK]

Hint: Check min_age before expecting warm phase actions [OK]

Common Mistakes:

• Assuming shrink needs delete phase
• Changing rollover size unrelated to shrink
• Placing shrink in wrong phase

5. You want to create an ILM policy that:
- Rolls over an index at 30GB or 15 days
- Moves the index to warm phase after rollover
- Deletes the index 60 days after rollover

Which policy configuration correctly implements this?

hard

A. {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb"}}}, "warm": {"min_age": "30d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

B. {"phases": {"hot": {"actions": {"rollover": {"max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {"shrink": {}}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

C. {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb", "max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

D. {"phases": {"hot": {"actions": {"rollover": {"max_size": "15gb", "max_age": "30d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

Solution

1. Step 1: Verify rollover conditions

   {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb", "max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}} uses max_size 30gb and max_age 15d, matching requirements.

2. Step 2: Check warm and delete phases

   Warm phase starts at 15d (after rollover), delete at 60d, as required.

3. Step 3: Confirm other options

   Other options have wrong sizes, ages, or missing actions.

4. Final Answer:

   {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb", "max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}} -> Option C

5. Quick Check:

   Rollover at 30gb/15d, warm at 15d, delete at 60d = B [OK]

Hint: Match rollover size and age, then warm and delete min_age [OK]

Common Mistakes:

• Swapping max_size and max_age values
• Setting warm min_age too late
• Missing delete action or wrong min_age