Index lifecycle management in Elasticsearch - Step-by-Step Execution

Concept Flow - Index lifecycle management

Create Index

↓

Attach Lifecycle Policy

↓

Index Enters Hot Phase

↓

Index Data Ingested

↓

Transition to Warm Phase

↓

Optimize and Shrink Index

↓

Transition to Cold Phase

↓

Read-Only, Less Frequent Access

↓

Transition to Delete Phase

↓

Delete Index

↓

End of Lifecycle

The index lifecycle starts with creation, then moves through phases (hot, warm, cold) with actions, ending in deletion.

Execution Sample

Elasticsearch

PUT /my-index-000001
{
  "settings": {
    "index.lifecycle.name": "my_policy"
  }
}

This creates an index and attaches a lifecycle policy named 'my_policy' to manage its phases.

Execution Table

Step	Action	Phase	Index State	Result
1	Create index with policy	N/A	Index created	Index 'my-index-000001' created with lifecycle policy 'my_policy'
2	Index enters hot phase	Hot	Writable, accepting data	Index is actively receiving data
3	Data ingestion	Hot	Growing index size	Documents are indexed
4	Transition to warm phase	Warm	Read-only, optimized	Index is shrunk and optimized
5	Transition to cold phase	Cold	Read-only, less frequent access	Index stored on cheaper hardware
6	Transition to delete phase	Delete	Marked for deletion	Index scheduled for deletion
7	Delete index	Delete	Deleted	Index 'my-index-000001' removed
8	End	N/A	No index	Lifecycle complete, index no longer exists

💡 Index deleted, lifecycle management ends

Variable Tracker

Variable	Start	After Step 2	After Step 4	After Step 6	Final
Index Phase	N/A	Hot	Warm	Delete	Deleted
Index State	Created	Writable	Read-only Optimized	Marked for Deletion	Deleted
Index Size	0	Growing	Stable	Stable	N/A

Key Moments - 3 Insights

Why does the index become read-only in the warm phase?

Because in the warm phase the index is optimized and shrunk to save resources, so it is set to read-only to prevent new data ingestion, as shown in execution_table row 4.

What triggers the transition from hot to warm phase?

The lifecycle policy defines conditions like index age or size; when met, the index transitions automatically, as seen between steps 3 and 4 in the execution_table.

Why is the index deleted at the end of lifecycle?

To free up storage and resources, the lifecycle policy deletes the index after it is no longer needed, as shown in execution_table steps 6 and 7.

Visual Quiz - 3 Questions

Look at the execution_table, what is the index state after step 4?

A Marked for deletion

B Writable and accepting data

C Read-only and optimized

D Deleted

💡 Hint

Check the 'Index State' column at step 4 in the execution_table

At which step does the index stop accepting new data?

A Step 4

B Step 2

C Step 3

D Step 6

💡 Hint

Look for when the index becomes read-only in the execution_table

If the lifecycle policy is removed, what would happen to the index lifecycle?

A Index would still transition phases automatically

B Index would remain in hot phase indefinitely

C Index would be deleted immediately

D Index would become read-only immediately

💡 Hint

Refer to the concept_flow where lifecycle policy controls phase transitions

Concept Snapshot

Index lifecycle management controls index phases: hot (write), warm (read-only, optimize), cold (less access), delete (remove).
Attach a policy to automate transitions based on age or size.
Phases save resources and manage data efficiently.
Deletion frees storage when data is no longer needed.

Full Transcript

Index lifecycle management in Elasticsearch automates how an index moves through different phases: hot, warm, cold, and delete. It starts by creating an index and attaching a lifecycle policy. The index begins in the hot phase where it accepts writes and data ingestion happens. When conditions like age or size are met, it transitions to the warm phase, becoming read-only and optimized to save resources. Later, it moves to the cold phase for less frequent access, stored on cheaper hardware. Finally, the index enters the delete phase where it is removed to free storage. This process helps manage data efficiently without manual intervention.

Practice

1. What is the main purpose of Index Lifecycle Management (ILM) in Elasticsearch?

easy

A. To automate index handling and improve cluster performance

B. To manually create indexes for each data type

C. To backup data to external storage

D. To encrypt index data for security

Solution

1. Step 1: Understand ILM's role

   ILM automates managing indexes through their lifecycle phases.

2. Step 2: Identify the benefit

   Automation helps save space and improves cluster health and performance.

3. Final Answer:

   To automate index handling and improve cluster performance -> Option A

4. Quick Check:

   ILM automates index management = A [OK]

Hint: ILM automates index tasks to keep Elasticsearch healthy [OK]

Common Mistakes:

• Thinking ILM is for manual index creation
• Confusing ILM with backup or encryption
• Assuming ILM only deletes indexes

2. Which of the following is the correct way to define a rollover action in an ILM policy JSON?

easy

A. "rollover": { "max_size": "50gb", "max_age": "30d" }

B. "rollover": "true"

C. "rollover": { "size": "50gb" }

D. "rollover": { "max_documents": 1000 }

Solution

1. Step 1: Check rollover action syntax

   Rollover requires conditions like max_size and max_age in a JSON object.

2. Step 2: Validate options

   "rollover": { "max_size": "50gb", "max_age": "30d" } correctly uses max_size and max_age keys with proper values.

3. Final Answer:

   "rollover": { "max_size": "50gb", "max_age": "30d" } -> Option A

4. Quick Check:

   Rollover needs max_size and max_age keys = A [OK]

Hint: Rollover needs max_size and max_age keys in JSON [OK]

Common Mistakes:

• Using string 'true' instead of object for rollover
• Missing max_age or max_size keys
• Using incorrect key names like 'size'

3. Given this ILM policy snippet:

{"phases": {"hot": {"actions": {"rollover": {"max_size": "10gb"}}}, "delete": {"min_age": "90d", "actions": {"delete": {}}}}}

What happens to an index after it reaches 10GB?

medium

A. It archives data to external storage

B. It rolls over to a new index and continues indexing

C. It stops accepting new data but stays

D. It is immediately deleted

Solution

1. Step 1: Analyze rollover action in hot phase

   The policy rolls over the index when it reaches 10GB.

2. Step 2: Understand delete phase timing

   Deletion happens only after 90 days, so not immediate.

3. Final Answer:

   It rolls over to a new index and continues indexing -> Option B

4. Quick Check:

   Rollover at 10GB, delete after 90d = D [OK]

Hint: Rollover triggers at size, delete waits min_age [OK]

Common Mistakes:

• Confusing rollover with immediate deletion
• Ignoring min_age delay for delete phase
• Assuming data stops indexing after rollover

4. You have this ILM policy JSON:

{"phases": {"hot": {"actions": {"rollover": {"max_size": "20gb"}}}, "warm": {"min_age": "10d", "actions": {"shrink": {"number_of_shards": 1}}}}}

After applying, the shrink action never runs. What is the likely cause?

medium

A. Rollover max_size is too small

B. Shrink action requires delete phase

C. Shrink action must be in hot phase

D. The index has not reached 10 days old yet

Solution

1. Step 1: Check warm phase min_age

   Shrink runs only after index is at least 10 days old.

2. Step 2: Confirm index age

   If index age is less than 10 days, shrink won't trigger yet.

3. Final Answer:

   The index has not reached 10 days old yet -> Option D

4. Quick Check:

   min_age controls action timing = C [OK]

Hint: Check min_age before expecting warm phase actions [OK]

Common Mistakes:

• Assuming shrink needs delete phase
• Changing rollover size unrelated to shrink
• Placing shrink in wrong phase

5. You want to create an ILM policy that:
- Rolls over an index at 30GB or 15 days
- Moves the index to warm phase after rollover
- Deletes the index 60 days after rollover

Which policy configuration correctly implements this?

hard

A. {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb"}}}, "warm": {"min_age": "30d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

B. {"phases": {"hot": {"actions": {"rollover": {"max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {"shrink": {}}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

C. {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb", "max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

D. {"phases": {"hot": {"actions": {"rollover": {"max_size": "15gb", "max_age": "30d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}}

Solution

1. Step 1: Verify rollover conditions

   {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb", "max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}} uses max_size 30gb and max_age 15d, matching requirements.

2. Step 2: Check warm and delete phases

   Warm phase starts at 15d (after rollover), delete at 60d, as required.

3. Step 3: Confirm other options

   Other options have wrong sizes, ages, or missing actions.

4. Final Answer:

   {"phases": {"hot": {"actions": {"rollover": {"max_size": "30gb", "max_age": "15d"}}}, "warm": {"min_age": "15d", "actions": {}}, "delete": {"min_age": "60d", "actions": {"delete": {}}}}} -> Option C

5. Quick Check:

   Rollover at 30gb/15d, warm at 15d, delete at 60d = B [OK]

Hint: Match rollover size and age, then warm and delete min_age [OK]

Common Mistakes:

• Swapping max_size and max_age values
• Setting warm min_age too late
• Missing delete action or wrong min_age