Index lifecycle management in Elasticsearch - Mini Project: Build & Apply

Index Lifecycle Management in Elasticsearch

📖 Scenario: You are managing logs for a web application. To save space and keep the system fast, you want to automatically delete old logs after some time.

🎯 Goal: Create an index lifecycle policy in Elasticsearch that automatically deletes indices older than 30 days.

📋 What You'll Learn

Create an index lifecycle policy named log_policy with a delete phase

Set the delete phase to trigger after 30 days

Apply the policy to an index template named log_template

The index template should match indices named logs-*

💡 Why This Matters

🌍 Real World

Managing log data efficiently by automatically deleting old data to save storage and improve search performance.

💼 Career

Index lifecycle management is a key skill for Elasticsearch administrators and DevOps engineers to maintain healthy and cost-effective search clusters.

1

Create the index lifecycle policy

Create an index lifecycle policy called log_policy with a delete phase that deletes indices after 30d.

Elasticsearch

# Your code here to create the index lifecycle policy named log_policy with a delete phase after 30 days

Need a hint?

Use the delete phase with min_age set to 30d and an empty delete action.

2

Create an index template with the lifecycle policy

Create an index template called log_template that matches indices named logs-* and applies the lifecycle policy log_policy.

Elasticsearch

{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

# Your code here to create the index template named log_template with pattern logs-* and lifecycle policy log_policy

Need a hint?

Set index_patterns to ["logs-*"] and in settings add index.lifecycle.name with value log_policy.

3

Apply the lifecycle policy to an existing index

Add the lifecycle policy log_policy to an existing index named logs-2024-06 by updating its settings.

Elasticsearch

{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

{
  "index_patterns": ["logs-*"] ,
  "template": {
    "settings": {
      "index.lifecycle.name": "log_policy"
    }
  },
  "name": "log_template"
}

# Your code here to update the index logs-2024-06 with lifecycle policy log_policy

Need a hint?

Update the index settings to include index.lifecycle.name with value log_policy.

4

Verify the lifecycle policy is active

Write a query to get the lifecycle policy status for the index logs-2024-06 to verify the policy log_policy is applied.

Elasticsearch

{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

{
  "index_patterns": ["logs-*"] ,
  "template": {
    "settings": {
      "index.lifecycle.name": "log_policy"
    }
  },
  "name": "log_template"
}

{
  "settings": {
    "index.lifecycle.name": "log_policy"
  }
}

# Your code here to get the lifecycle status of index logs-2024-06

Need a hint?

Use the _ilm/explain API on the index logs-2024-06 to check lifecycle status.