Alerting and notifications in Elasticsearch - Mini Project: Build & Apply

Alerting and notifications

📖 Scenario: You are managing a server that stores logs in Elasticsearch. You want to get alerts when the number of error logs goes above a certain limit. This helps you fix problems quickly.

🎯 Goal: Create a watch in Elasticsearch that checks the number of error logs in the last 5 minutes. If the count is more than 10, send an email notification.

📋 What You'll Learn

Create an Elasticsearch watch named error_log_alert

Set the watch to run every 1 minute

Query the logs index for documents with level: error in the last 5 minutes

Trigger an action if the count of error logs is greater than 10

Send an email notification with subject Error Alert and body High number of error logs detected

💡 Why This Matters

🌍 Real World

Monitoring server logs to detect and respond to errors quickly helps keep systems reliable and reduces downtime.

💼 Career

Many IT and DevOps roles require setting up alerts in Elasticsearch to maintain system health and notify teams about issues.

1

Create the watch structure

Create a watch named error_log_alert with an empty input, condition, and actions section.

Elasticsearch

{
  "trigger": {},
  "input": {},
  "condition": {},
  "actions": {}
}
# Create the watch structure with empty sections

Need a hint?

Start by defining the main parts of the watch: trigger, input, condition, and actions.

2

Configure the trigger and input

Set the watch trigger to run every 1 minute using schedule. Set the input to a search query on the logs index that counts documents with level: error in the last 5 minutes.

Elasticsearch

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs"],
        "body": {
          "query": {
            "bool": {
              "filter": [
                { "term": { "level": "error" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {},
  "actions": {}
}
# Add trigger schedule and input search query

Need a hint?

Use the schedule interval for the trigger and a search input with a bool filter for level and timestamp.

3

Add the condition to check error count

Add a condition that checks if the search hits.total.value is greater than 10.

Elasticsearch

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs"],
        "body": {
          "query": {
            "bool": {
              "filter": [
                { "term": { "level": "error" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total.value": {
        "gt": 10
      }
    }
  },
  "actions": {}
}
# Add condition to check if error count is greater than 10

Need a hint?

Use the compare condition to check the count from the search results.

4

Add the email notification action

Add an actions section with an action named send_email that sends an email with subject set to Error Alert and body set to High number of error logs detected.

Elasticsearch

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs"],
        "body": {
          "query": {
            "bool": {
              "filter": [
                { "term": { "level": "error" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total.value": {
        "gt": 10
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "to": "[e-mail]",
        "subject": "Error Alert",
        "body": "High number of error logs detected"
      }
    }
  }
}
# Add email action to send alert

Need a hint?

Use the email action with to, subject, and body fields to send the alert.