Alerting and notifications in Elasticsearch - Mini Project: Build & Apply

Alerting and notifications

📖 Scenario: You are managing a server that stores logs in Elasticsearch. You want to get alerts when the number of error logs goes above a certain limit. This helps you fix problems quickly.

🎯 Goal: Create a watch in Elasticsearch that checks the number of error logs in the last 5 minutes. If the count is more than 10, send an email notification.

📋 What You'll Learn

Create an Elasticsearch watch named error_log_alert

Set the watch to run every 1 minute

Query the logs index for documents with level: error in the last 5 minutes

Trigger an action if the count of error logs is greater than 10

Send an email notification with subject Error Alert and body High number of error logs detected

💡 Why This Matters

🌍 Real World

Monitoring server logs to detect and respond to errors quickly helps keep systems reliable and reduces downtime.

💼 Career

Many IT and DevOps roles require setting up alerts in Elasticsearch to maintain system health and notify teams about issues.

1

Create the watch structure

Create a watch named error_log_alert with an empty input, condition, and actions section.

Elasticsearch

{
  "trigger": {},
  "input": {},
  "condition": {},
  "actions": {}
}
# Create the watch structure with empty sections

Hint

Start by defining the main parts of the watch: trigger, input, condition, and actions.

2

Configure the trigger and input

Set the watch trigger to run every 1 minute using schedule. Set the input to a search query on the logs index that counts documents with level: error in the last 5 minutes.

Elasticsearch

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs"],
        "body": {
          "query": {
            "bool": {
              "filter": [
                { "term": { "level": "error" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {},
  "actions": {}
}
# Add trigger schedule and input search query

Hint

Use the schedule interval for the trigger and a search input with a bool filter for level and timestamp.

3

Add the condition to check error count

Add a condition that checks if the search hits.total.value is greater than 10.

Elasticsearch

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs"],
        "body": {
          "query": {
            "bool": {
              "filter": [
                { "term": { "level": "error" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total.value": {
        "gt": 10
      }
    }
  },
  "actions": {}
}
# Add condition to check if error count is greater than 10

Hint

Use the compare condition to check the count from the search results.

4

Add the email notification action

Add an actions section with an action named send_email that sends an email with subject set to Error Alert and body set to High number of error logs detected.

Elasticsearch

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs"],
        "body": {
          "query": {
            "bool": {
              "filter": [
                { "term": { "level": "error" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total.value": {
        "gt": 10
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "to": "[e-mail]",
        "subject": "Error Alert",
        "body": "High number of error logs detected"
      }
    }
  }
}
# Add email action to send alert

Hint

Use the email action with to, subject, and body fields to send the alert.

Practice

1. What is the main purpose of alerting in Elasticsearch?

easy

A. To automatically notify you when certain data conditions are met

B. To store large amounts of data efficiently

C. To visualize data in dashboards

D. To backup Elasticsearch indices

Solution

1. Step 1: Understand alerting concept

   Alerting watches your data and triggers notifications when specific conditions happen.

2. Step 2: Identify main purpose

   The main goal is to notify users automatically about important data changes or events.

3. Final Answer:

   To automatically notify you when certain data conditions are met -> Option A

4. Quick Check:

   Alerting = automatic notifications [OK]

Hint: Alerting means automatic notifications on data changes [OK]

Common Mistakes:

• Confusing alerting with data storage
• Thinking alerting is for data visualization
• Mixing alerting with backup processes

2. Which of the following is the correct syntax to define a trigger in an Elasticsearch alerting watch?

easy

A. "trigger": { "schedule": { "interval": "10m" } }

B. "trigger": "interval": "10m"

C. "trigger": { "interval": "10m" }

D. "trigger": { "time": "10m" }

Solution

1. Step 1: Recall trigger syntax in watch

   Triggers use a schedule object with an interval field inside curly braces.

2. Step 2: Match correct JSON structure

   "trigger": { "schedule": { "interval": "10m" } } correctly nests schedule and interval inside trigger with proper braces and quotes.

3. Final Answer:

   "trigger": { "schedule": { "interval": "10m" } } -> Option A

4. Quick Check:

   Trigger uses schedule with interval [OK]

Hint: Trigger syntax always nests schedule and interval inside braces [OK]

Common Mistakes:

• Missing braces around schedule
• Using wrong keys like 'time' instead of 'schedule'
• Incorrect JSON structure without nested objects

3. Given this watch input snippet, what type of input is being used?

{
  "input": {
    "search": {
      "request": {
        "indices": ["logs"],
        "body": {
          "query": { "match_all": {} }
        }
      }
    }
  }
}

medium

A. Webhook input

B. HTTP input

C. Search input

D. Script input

Solution

1. Step 1: Identify input type from JSON keys

   The input uses the key "search" with a request containing indices and a query.

2. Step 2: Match input type to Elasticsearch alerting inputs

   This matches the Search input type, which runs a search query on indices.

3. Final Answer:

   Search input -> Option C

4. Quick Check:

   Input with "search" key = Search input [OK]

Hint: Look for 'search' key to identify Search input type [OK]

Common Mistakes:

• Confusing search input with HTTP or webhook inputs
• Ignoring the 'search' key and guessing script input
• Not recognizing the query structure inside input

4. You wrote this action in your watch but it fails to send an email:

"actions": {
  "send_email": {
    "email": {
      "to": "[e-mail]",
      "subject": "Alert!",
      "body": "Condition met"
    }
  }
}

What is the likely error?

medium

A. Incorrect 'to' email format

B. Body must be an object, not a string

C. Missing 'trigger' section in watch

D. Missing 'from' field in email action

Solution

1. Step 1: Check required fields for email action

   Email action requires a 'from' field to specify sender address.

2. Step 2: Identify missing 'from' field

   The given action lacks the 'from' field, causing failure to send email.

3. Final Answer:

   Missing 'from' field in email action -> Option D

4. Quick Check:

   Email action needs 'from' field [OK]

Hint: Email actions always need a 'from' address [OK]

Common Mistakes:

• Assuming 'to' format is wrong when it is correct
• Forgetting to add 'from' sender email
• Thinking trigger absence causes email failure

5. You want to create an alert that sends a Slack message only if the number of errors in logs exceeds 100 in the last 5 minutes. Which condition correctly implements this in the watch?

hard

A. "condition": { "script": { "source": "return ctx.payload.hits.total.value > 100" } }

B. "condition": { "compare": { "ctx.payload.hits.total.value": { "gt": 100 } } }

C. "condition": { "script": { "source": "return ctx.payload.hits.total > 100" } }

D. "condition": { "compare": { "ctx.payload.hits.total": { "gte": 100 } } }

Solution

1. Step 1: Understand payload structure for hits total

   In Elasticsearch 7+, total hits count is accessed as ctx.payload.hits.total.value.

2. Step 2: Choose correct condition syntax

   The compare condition with 'gt' operator on ctx.payload.hits.total.value correctly checks if errors exceed 100.

3. Final Answer:

   "condition": { "compare": { "ctx.payload.hits.total.value": { "gt": 100 } } } -> Option B

4. Quick Check:

   Use compare with ctx.payload.hits.total.value > 100 [OK]

Hint: Use compare on ctx.payload.hits.total.value for counts [OK]

Common Mistakes:

• Using ctx.payload.hits.total instead of .value
• Using script with wrong field name
• Using 'gte' instead of 'gt' when strictly greater needed