Alerting and notifications in Elasticsearch - Step-by-Step Execution

Concept Flow - Alerting and notifications

Define Watch

↓

Set Trigger Condition

↓

Monitor Data

↓

Condition Met?

No→Wait

Yes↓

Execute Actions

↓

Send Notifications

↓

End or Repeat

This flow shows how an alert is set up, monitors data, checks conditions, and sends notifications when triggered.

Execution Sample

Elasticsearch

PUT _watcher/watch/error_watch
{
  "trigger": { "schedule": { "interval": "10s" } },
  "input": { "search": { "request": { "indices": ["logs"], "body": { "query": { "match": { "level": "error" } } } } } },
  "condition": { "compare": { "ctx.payload.hits.total.value": { "gt": 0 } } },
  "actions": { "email_admin": { "email": { "to": "[e-mail]", "subject": "Error Logs Alert", "body": "There are error logs in the last 10 seconds." } } }
}

This watch checks every 10 seconds for error logs and sends an email if any are found.

Execution Table

Step	Action	Evaluation	Result
1	Trigger fires every 10 seconds	Time reached	Proceed to input search
2	Search logs index for 'level: error'	Query executed	Hits found: 3
3	Check condition 'hits.total.value > 0'	3 > 0	True
4	Execute action: send email	Email sent to [e-mail]	Notification sent
5	Wait for next trigger	Next 10 seconds	Cycle repeats
6	Trigger fires again	Time reached	Proceed to input search
7	Search logs index for 'level: error'	Query executed	Hits found: 0
8	Check condition 'hits.total.value > 0'	0 > 0	False
9	No action executed	No notification	Wait for next trigger

💡 Execution cycles continuously every 10 seconds, sending notifications only when condition is true.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 7	After Step 8
hits.total.value	0	3	3	3	0	0
condition_met	false	N/A	true	true	N/A	false
notification_sent	false	false	false	true	false	false

Key Moments - 3 Insights

Why does the watch send an email only sometimes?

Because the condition 'hits.total.value > 0' is only true when error logs exist (see execution_table rows 3 and 8). When no errors are found, the condition is false and no email is sent.

What triggers the watch to run its check?

The watch runs every 10 seconds as defined by the trigger schedule (execution_table rows 1 and 6). This periodic trigger initiates the search and condition evaluation.

What happens if the search finds zero error logs?

The condition evaluates to false (execution_table row 8), so no actions like sending emails are executed (row 9). The watch waits for the next trigger cycle.

Visual Quiz - 3 Questions

Look at the execution table, what is the value of 'hits.total.value' after step 2?

A 3

B 0

C undefined

D 10

💡 Hint

Check the 'hits.total.value' variable in variable_tracker after Step 2.

At which step does the condition evaluate to false?

A Step 3

B Step 8

C Step 4

D Step 2

💡 Hint

Look at the 'Condition' column in execution_table rows where condition is checked.

If the trigger interval changed to 30 seconds, how would the execution table change?

A Steps would occur every 30 seconds instead of 10

B Condition would never be true

C Emails would be sent more frequently

D Search would look in a different index

💡 Hint

Refer to the trigger action timing in execution_table steps 1 and 6.

Concept Snapshot

Alerting in Elasticsearch uses watches.
A watch has a trigger (time-based), input (data query), condition (checks data), and actions (notifications).
When the condition is true, actions like emails are sent.
Watches run repeatedly as scheduled.
This helps monitor data and alert on important events.

Full Transcript

This visual execution shows how Elasticsearch alerting works using watches. First, a watch is defined with a trigger that runs every 10 seconds. When triggered, it searches the logs index for error-level logs. The condition checks if any error logs exist. If yes, an email notification is sent to the admin. The watch then waits for the next trigger cycle. If no errors are found, no notification is sent. Variables like hits.total.value track the number of errors found. The condition_met variable shows if the alert condition is true. The notification_sent variable tracks if an email was sent. This cycle repeats continuously, enabling real-time alerting based on data changes.

Practice

1. What is the main purpose of alerting in Elasticsearch?

easy

A. To automatically notify you when certain data conditions are met

B. To store large amounts of data efficiently

C. To visualize data in dashboards

D. To backup Elasticsearch indices

Solution

1. Step 1: Understand alerting concept

   Alerting watches your data and triggers notifications when specific conditions happen.

2. Step 2: Identify main purpose

   The main goal is to notify users automatically about important data changes or events.

3. Final Answer:

   To automatically notify you when certain data conditions are met -> Option A

4. Quick Check:

   Alerting = automatic notifications [OK]

Hint: Alerting means automatic notifications on data changes [OK]

Common Mistakes:

• Confusing alerting with data storage
• Thinking alerting is for data visualization
• Mixing alerting with backup processes

2. Which of the following is the correct syntax to define a trigger in an Elasticsearch alerting watch?

easy

A. "trigger": { "schedule": { "interval": "10m" } }

B. "trigger": "interval": "10m"

C. "trigger": { "interval": "10m" }

D. "trigger": { "time": "10m" }

Solution

1. Step 1: Recall trigger syntax in watch

   Triggers use a schedule object with an interval field inside curly braces.

2. Step 2: Match correct JSON structure

   "trigger": { "schedule": { "interval": "10m" } } correctly nests schedule and interval inside trigger with proper braces and quotes.

3. Final Answer:

   "trigger": { "schedule": { "interval": "10m" } } -> Option A

4. Quick Check:

   Trigger uses schedule with interval [OK]

Hint: Trigger syntax always nests schedule and interval inside braces [OK]

Common Mistakes:

• Missing braces around schedule
• Using wrong keys like 'time' instead of 'schedule'
• Incorrect JSON structure without nested objects

3. Given this watch input snippet, what type of input is being used?

{
  "input": {
    "search": {
      "request": {
        "indices": ["logs"],
        "body": {
          "query": { "match_all": {} }
        }
      }
    }
  }
}

medium

A. Webhook input

B. HTTP input

C. Search input

D. Script input

Solution

1. Step 1: Identify input type from JSON keys

   The input uses the key "search" with a request containing indices and a query.

2. Step 2: Match input type to Elasticsearch alerting inputs

   This matches the Search input type, which runs a search query on indices.

3. Final Answer:

   Search input -> Option C

4. Quick Check:

   Input with "search" key = Search input [OK]

Hint: Look for 'search' key to identify Search input type [OK]

Common Mistakes:

• Confusing search input with HTTP or webhook inputs
• Ignoring the 'search' key and guessing script input
• Not recognizing the query structure inside input

4. You wrote this action in your watch but it fails to send an email:

"actions": {
  "send_email": {
    "email": {
      "to": "[e-mail]",
      "subject": "Alert!",
      "body": "Condition met"
    }
  }
}

What is the likely error?

medium

A. Incorrect 'to' email format

B. Body must be an object, not a string

C. Missing 'trigger' section in watch

D. Missing 'from' field in email action

Solution

1. Step 1: Check required fields for email action

   Email action requires a 'from' field to specify sender address.

2. Step 2: Identify missing 'from' field

   The given action lacks the 'from' field, causing failure to send email.

3. Final Answer:

   Missing 'from' field in email action -> Option D

4. Quick Check:

   Email action needs 'from' field [OK]

Hint: Email actions always need a 'from' address [OK]

Common Mistakes:

• Assuming 'to' format is wrong when it is correct
• Forgetting to add 'from' sender email
• Thinking trigger absence causes email failure

5. You want to create an alert that sends a Slack message only if the number of errors in logs exceeds 100 in the last 5 minutes. Which condition correctly implements this in the watch?

hard

A. "condition": { "script": { "source": "return ctx.payload.hits.total.value > 100" } }

B. "condition": { "compare": { "ctx.payload.hits.total.value": { "gt": 100 } } }

C. "condition": { "script": { "source": "return ctx.payload.hits.total > 100" } }

D. "condition": { "compare": { "ctx.payload.hits.total": { "gte": 100 } } }

Solution

1. Step 1: Understand payload structure for hits total

   In Elasticsearch 7+, total hits count is accessed as ctx.payload.hits.total.value.

2. Step 2: Choose correct condition syntax

   The compare condition with 'gt' operator on ctx.payload.hits.total.value correctly checks if errors exceed 100.

3. Final Answer:

   "condition": { "compare": { "ctx.payload.hits.total.value": { "gt": 100 } } } -> Option B

4. Quick Check:

   Use compare with ctx.payload.hits.total.value > 100 [OK]

Hint: Use compare on ctx.payload.hits.total.value for counts [OK]

Common Mistakes:

• Using ctx.payload.hits.total instead of .value
• Using script with wrong field name
• Using 'gte' instead of 'gt' when strictly greater needed