Machine learning anomaly detection in Elasticsearch - Step-by-Step Execution

Concept Flow - Machine learning anomaly detection

Input Data

↓

Feature Extraction

↓

Train ML Model

↓

Detect Anomalies

↓

Output Anomaly Scores

↓

Alert or Visualize

Data flows from input through feature extraction, model training, anomaly detection, and finally outputs anomaly scores for alerts or visualization.

Execution Sample

Elasticsearch

POST _ml/anomaly_detectors/my_detector/_evaluate
{
  "data": [
    {"timestamp": "2024-06-01T00:00:00Z", "value": 10},
    {"timestamp": "2024-06-01T01:00:00Z", "value": 1000}
  ]
}

This request sends sample data to an Elasticsearch ML anomaly detector to evaluate anomaly scores.

Execution Table

Step	Input Data Point	Feature Extracted	Model Score	Anomaly Score	Action
1	{"timestamp": "2024-06-01T00:00:00Z", "value": 10}	value=10	Normal	Low (0.1)	No alert
2	{"timestamp": "2024-06-01T01:00:00Z", "value": 1000}	value=1000	Anomalous	High (0.95)	Alert generated
3	End of data	-	-	-	Stop evaluation

💡 All data points processed, anomaly scores assigned, evaluation stops.

Variable Tracker

Variable	Start	After 1	After 2	Final
value	N/A	10	1000	1000
anomaly_score	N/A	0.1	0.95	0.95
alert	False	False	True	True

Key Moments - 2 Insights

Why does the anomaly score jump from 0.1 to 0.95 between the two data points?

Because the second value (1000) is very different from normal values seen before, the model flags it as anomalous, shown in execution_table row 2.

What does a low anomaly score mean in this context?

A low anomaly score means the data point fits the normal pattern learned by the model, so no alert is triggered, as seen in execution_table row 1.

Visual Quiz - 3 Questions

Look at the execution table, what is the anomaly score for the first data point?

A 1.0

B 0.95

C 0.1

D 0.5

💡 Hint

Check the 'Anomaly Score' column in row 1 of the execution_table.

At which step does the model generate an alert?

A Step 1

B Step 2

C Step 3

D No alert generated

💡 Hint

Look at the 'Action' column in the execution_table for when alert is True.

If the second data point had a value of 15 instead of 1000, how would the anomaly score likely change?

A It would stay low, similar to 0.1

B It would be very high, close to 0.95

C It would be exactly 0.5

D It would cause an error

💡 Hint

Refer to variable_tracker for how anomaly_score changes with value differences.

Concept Snapshot

Machine learning anomaly detection in Elasticsearch:
- Input data is processed to extract features.
- ML model learns normal patterns.
- Each data point gets an anomaly score.
- High scores indicate unusual data.
- Alerts or visualizations help monitor anomalies.

Full Transcript

Machine learning anomaly detection in Elasticsearch works by taking input data and extracting features like numeric values. The ML model is trained to understand what normal data looks like. When new data points arrive, the model scores them based on how unusual they are. Low anomaly scores mean the data fits normal patterns, while high scores mean the data is unusual and may indicate a problem. Alerts can be generated for high anomaly scores to notify users. This process helps monitor data streams for unexpected changes or errors.