Machine learning anomaly detection in Elasticsearch

Introduction

Machine learning anomaly detection helps find unusual patterns in data automatically. It spots things that don't fit normal behavior.

Detecting unusual spikes in website traffic that might mean a problem or attack.

Finding errors or faults in machines by monitoring sensor data.

Spotting fraud in financial transactions by noticing strange activity.

Monitoring server logs to catch unexpected errors or failures.

Checking user behavior to identify potential security breaches.

Syntax

Elasticsearch

POST _ml/anomaly_detectors/<job_id>
{
  "description": "Detect anomalies in data",
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "function": "mean",
        "field_name": "value"
      }
    ]
  },
  "data_description": {
    "time_field": "timestamp"
  }
}

job_id is a unique name for your anomaly detection job.

bucket_span defines the time window size for analysis, like 15 minutes.

Examples

This example creates a job to find unusual total sales per hour.

Elasticsearch

POST _ml/anomaly_detectors/sales_anomaly_job
{
  "description": "Detect anomalies in sales data",
  "analysis_config": {
    "bucket_span": "1h",
    "detectors": [
      {
        "function": "sum",
        "field_name": "sales_amount"
      }
    ]
  },
  "data_description": {
    "time_field": "sale_time"
  }
}

This job looks for high CPU usage every 5 minutes.

Elasticsearch

POST _ml/anomaly_detectors/cpu_usage_job
{
  "description": "Detect CPU usage spikes",
  "analysis_config": {
    "bucket_span": "5m",
    "detectors": [
      {
        "function": "max",
        "field_name": "cpu_percent"
      }
    ]
  },
  "data_description": {
    "time_field": "timestamp"
  }
}

Sample Program

This example creates an anomaly detection job for temperature data, opens the job, starts a datafeed to read from the 'sensor_data' index, and then fetches detected anomalies.

Elasticsearch

POST _ml/anomaly_detectors/temperature_anomaly_job
{
  "description": "Detect temperature anomalies",
  "analysis_config": {
    "bucket_span": "10m",
    "detectors": [
      {
        "function": "mean",
        "field_name": "temperature"
      }
    ]
  },
  "data_description": {
    "time_field": "timestamp"
  }
}

POST _ml/anomaly_detectors/temperature_anomaly_job/_open

POST _ml/datafeeds/datafeed-temperature_anomaly_job
{
  "job_id": "temperature_anomaly_job",
  "indices": ["sensor_data"]
}

POST _ml/datafeeds/datafeed-temperature_anomaly_job/_start

GET _ml/anomaly_detectors/temperature_anomaly_job/results/anomalies

Important Notes

Always choose a bucket_span that matches how often your data updates.

After creating a job, you must open it before starting the datafeed.

Check anomaly scores to decide if a result is important; higher scores mean more unusual.

Summary

Machine learning anomaly detection finds unusual data patterns automatically.

Use it to monitor systems, detect fraud, or find errors early.

In Elasticsearch, create a job, open it, start a datafeed, then check results.