Machine learning anomaly detection in Elasticsearch - Mini Project: Build & Apply

Machine learning anomaly detection

📖 Scenario: You work for a company that collects website traffic data. You want to find unusual spikes or drops in the number of visitors using Elasticsearch's machine learning anomaly detection.

🎯 Goal: Create a simple anomaly detection job in Elasticsearch to identify unusual visitor counts over time.

📋 What You'll Learn

Create an index with sample visitor count data

Define a machine learning job configuration

Use the job to detect anomalies in visitor counts

Output the anomaly detection results

💡 Why This Matters

🌍 Real World

Detecting unusual website traffic spikes helps companies respond quickly to potential issues or opportunities.

💼 Career

Anomaly detection skills are valuable for data scientists and engineers working with monitoring, security, and business intelligence.

1

Create sample visitor count data index

Create an Elasticsearch index called visitor_counts with documents containing timestamp and count fields. Insert these exact 5 documents: {"timestamp": "2024-01-01T00:00:00Z", "count": 100}, {"timestamp": "2024-01-02T00:00:00Z", "count": 110}, {"timestamp": "2024-01-03T00:00:00Z", "count": 95}, {"timestamp": "2024-01-04T00:00:00Z", "count": 300}, {"timestamp": "2024-01-05T00:00:00Z", "count": 105}.

Elasticsearch

# Create index visitor_counts and insert 5 documents with timestamp and count
# Your code here

Hint

Use PUT to create the index with mappings, then POST _bulk to insert documents.

2

Define anomaly detection job configuration

Create a machine learning job configuration called visitor_count_anomaly_job that analyzes the visitor_counts index. Use timestamp as the time field and count as the analysis field.

Elasticsearch

PUT /visitor_counts
{
  "mappings": {
    "properties": {
      "timestamp": {"type": "date"},
      "count": {"type": "integer"}
    }
  }
}

POST /visitor_counts/_bulk
{ "index": {} }
{ "timestamp": "2024-01-01T00:00:00Z", "count": 100 }
{ "index": {} }
{ "timestamp": "2024-01-02T00:00:00Z", "count": 110 }
{ "index": {} }
{ "timestamp": "2024-01-03T00:00:00Z", "count": 95 }
{ "index": {} }
{ "timestamp": "2024-01-04T00:00:00Z", "count": 300 }
{ "index": {} }
{ "timestamp": "2024-01-05T00:00:00Z", "count": 105 }

# Create ML job visitor_count_anomaly_job with timestamp and count fields
# Your code here

Hint

Use PUT _ml/anomaly_detectors/visitor_count_anomaly_job with analysis_config and data_description.

3

Start the datafeed to run anomaly detection

Start the datafeed called datafeed-visitor_count_anomaly_job to begin analyzing data from the visitor_counts index.

Elasticsearch

PUT /visitor_counts
{
  "mappings": {
    "properties": {
      "timestamp": {"type": "date"},
      "count": {"type": "integer"}
    }
  }
}

POST /visitor_counts/_bulk
{ "index": {} }
{ "timestamp": "2024-01-01T00:00:00Z", "count": 100 }
{ "index": {} }
{ "timestamp": "2024-01-02T00:00:00Z", "count": 110 }
{ "index": {} }
{ "timestamp": "2024-01-03T00:00:00Z", "count": 95 }
{ "index": {} }
{ "timestamp": "2024-01-04T00:00:00Z", "count": 300 }
{ "index": {} }
{ "timestamp": "2024-01-05T00:00:00Z", "count": 105 }

PUT _ml/anomaly_detectors/visitor_count_anomaly_job
{
  "description": "Detect anomalies in visitor counts",
  "analysis_config": {
    "bucket_span": "1d",
    "detectors": [
      {
        "function": "mean",
        "field_name": "count"
      }
    ]
  },
  "data_description": {
    "time_field": "timestamp"
  },
  "datafeed_config": {
    "indices": ["visitor_counts"]
  }
}

# Start the datafeed datafeed-visitor_count_anomaly_job
# Your code here

Hint

Use POST _ml/datafeeds/datafeed-visitor_count_anomaly_job/_start to start the datafeed.

4

Get anomaly detection results

Retrieve the anomaly detection results for the job visitor_count_anomaly_job and print the bucket_score for each bucket.

Elasticsearch

PUT /visitor_counts
{
  "mappings": {
    "properties": {
      "timestamp": {"type": "date"},
      "count": {"type": "integer"}
    }
  }
}

POST /visitor_counts/_bulk
{ "index": {} }
{ "timestamp": "2024-01-01T00:00:00Z", "count": 100 }
{ "index": {} }
{ "timestamp": "2024-01-02T00:00:00Z", "count": 110 }
{ "index": {} }
{ "timestamp": "2024-01-03T00:00:00Z", "count": 95 }
{ "index": {} }
{ "timestamp": "2024-01-04T00:00:00Z", "count": 300 }
{ "index": {} }
{ "timestamp": "2024-01-05T00:00:00Z", "count": 105 }

PUT _ml/anomaly_detectors/visitor_count_anomaly_job
{
  "description": "Detect anomalies in visitor counts",
  "analysis_config": {
    "bucket_span": "1d",
    "detectors": [
      {
        "function": "mean",
        "field_name": "count"
      }
    ]
  },
  "data_description": {
    "time_field": "timestamp"
  },
  "datafeed_config": {
    "indices": ["visitor_counts"]
  }
}

POST _ml/datafeeds/datafeed-visitor_count_anomaly_job/_start

# Retrieve anomaly results and print bucket_score
# Your code here

Hint

Use GET _ml/anomaly_detectors/visitor_count_anomaly_job/results/buckets to get anomaly scores.

Practice

1. What is the main purpose of machine learning anomaly detection in Elasticsearch?

easy

A. To automatically find unusual patterns in data

B. To store large amounts of data efficiently

C. To create visual dashboards for data

D. To backup Elasticsearch clusters

Solution

1. Step 1: Understand anomaly detection goal

   Machine learning anomaly detection is designed to find unusual or unexpected patterns in data automatically.

2. Step 2: Compare options with purpose

   Options B, C, and D describe other Elasticsearch features, not anomaly detection.

3. Final Answer:

   To automatically find unusual patterns in data -> Option A

4. Quick Check:

   Purpose of anomaly detection = find unusual patterns [OK]

Hint: Anomaly detection finds unusual data automatically [OK]

Common Mistakes:

• Confusing anomaly detection with data storage
• Thinking anomaly detection creates dashboards
• Mixing anomaly detection with backup tasks

2. Which Elasticsearch API call starts the anomaly detection process by feeding data to the job?

easy

A. POST _ml/anomaly_detectors/<job_id>/_start_datafeed

B. GET _ml/anomaly_detectors/<job_id>/results

C. PUT _ml/anomaly_detectors/<job_id>

D. DELETE _ml/anomaly_detectors/<job_id>

Solution

1. Step 1: Identify datafeed start API

   The API to start feeding data to an anomaly detection job is POST _ml/anomaly_detectors/<job_id>/_start_datafeed.

2. Step 2: Eliminate other options

   GET retrieves results, PUT creates or updates jobs, DELETE removes jobs.

3. Final Answer:

   POST _ml/anomaly_detectors/<job_id>/_start_datafeed -> Option A

4. Quick Check:

   Start datafeed = POST _start_datafeed [OK]

Hint: Start datafeed uses POST with _start_datafeed endpoint [OK]

Common Mistakes:

• Using GET instead of POST to start datafeed
• Confusing job creation with starting datafeed
• Deleting job instead of starting datafeed

3. Given this Elasticsearch ML job result snippet:

{"job_id":"sales_anomaly","results":[{"timestamp":1680000000000,"anomaly_score":75},{"timestamp":1680003600000,"anomaly_score":5}]}

Which timestamp shows a likely anomaly?

medium

A. Neither timestamp

B. 1680003600000

C. Both timestamps

D. 1680000000000

Solution

1. Step 1: Understand anomaly score meaning

   Higher anomaly scores indicate more unusual data points. A score of 75 is high, 5 is low.

2. Step 2: Identify timestamp with high score

   The timestamp 1680000000000 has anomaly_score 75, indicating a likely anomaly.

3. Final Answer:

   1680000000000 -> Option D

4. Quick Check:

   High anomaly score = likely anomaly [OK]

Hint: Higher anomaly_score means more likely anomaly [OK]

Common Mistakes:

• Choosing low anomaly score as anomaly
• Selecting both timestamps without checking scores
• Ignoring anomaly_score values

4. You created an anomaly detection job but see no results after starting the datafeed. What is a likely cause?

medium

A. The job was deleted before starting

B. The Elasticsearch cluster is offline

C. The datafeed is not running or has stopped

D. The anomaly scores are all zero

Solution

1. Step 1: Check datafeed status

   If no results appear, the datafeed may not be running or has stopped feeding data to the job.

2. Step 2: Evaluate other options

   Job deletion would prevent starting datafeed; cluster offline causes broader failures; zero scores still produce results.

3. Final Answer:

   The datafeed is not running or has stopped -> Option C

4. Quick Check:

   No results usually mean datafeed stopped [OK]

Hint: No results? Check if datafeed is running [OK]

Common Mistakes:

• Assuming zero scores mean no results
• Ignoring datafeed status
• Blaming cluster offline without checking datafeed

5. You want to detect unusual spikes in website traffic using Elasticsearch ML anomaly detection. Which steps should you follow to set this up correctly?

hard

A. Backup traffic data, create index pattern, then visualize spikes

B. Create a job with traffic data, start datafeed, then analyze anomaly results

C. Create a dashboard, upload traffic logs, then run anomaly detection manually

D. Delete old data, create job without datafeed, then check results

Solution

1. Step 1: Create ML job with traffic data

   Define an anomaly detection job using the website traffic data to analyze patterns.

2. Step 2: Start the datafeed to feed data into the job

   Start the datafeed so the job can process incoming traffic data continuously.

3. Step 3: Analyze the anomaly detection results

   Review the results to identify unusual spikes or anomalies in traffic.

4. Final Answer:

   Create a job with traffic data, start datafeed, then analyze anomaly results -> Option B

5. Quick Check:

   Job + datafeed + analyze = correct setup [OK]

Hint: Job creation + datafeed start + result check = setup [OK]

Common Mistakes:

• Skipping datafeed start step
• Confusing dashboards with anomaly detection setup
• Deleting data before analysis