Log management pipeline in Elasticsearch - Mini Project: Build & Apply

Log Management Pipeline

📖 Scenario: You work as a system administrator managing server logs. You want to organize logs in Elasticsearch to quickly find errors and monitor system health.

🎯 Goal: Build a simple Elasticsearch index and pipeline to store logs, filter error logs, and add a timestamp field.

📋 What You'll Learn

Create an Elasticsearch index called server_logs with fields message and level

Define a pipeline that adds a timestamp field with the current time

Filter logs to only include those with level equal to error

Ingest sample logs using the pipeline

💡 Why This Matters

🌍 Real World

System administrators and DevOps engineers use Elasticsearch pipelines to organize and filter logs for monitoring and troubleshooting.

💼 Career

Understanding how to create indices and pipelines in Elasticsearch is essential for roles involving log management, monitoring, and data analysis.

1

Create the server_logs index

Create an Elasticsearch index called server_logs with two fields: message of type text and level of type keyword. Write the JSON mapping for this index.

Elasticsearch

{
  "mappings": {
    "properties": {
      // Define fields here
    }
  }
}

Hint

Use mappings to define fields. message should be text for full-text search. level should be keyword for exact matching.

2

Define an ingest pipeline to add a timestamp

Create an ingest pipeline called add_timestamp that adds a timestamp field with the current date and time using the set processor.

Elasticsearch

{
  "mappings": {
    "properties": {
      "message": { "type": "text" },
      "level": { "type": "keyword" }
    }
  }
}

{
  "description": "Add current timestamp",
  "processors": [
    // Your code here
  ]
}

Hint

Use the set processor to add a field. The value {{_ingest.timestamp}} inserts the current time.

3

Filter logs to only include errors

Add a pipeline processor to filter logs so only documents with level equal to error are processed further. Use the drop processor inside a conditional processor to drop non-error logs.

Elasticsearch

{
  "mappings": {
    "properties": {
      "message": { "type": "text" },
      "level": { "type": "keyword" }
    }
  }
}

{
  "description": "Add current timestamp and filter errors",
  "processors": [
    {
      "set": {
        "field": "timestamp",
        "value": "{{_ingest.timestamp}}"
      }
    },
    // Your code here
  ]
}

Hint

Use the drop processor with an if condition to remove logs where level is not error.

4

Ingest sample logs using the pipeline

Index two sample log documents into the server_logs index using the add_timestamp pipeline. The first log has message "Disk full" and level "error". The second log has message "User login" and level "info".

Elasticsearch

{
  "mappings": {
    "properties": {
      "message": { "type": "text" },
      "level": { "type": "keyword" }
    }
  }
}

{
  "description": "Add current timestamp and filter errors",
  "processors": [
    {
      "set": {
        "field": "timestamp",
        "value": "{{_ingest.timestamp}}"
      }
    },
    {
      "drop": {
        "if": "ctx.level != 'error'"
      }
    }
  ]
}

// Your code here to index documents

Hint

Use the POST method to index documents with the pipeline parameter set to add_timestamp.

Practice

1. What is the main purpose of a log management pipeline in Elasticsearch?

easy

A. To encrypt data before sending it to Elasticsearch

B. To create visual dashboards from raw data

C. To collect, process, and store logs for easy searching and alerting

D. To backup Elasticsearch indices automatically

Solution

1. Step 1: Understand the role of a log management pipeline

   A log management pipeline is designed to handle logs by collecting, processing, and storing them.

2. Step 2: Identify the main goal

   The goal is to organize logs so they can be searched easily and alerts can be created.

3. Final Answer:

   To collect, process, and store logs for easy searching and alerting -> Option C

4. Quick Check:

   Log pipeline purpose = collect, process, store logs [OK]

Hint: Remember: pipeline = collect + process + store logs [OK]

Common Mistakes:

• Confusing log pipeline with visualization tools
• Thinking it only backs up data
• Assuming it encrypts logs by default

2. Which section is NOT part of a typical Elasticsearch log management pipeline configuration?

easy

A. authentication

B. filter

C. output

D. input

Solution

1. Step 1: Recall pipeline sections

   A typical pipeline has input, filter, and output sections to handle logs.

2. Step 2: Identify the section not included

   Authentication is not a standard section in the pipeline configuration; it is handled elsewhere.

3. Final Answer:

   authentication -> Option A

4. Quick Check:

   Pipeline sections = input, filter, output [OK]

Hint: Pipeline = input + filter + output only [OK]

Common Mistakes:

• Thinking authentication is part of pipeline config
• Confusing pipeline sections with security settings
• Assuming output means authentication

3. Given this pipeline snippet, what will be the output field after processing?

{
  "input": { "type": "file", "path": "/var/log/app.log" },
  "filter": { "grok": { "match": { "message": "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:msg}" } } },
  "output": { "elasticsearch": { "index": "app-logs" } }
}

medium

A. The original message field is deleted

B. A new field named 'msg' extracted from the log message

C. Logs are sent to a file instead of Elasticsearch

D. The timestamp field is removed

Solution

1. Step 1: Analyze the filter section

   The grok filter extracts parts of the log message into fields: timestamp, level, and msg.

2. Step 2: Determine output effect

   The output sends logs to Elasticsearch index 'app-logs' with the new fields added, including 'msg'.

3. Final Answer:

   A new field named 'msg' extracted from the log message -> Option B

4. Quick Check:

   Grok adds 'msg' field from message [OK]

Hint: Grok filter extracts fields like 'msg' from logs [OK]

Common Mistakes:

• Assuming original message is deleted
• Thinking output sends logs to a file
• Believing timestamp is removed

4. Identify the error in this pipeline configuration snippet:

{
  "input": { "type": "file", "path": "/var/log/app.log" },
  "filter": { "grok": { "match": { "message": "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}" } } },
  "output": { "elasticsearch": { "index": "app-logs" }
}

medium

A. Input type 'file' is invalid

B. Incorrect grok pattern syntax

C. Output index name cannot contain hyphens

D. Missing closing brace for the output section

Solution

1. Step 1: Check JSON structure

   The output section is missing a closing brace '}' at the end, causing invalid JSON.

2. Step 2: Validate other parts

   The grok pattern syntax is correct, input type 'file' is valid, and index names can have hyphens.

3. Final Answer:

   Missing closing brace for the output section -> Option D

4. Quick Check:

   JSON braces must be balanced [OK]

Hint: Check all braces and commas in JSON config [OK]

Common Mistakes:

• Ignoring missing braces causing syntax errors
• Assuming grok pattern is wrong without checking
• Thinking index names can't have hyphens

5. You want to create a log management pipeline that drops logs with level 'DEBUG' and adds a new field 'environment' with value 'production'. Which filter configuration achieves this?

hard

A. { "drop": { "if": "[level] == 'DEBUG'" }, "mutate": { "add_field": { "environment": "production" } } }

B. { "if": "[level] == 'DEBUG'", "drop": {}, "add_field": { "environment": "production" } }

C. { "mutate": { "drop": "[level] == 'DEBUG'", "add_field": { "environment": "production" } } }

D. { "filter": { "drop": { "condition": "level == 'DEBUG'" }, "add_field": { "environment": "production" } } }

Solution

1. Step 1: Understand filter syntax for dropping logs

   The 'drop' filter uses an 'if' condition to remove logs matching criteria.

2. Step 2: Add a new field using 'mutate' filter

   The 'mutate' filter's 'add_field' adds new fields to the log event.

3. Step 3: Combine drop and mutate correctly

   { "drop": { "if": "[level] == 'DEBUG'" }, "mutate": { "add_field": { "environment": "production" } } } correctly uses 'drop' with 'if' and 'mutate' with 'add_field' in the right structure.

4. Final Answer:

   { "drop": { "if": "[level] == 'DEBUG'" }, "mutate": { "add_field": { "environment": "production" } } } -> Option A

5. Quick Check:

   Drop with if + mutate add_field = { "drop": { "if": "[level] == 'DEBUG'" }, "mutate": { "add_field": { "environment": "production" } } } [OK]

Hint: Use 'drop' with 'if' and 'mutate' to add fields [OK]

Common Mistakes:

• Placing 'drop' inside 'mutate' incorrectly
• Using wrong syntax for conditions
• Trying to add fields inside 'drop' filter