Log management pipeline in Elasticsearch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to define an Elasticsearch index for logs with a timestamp field.

Elasticsearch

{
  "mappings": {
    "properties": {
      "timestamp": { "type": [1] }
    }
  }
}

Drag options to blanks, or click blank then click option'

Akeyword

Btext

Cdate

Dinteger

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'text' or 'keyword' type for timestamp fields.

Using 'integer' type for dates.

The timestamp field should be of type 'date' to store time values correctly.

2fill in blank

medium

Complete the pipeline processor to parse a JSON log message.

Elasticsearch

{
  "processors": [
    {
      "json": {
        "field": [1]
      }
    }
  ]
}

Drag options to blanks, or click blank then click option'

A"timestamp"

B"message"

C"log"

D"source"

Attempts:

3 left

💡 Hint

Common Mistakes

Parsing the wrong field like 'timestamp' or 'source'.

Forgetting to quote the field name.

The JSON processor parses the 'message' field containing the JSON log string.

3fill in blank

hard

Fix the error in the ingest pipeline to add a new field with a static value.

Elasticsearch

{
  "processors": [
    {
      "set": {
        "field": "log_level",
        "value": [1]
      }
    }
  ]
}

Drag options to blanks, or click blank then click option'

A"INFO"

BINFO

Cinfo

D'INFO'

Attempts:

3 left

💡 Hint

Common Mistakes

Using unquoted strings causing JSON errors.

Using single quotes instead of double quotes.

The value must be a string, so it needs to be quoted with double quotes.

4fill in blank

hard

Fill both blanks to create a pipeline that drops logs with level 'debug'.

Elasticsearch

{
  "processors": [
    {
      "drop": {
        "if": "ctx.[1] == '[2]'"
      }
    }
  ]
}

Drag options to blanks, or click blank then click option'

Alog_level

Bdebug

Clevel

Dinfo

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong field names like 'level'.

Checking for 'info' instead of 'debug'.

The condition checks if 'log_level' equals 'debug' to drop those logs.

5fill in blank

hard

Fill all three blanks to create a pipeline that renames 'host' to 'hostname', adds a tag, and removes 'temp_field'.

Elasticsearch

{
  "processors": [
    {
      "rename": {
        "field": [1],
        "target_field": [2]
      }
    },
    {
      "append": {
        "field": "tags",
        "value": [[3]]
      }
    },
    {
      "remove": {
        "field": "temp_field"
      }
    }
  ]
}

Drag options to blanks, or click blank then click option'

A"host"

B"hostname"

C"processed"

D"temp"

Attempts:

3 left

💡 Hint

Common Mistakes

Not quoting field names and tag values.

Using wrong tag values like 'temp'.

Rename 'host' to 'hostname', add tag 'processed', and remove 'temp_field'.

Practice

1. What is the main purpose of a log management pipeline in Elasticsearch?

easy

A. To encrypt data before sending it to Elasticsearch

B. To create visual dashboards from raw data

C. To collect, process, and store logs for easy searching and alerting

D. To backup Elasticsearch indices automatically

Solution

1. Step 1: Understand the role of a log management pipeline

   A log management pipeline is designed to handle logs by collecting, processing, and storing them.

2. Step 2: Identify the main goal

   The goal is to organize logs so they can be searched easily and alerts can be created.

3. Final Answer:

   To collect, process, and store logs for easy searching and alerting -> Option C

4. Quick Check:

   Log pipeline purpose = collect, process, store logs [OK]

Hint: Remember: pipeline = collect + process + store logs [OK]

Common Mistakes:

• Confusing log pipeline with visualization tools
• Thinking it only backs up data
• Assuming it encrypts logs by default

2. Which section is NOT part of a typical Elasticsearch log management pipeline configuration?

easy

A. authentication

B. filter

C. output

D. input

Solution

1. Step 1: Recall pipeline sections

   A typical pipeline has input, filter, and output sections to handle logs.

2. Step 2: Identify the section not included

   Authentication is not a standard section in the pipeline configuration; it is handled elsewhere.

3. Final Answer:

   authentication -> Option A

4. Quick Check:

   Pipeline sections = input, filter, output [OK]

Hint: Pipeline = input + filter + output only [OK]

Common Mistakes:

• Thinking authentication is part of pipeline config
• Confusing pipeline sections with security settings
• Assuming output means authentication

3. Given this pipeline snippet, what will be the output field after processing?

{
  "input": { "type": "file", "path": "/var/log/app.log" },
  "filter": { "grok": { "match": { "message": "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:msg}" } } },
  "output": { "elasticsearch": { "index": "app-logs" } }
}

medium

A. The original message field is deleted

B. A new field named 'msg' extracted from the log message

C. Logs are sent to a file instead of Elasticsearch

D. The timestamp field is removed

Solution

1. Step 1: Analyze the filter section

   The grok filter extracts parts of the log message into fields: timestamp, level, and msg.

2. Step 2: Determine output effect

   The output sends logs to Elasticsearch index 'app-logs' with the new fields added, including 'msg'.

3. Final Answer:

   A new field named 'msg' extracted from the log message -> Option B

4. Quick Check:

   Grok adds 'msg' field from message [OK]

Hint: Grok filter extracts fields like 'msg' from logs [OK]

Common Mistakes:

• Assuming original message is deleted
• Thinking output sends logs to a file
• Believing timestamp is removed

4. Identify the error in this pipeline configuration snippet:

{
  "input": { "type": "file", "path": "/var/log/app.log" },
  "filter": { "grok": { "match": { "message": "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level}" } } },
  "output": { "elasticsearch": { "index": "app-logs" }
}

medium

A. Input type 'file' is invalid

B. Incorrect grok pattern syntax

C. Output index name cannot contain hyphens

D. Missing closing brace for the output section

Solution

1. Step 1: Check JSON structure

   The output section is missing a closing brace '}' at the end, causing invalid JSON.

2. Step 2: Validate other parts

   The grok pattern syntax is correct, input type 'file' is valid, and index names can have hyphens.

3. Final Answer:

   Missing closing brace for the output section -> Option D

4. Quick Check:

   JSON braces must be balanced [OK]

Hint: Check all braces and commas in JSON config [OK]

Common Mistakes:

• Ignoring missing braces causing syntax errors
• Assuming grok pattern is wrong without checking
• Thinking index names can't have hyphens

5. You want to create a log management pipeline that drops logs with level 'DEBUG' and adds a new field 'environment' with value 'production'. Which filter configuration achieves this?

hard

A. { "drop": { "if": "[level] == 'DEBUG'" }, "mutate": { "add_field": { "environment": "production" } } }

B. { "if": "[level] == 'DEBUG'", "drop": {}, "add_field": { "environment": "production" } }

C. { "mutate": { "drop": "[level] == 'DEBUG'", "add_field": { "environment": "production" } } }

D. { "filter": { "drop": { "condition": "level == 'DEBUG'" }, "add_field": { "environment": "production" } } }

Solution

1. Step 1: Understand filter syntax for dropping logs

   The 'drop' filter uses an 'if' condition to remove logs matching criteria.

2. Step 2: Add a new field using 'mutate' filter

   The 'mutate' filter's 'add_field' adds new fields to the log event.

3. Step 3: Combine drop and mutate correctly

   { "drop": { "if": "[level] == 'DEBUG'" }, "mutate": { "add_field": { "environment": "production" } } } correctly uses 'drop' with 'if' and 'mutate' with 'add_field' in the right structure.

4. Final Answer:

   { "drop": { "if": "[level] == 'DEBUG'" }, "mutate": { "add_field": { "environment": "production" } } } -> Option A

5. Quick Check:

   Drop with if + mutate add_field = { "drop": { "if": "[level] == 'DEBUG'" }, "mutate": { "add_field": { "environment": "production" } } } [OK]

Hint: Use 'drop' with 'if' and 'mutate' to add fields [OK]

Common Mistakes:

• Placing 'drop' inside 'mutate' incorrectly
• Using wrong syntax for conditions
• Trying to add fields inside 'drop' filter