Logstash overview in Elasticsearch - Mini Project: Build & Apply

Logstash Overview

📖 Scenario: You are working as a data engineer who needs to collect and process logs from different sources to analyze them in Elasticsearch. You will use Logstash to set up a simple pipeline that reads data, processes it, and outputs it.

🎯 Goal: Build a basic Logstash configuration that reads input from a file, applies a filter to parse the data, and outputs the result to the console.

📋 What You'll Learn

Create an input section that reads from a file called logs.txt

Add a filter section that uses the grok plugin to parse log lines

Create an output section that prints the processed logs to the console

Use exact plugin names and configuration keys as specified

💡 Why This Matters

🌍 Real World

Logstash is used in real companies to collect and process logs from servers, applications, and devices to monitor and analyze system behavior.

💼 Career

Understanding Logstash configuration is important for roles like DevOps engineer, data engineer, and system administrator who manage log data pipelines.

1

Create the input section

Write the Logstash configuration input section to read from a file called logs.txt. Use the file input plugin with the path set to "logs.txt".

Elasticsearch

# Your code here
input {
  # Add file input plugin here
}

Need a hint?

The input section starts with input { and uses the file plugin with path => "logs.txt".

2

Add the filter section with grok plugin

Add a filter section that uses the grok plugin to parse log lines. Use the pattern "%{COMMONAPACHELOG}" inside the grok match.

Elasticsearch

input {
  file {
    path => "logs.txt"
  }
}

# Add filter section with grok plugin here

Need a hint?

The filter section uses grok with match => { "message" => "%{COMMONAPACHELOG}" } to parse Apache logs.

3

Create the output section to print to console

Add an output section that uses the stdout plugin with codec => rubydebug to print the processed logs to the console.

Elasticsearch

input {
  file {
    path => "logs.txt"
  }
}

filter {
  grok {
    match => { "message" => "%{COMMONAPACHELOG}" }
  }
}

# Add output section with stdout plugin here

Need a hint?

The output section uses stdout with codec => rubydebug to print logs nicely.

4

Run and display the Logstash configuration

Verify the entire Logstash configuration is correct.

Elasticsearch

# Logstash configuration complete - verify all sections

Need a hint?

Ensure your configuration includes input, filter with grok, and output sections as specified in previous steps.