Log management pipeline in Elasticsearch

Introduction

A log management pipeline helps collect, process, and store logs so you can easily find and understand what happened in your systems.

You want to gather logs from many servers in one place.

You need to filter or modify logs before saving them.

You want to search logs quickly to find errors or issues.

You want to create alerts based on certain log messages.

You want to keep logs organized and easy to analyze.

Syntax

Elasticsearch

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMMONAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "weblogs-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

This example uses Logstash syntax to define a pipeline.

It has three parts: input (where logs come from), filter (how logs are processed), and output (where logs go).

Examples

This input reads logs from a file starting at the beginning.

Elasticsearch

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}

This filter extracts fields from Apache logs using a pattern.

Elasticsearch

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

This output sends processed logs to Elasticsearch with a daily index.

Elasticsearch

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "app-logs-%{+YYYY.MM.dd}"
  }
}

Sample Program

This simple pipeline reads logs from the keyboard, extracts a log level and message, then prints the structured data.

Elasticsearch

input {
  stdin {}
}

filter {
  grok {
    match => { "message" => "%{WORD:level}: %{GREEDYDATA:msg}" }
  }
}

output {
  stdout { codec => rubydebug }
}

Important Notes

Use grok filters to parse unstructured log text into fields.

Always test your pipeline with sample logs to check parsing.

Keep your pipeline simple and add complexity step-by-step.

Summary

A log management pipeline collects, processes, and stores logs.

It has input, filter, and output sections.

Use it to organize logs for easy searching and alerting.