Beats (Filebeat, Metricbeat) in Elasticsearch

Introduction

Beats are small programs that collect data from your computers and send it to Elasticsearch. Filebeat collects log files, and Metricbeat collects system metrics like CPU and memory.

You want to watch log files from your web server to find errors.

You need to track how much CPU and memory your servers use over time.

You want to send data from many computers to one place for easy searching.

You want to monitor your applications and systems without installing heavy software.

You want to quickly set up data collection for your Elasticsearch dashboards.

Syntax

Elasticsearch

filebeat setup
metricbeat setup
filebeat -e
metricbeat -e

filebeat setup prepares dashboards and index templates in Elasticsearch.

metricbeat setup does the same for Metricbeat data.

Examples

This sets up Filebeat and then starts it in the foreground to send logs.

Elasticsearch

filebeat setup
filebeat -e

This sets up Metricbeat and then starts it to send system metrics.

Elasticsearch

metricbeat setup
metricbeat -e

Example Filebeat config to read all logs in /var/log and send to Elasticsearch on localhost.

Elasticsearch

filebeat.yml
# Configure paths to log files
filebeat.inputs:
- type: log
  paths:
    - /var/log/*.log
output.elasticsearch:
  hosts: ["localhost:9200"]

Example Metricbeat config to collect CPU and memory metrics every 10 seconds.

Elasticsearch

metricbeat.yml
metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - memory
  period: 10s
output.elasticsearch:
  hosts: ["localhost:9200"]

Sample Program

This example shows a basic Filebeat setup to send system logs to Elasticsearch. The comments explain how to run it and what output to expect.

Elasticsearch

# This is a simple Filebeat config example
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/syslog
output.elasticsearch:
  hosts: ["localhost:9200"]

# To run Filebeat:
# 1. Save this as filebeat.yml
# 2. Run 'filebeat setup' to prepare Elasticsearch
# 3. Run 'filebeat -e' to start sending logs

# Output example when running 'filebeat -e':
# 2024-06-01T12:00:00.000Z INFO  instance/beat.go:123 Setup Beat: filebeat; Version: 8.0.0
# 2024-06-01T12:00:01.000Z INFO  crawler/crawler.go:75 Loading and starting input: log
# 2024-06-01T12:00:02.000Z INFO  pipeline/output.go:95 Connecting to backoff(elasticsearch(http://localhost:9200))
# 2024-06-01T12:00:03.000Z INFO  pipeline/output.go:105 Connection to backoff(elasticsearch(http://localhost:9200)) established
# 2024-06-01T12:00:04.000Z INFO  crawler/crawler.go:95 Input started: log

Important Notes

Beats are lightweight and easy to install on many machines.

Always run setup commands before starting Beats to prepare Elasticsearch.

Use the -e flag to run Beats in the foreground and see logs directly.

Summary

Beats collect data like logs and metrics from your computers.

Filebeat is for logs; Metricbeat is for system metrics.

They send data to Elasticsearch for easy searching and monitoring.