Beats (Filebeat, Metricbeat) in Elasticsearch - Step-by-Step Execution

Concept Flow - Beats (Filebeat, Metricbeat)

Start Beats

↓

Collect Data

↓

Filebeat

↓

Read Logs

↓

Send Data to Elasticsearch

↓

Data Indexed & Searchable

Beats start by collecting data: Filebeat reads log files, Metricbeat collects system metrics, then both send data to Elasticsearch for indexing.

Execution Sample

Elasticsearch

filebeat
metricbeat

Starts Filebeat to read logs and Metricbeat to collect metrics, sending data to Elasticsearch.

Execution Table

Step	Beat	Action	Data Collected	Data Sent To
1	Filebeat	Starts reading log files	Log lines from /var/log/syslog	Elasticsearch
2	Metricbeat	Starts collecting system metrics	CPU, Memory, Disk usage	Elasticsearch
3	Filebeat	Sends collected logs	Batch of log lines	Elasticsearch
4	Metricbeat	Sends collected metrics	Batch of system metrics	Elasticsearch
5	Elasticsearch	Indexes received data	Logs and metrics indexed	Ready for search

💡 Both Beats continue running, sending data continuously until stopped.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Filebeat Logs	empty	logs read	logs read	logs sent	logs sent	continuous reading
Metricbeat Metrics	empty	empty	metrics collected	metrics collected	metrics sent	continuous collecting
Elasticsearch Index	empty	empty	empty	logs indexed	logs and metrics indexed	data searchable

Key Moments - 3 Insights

Why does Filebeat keep running after sending logs?

Filebeat continuously monitors log files for new entries, so it keeps running to send new logs as they appear (see execution_table rows 1,3).

How does Metricbeat know what metrics to collect?

Metricbeat uses predefined modules to collect specific system metrics like CPU and memory (see execution_table row 2).

What happens if Elasticsearch is down when Beats send data?

Beats will retry sending data and buffer it locally until Elasticsearch is available again, ensuring no data loss.

Visual Quiz - 3 Questions

Look at the execution table, what data does Filebeat send at step 3?

A Empty data

B CPU and Memory metrics

C Batch of log lines

D Configuration files

💡 Hint

Check the 'Data Sent To' and 'Data Collected' columns at step 3 in the execution_table.

At which step does Metricbeat start sending data to Elasticsearch?

A Step 4

B Step 2

C Step 1

D Step 5

💡 Hint

Look for Metricbeat's 'Sends collected metrics' action in the execution_table.

If Filebeat stopped reading logs, which variable in variable_tracker would stop changing?

A Metricbeat Metrics

B Filebeat Logs

C Elasticsearch Index

D None

💡 Hint

Check the 'Filebeat Logs' row in variable_tracker to see what changes when logs are read.

Concept Snapshot

Beats are lightweight data shippers.
Filebeat reads and ships log files.
Metricbeat collects and ships system metrics.
Both send data to Elasticsearch.
Elasticsearch indexes data for search.
Beats run continuously to send new data.

Full Transcript

Beats like Filebeat and Metricbeat are small programs that collect data and send it to Elasticsearch. Filebeat reads log files continuously, while Metricbeat collects system metrics like CPU and memory usage. Both send their data in batches to Elasticsearch, which indexes it so you can search and analyze it. Beats keep running to send new data as it appears. If Elasticsearch is down, Beats will retry sending data to avoid loss.

Practice

1. What is the primary purpose of Filebeat in the Elastic Stack?

easy

A. To visualize data in dashboards

B. To collect system metrics like CPU and memory usage

C. To collect and ship log files from servers

D. To store data in Elasticsearch

Solution

1. Step 1: Understand Filebeat's role

   Filebeat is designed to collect log files from servers and forward them.

2. Step 2: Differentiate from Metricbeat

   Metricbeat collects system metrics, not logs.

3. Final Answer:

   To collect and ship log files from servers -> Option C

4. Quick Check:

   Filebeat = log collection [OK]

Hint: Filebeat = logs, Metricbeat = metrics [OK]

Common Mistakes:

• Confusing Filebeat with Metricbeat
• Thinking Filebeat visualizes data
• Assuming Filebeat stores data

2. Which of the following is the correct syntax to start Filebeat on a Linux system?

easy

A. sudo systemctl start filebeat

B. filebeat run start

C. start filebeat service

D. service filebeat enable

Solution

1. Step 1: Identify standard Linux service start command

   On modern Linux systems, services are started with 'sudo systemctl start servicename'.

2. Step 2: Match with Filebeat service

   Filebeat service is named 'filebeat', so the command is 'sudo systemctl start filebeat'.

3. Final Answer:

   sudo systemctl start filebeat -> Option A

4. Quick Check:

   systemctl start filebeat = correct syntax [OK]

Hint: Use systemctl to manage services on Linux [OK]

Common Mistakes:

• Using incorrect command order
• Confusing 'start' with 'enable'
• Using non-standard service commands

3. Given this Filebeat configuration snippet:

filebeat.inputs:
- type: log
  paths:
    - /var/log/syslog
output.elasticsearch:
  hosts: ["localhost:9200"]

What will Filebeat do when started?

medium

A. Store logs locally without sending

B. Collect system metrics and send to Elasticsearch

C. Send logs to a remote Logstash server

D. Collect logs from /var/log/syslog and send to Elasticsearch at localhost:9200

Solution

1. Step 1: Analyze input configuration

   The input type is 'log' and path is '/var/log/syslog', so Filebeat collects logs from this file.

2. Step 2: Analyze output configuration

   Output is set to Elasticsearch at 'localhost:9200', so logs are sent there.

3. Final Answer:

   Collect logs from /var/log/syslog and send to Elasticsearch at localhost:9200 -> Option D

4. Quick Check:

   Input logs + output Elasticsearch = Collect logs from /var/log/syslog and send to Elasticsearch at localhost:9200 [OK]

Hint: Check input type and output host to predict behavior [OK]

Common Mistakes:

• Confusing logs with metrics
• Assuming output is Logstash
• Ignoring the paths setting

4. You configured Metricbeat to collect CPU metrics but see no data in Elasticsearch. Which of these is a likely cause?

medium

A. Metricbeat service is not running

B. Filebeat is collecting logs instead of metrics

C. Elasticsearch is not installed

D. Metricbeat configuration file is empty

Solution

1. Step 1: Check Metricbeat service status

   If Metricbeat is not running, it cannot collect or send metrics.

2. Step 2: Consider other options

   Filebeat collecting logs is unrelated; Elasticsearch not installed would cause errors; empty config would cause no collection.

3. Final Answer:

   Metricbeat service is not running -> Option A

4. Quick Check:

   Service running = data sent [OK]

Hint: Always verify Metricbeat service is active [OK]

Common Mistakes:

• Mixing Filebeat and Metricbeat roles
• Ignoring service status
• Assuming Elasticsearch absence without error

5. You want to monitor both system logs and CPU metrics on a server using Beats. Which setup is best?

hard

A. Run Filebeat and Metricbeat but send data to different Elasticsearch clusters

B. Run Filebeat for logs and Metricbeat for CPU metrics, both sending to Elasticsearch

C. Run Metricbeat for logs and Filebeat for metrics

D. Run only Filebeat configured for logs and metrics

Solution

1. Step 1: Identify Beats roles

   Filebeat collects logs; Metricbeat collects system metrics like CPU.

2. Step 2: Choose correct combination

   Running both Beats with their proper roles and sending data to the same Elasticsearch cluster is best for unified monitoring.

3. Final Answer:

   Run Filebeat for logs and Metricbeat for CPU metrics, both sending to Elasticsearch -> Option B

4. Quick Check:

   Filebeat = logs + Metricbeat = metrics [OK]

Hint: Use Filebeat for logs and Metricbeat for metrics together [OK]

Common Mistakes:

• Mixing roles of Filebeat and Metricbeat
• Sending data to separate clusters unnecessarily
• Assuming one Beat can do both well