Beats (Filebeat, Metricbeat) in Elasticsearch - Step-by-Step Execution

Concept Flow - Beats (Filebeat, Metricbeat)

Start Beats

↓

Collect Data

↓

Filebeat

↓

Read Logs

↓

Send Data to Elasticsearch

↓

Data Indexed & Searchable

Beats start by collecting data: Filebeat reads log files, Metricbeat collects system metrics, then both send data to Elasticsearch for indexing.

Execution Sample

Elasticsearch

filebeat
metricbeat

Starts Filebeat to read logs and Metricbeat to collect metrics, sending data to Elasticsearch.

Execution Table

Step	Beat	Action	Data Collected	Data Sent To
1	Filebeat	Starts reading log files	Log lines from /var/log/syslog	Elasticsearch
2	Metricbeat	Starts collecting system metrics	CPU, Memory, Disk usage	Elasticsearch
3	Filebeat	Sends collected logs	Batch of log lines	Elasticsearch
4	Metricbeat	Sends collected metrics	Batch of system metrics	Elasticsearch
5	Elasticsearch	Indexes received data	Logs and metrics indexed	Ready for search

💡 Both Beats continue running, sending data continuously until stopped.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Filebeat Logs	empty	logs read	logs read	logs sent	logs sent	continuous reading
Metricbeat Metrics	empty	empty	metrics collected	metrics collected	metrics sent	continuous collecting
Elasticsearch Index	empty	empty	empty	logs indexed	logs and metrics indexed	data searchable

Key Moments - 3 Insights

Why does Filebeat keep running after sending logs?

Filebeat continuously monitors log files for new entries, so it keeps running to send new logs as they appear (see execution_table rows 1,3).

How does Metricbeat know what metrics to collect?

Metricbeat uses predefined modules to collect specific system metrics like CPU and memory (see execution_table row 2).

What happens if Elasticsearch is down when Beats send data?

Beats will retry sending data and buffer it locally until Elasticsearch is available again, ensuring no data loss.

Visual Quiz - 3 Questions

Look at the execution table, what data does Filebeat send at step 3?

A Empty data

B CPU and Memory metrics

C Batch of log lines

D Configuration files

💡 Hint

Check the 'Data Sent To' and 'Data Collected' columns at step 3 in the execution_table.

At which step does Metricbeat start sending data to Elasticsearch?

A Step 4

B Step 2

C Step 1

D Step 5

💡 Hint

Look for Metricbeat's 'Sends collected metrics' action in the execution_table.

If Filebeat stopped reading logs, which variable in variable_tracker would stop changing?

A Metricbeat Metrics

B Filebeat Logs

C Elasticsearch Index

D None

💡 Hint

Check the 'Filebeat Logs' row in variable_tracker to see what changes when logs are read.

Concept Snapshot

Beats are lightweight data shippers.
Filebeat reads and ships log files.
Metricbeat collects and ships system metrics.
Both send data to Elasticsearch.
Elasticsearch indexes data for search.
Beats run continuously to send new data.

Full Transcript

Beats like Filebeat and Metricbeat are small programs that collect data and send it to Elasticsearch. Filebeat reads log files continuously, while Metricbeat collects system metrics like CPU and memory usage. Both send their data in batches to Elasticsearch, which indexes it so you can search and analyze it. Beats keep running to send new data as it appears. If Elasticsearch is down, Beats will retry sending data to avoid loss.