Beats (Filebeat, Metricbeat) in Elasticsearch - Mini Project: Build & Apply

Beats Data Collection Setup with Filebeat and Metricbeat

📖 Scenario: You are setting up data collection for your servers using Elastic Beats. Filebeat will collect log files, and Metricbeat will collect system metrics. This helps you monitor your infrastructure in real-time.

🎯 Goal: Configure Filebeat and Metricbeat with basic settings, enable modules, and output data to Elasticsearch. You will write configuration snippets step-by-step to build a working Beats setup.

📋 What You'll Learn

Create a basic Filebeat configuration with input paths

Create a basic Metricbeat configuration with system module enabled

Configure Elasticsearch output for both Filebeat and Metricbeat

Print the final combined configuration snippets

💡 Why This Matters

🌍 Real World

Elastic Beats are used in real companies to collect logs and metrics from servers and applications, helping teams monitor system health and troubleshoot issues.

💼 Career

Knowing how to configure Beats is important for roles in DevOps, Site Reliability Engineering, and IT monitoring, where managing infrastructure data is key.

1

Create Filebeat input configuration

Create a variable called filebeat_config that contains a YAML string with a filebeat.inputs section. It should have one input of type log with paths set to ["/var/log/syslog", "/var/log/auth.log"].

Elasticsearch

# Your code here

Need a hint?

Use triple quotes to create a multi-line string. Follow YAML indentation carefully.

2

Create Metricbeat system module configuration

Create a variable called metricbeat_config that contains a YAML string with metricbeat.modules section. It should enable the system module with metricsets cpu and memory.

Elasticsearch

filebeat_config = """
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/syslog
      - /var/log/auth.log
"""
# Your code here

Need a hint?

Remember to keep the YAML indentation consistent and use triple quotes for multi-line strings.

3

Add Elasticsearch output configuration

Add a variable called output_config containing a YAML string that configures output.elasticsearch with hosts set to ["http://localhost:9200"]. Then combine filebeat_config, metricbeat_config, and output_config into a new variable called combined_config by joining them with two newlines.

Elasticsearch

filebeat_config = """
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/syslog
      - /var/log/auth.log
"""

metricbeat_config = """
metricbeat.modules:
  - module: system
    metricsets:
      - cpu
      - memory
    enabled: true
"""

# Your code here

Need a hint?

Use string concatenation with newlines to combine the configurations.

4

Print the combined Beats configuration

Write a print statement to display the combined_config variable.

Elasticsearch

filebeat_config = """
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/syslog
      - /var/log/auth.log
"""

metricbeat_config = """
metricbeat.modules:
  - module: system
    metricsets:
      - cpu
      - memory
    enabled: true
"""

output_config = """
output.elasticsearch:
  hosts: ["http://localhost:9200"]
"""

combined_config = filebeat_config + "\n\n" + metricbeat_config + "\n\n" + output_config

# Your code here

Need a hint?

Use print(combined_config) to show the full configuration.