Infrastructure monitoring in Elasticsearch - Step-by-Step Execution

Concept Flow - Infrastructure monitoring

Start Monitoring Setup

↓

Install Metricbeat

↓

Configure Metricbeat to collect system metrics

↓

Start Metricbeat Service

↓

Metricbeat sends data to Elasticsearch

↓

Elasticsearch stores metrics

↓

Use Kibana to visualize metrics

↓

Analyze system health and performance

↓

Adjust monitoring or alerting as needed

↓

End

This flow shows how Metricbeat collects system metrics, sends them to Elasticsearch, and how Kibana visualizes them for monitoring infrastructure health.

Execution Sample

Elasticsearch

metricbeat setup
sudo systemctl start metricbeat
curl -X GET "localhost:9200/_cat/indices?v"
curl -X GET "localhost:9200/metricbeat-*/_search?size=1"

This sequence sets up Metricbeat, starts it, then queries Elasticsearch to check metric data indices and sample documents.

Execution Table

Step	Action	Command/Query	Result/Output
1	Setup Metricbeat dashboards and index templates	metricbeat setup	Loaded dashboards, index templates, and ILM policies successfully
2	Start Metricbeat service to begin data collection	sudo systemctl start metricbeat	Metricbeat service started and running
3	Check Elasticsearch indices for Metricbeat data	curl -X GET "localhost:9200/_cat/indices?v"	List shows metricbeat-7.x.x-YYYY.MM.DD index with document count > 0
4	Query a sample document from Metricbeat index	curl -X GET "localhost:9200/metricbeat-*/_search?size=1"	Returns JSON with system metrics like CPU, memory usage
5	End		Monitoring data is flowing and stored in Elasticsearch

💡 Metricbeat is running and sending system metrics to Elasticsearch indices, ready for visualization.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Metricbeat Service Status	Stopped	Stopped	Running	Running	Running	Running
Elasticsearch Indices	None	Templates loaded	Templates loaded	metricbeat-7.x.x-YYYY.MM.DD created	metricbeat index has documents	metricbeat index has documents
Metric Data Documents	0	0	0	>0	>0	>0

Key Moments - 3 Insights

Why do we run 'metricbeat setup' before starting the service?

The 'metricbeat setup' command loads dashboards and index templates into Elasticsearch, preparing it to store and visualize the incoming data correctly, as shown in step 1 of the execution table.

How do we know Metricbeat is successfully sending data to Elasticsearch?

Step 3 and 4 show Elasticsearch indices exist and contain documents with system metrics, confirming data flow from Metricbeat.

What does the 'curl' command in step 4 return?

It returns a JSON document with system metrics like CPU and memory usage, demonstrating that monitoring data is stored and queryable.

Visual Quiz - 3 Questions

Look at the execution table, what is the Metricbeat service status after step 2?

A Stopped

B Running

C Failed

D Starting

💡 Hint

Check the 'Metricbeat Service Status' variable in variable_tracker after step 2.

At which step do Elasticsearch indices for Metricbeat data first appear?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the 'Elasticsearch Indices' variable in variable_tracker and the execution_table rows.

If Metricbeat was not sending data, what would the output of step 4 likely show?

A Empty hits array or no documents

B Error connecting to Elasticsearch

C JSON with system metrics

D List of dashboards

💡 Hint

Step 4 shows a sample document; if no data, the hits array would be empty.

Concept Snapshot

Infrastructure Monitoring with Metricbeat and Elasticsearch:
- Run 'metricbeat setup' to load dashboards and templates.
- Start Metricbeat service to collect system metrics.
- Metricbeat sends data to Elasticsearch indices.
- Use Kibana to visualize and analyze metrics.
- Monitor CPU, memory, disk, and network health.
- Adjust alerts based on metric trends.

Full Transcript

Infrastructure monitoring with Elasticsearch involves installing Metricbeat on your system to collect metrics like CPU and memory usage. First, you run 'metricbeat setup' to prepare Elasticsearch with dashboards and templates. Then, you start the Metricbeat service to begin sending data. You can verify data flow by querying Elasticsearch indices and checking for metric documents. This setup allows you to visualize system health in Kibana and respond to performance issues quickly.