Centralized logging (EFK stack) in Kubernetes - Step-by-Step Execution

Process Flow - Centralized logging (EFK stack)

Logs generated by pods

↓

Fluentd collects logs

↓

Fluentd forwards logs to Elasticsearch

↓

Elasticsearch stores and indexes logs

↓

Kibana queries Elasticsearch

↓

User views logs in Kibana dashboard

Logs flow from pods to Fluentd, then to Elasticsearch for storage, and finally Kibana displays them.

Execution Sample

Kubernetes

kubectl apply -f fluentd-config.yaml
kubectl apply -f elasticsearch-deployment.yaml
kubectl apply -f kibana-deployment.yaml

Deploy Fluentd, Elasticsearch, and Kibana components to collect, store, and visualize logs.

Process Table

Step	Action	Component	Log Data State	System State Change
1	Pod generates logs	Pod	Raw logs created	Logs available in pod filesystem
2	Fluentd collects logs	Fluentd	Reads raw logs	Logs buffered in Fluentd
3	Fluentd forwards logs	Fluentd	Logs sent	Logs sent to Elasticsearch service
4	Elasticsearch stores logs	Elasticsearch	Logs indexed	Logs stored and searchable
5	Kibana queries logs	Kibana	Query sent	Logs retrieved for display
6	User views logs	Kibana UI	Logs displayed	User sees logs in dashboard
7	End	-	-	Logging cycle complete

💡 All logs collected, stored, and displayed successfully.

Status Tracker

Variable	Start	After Step 2	After Step 4	Final
Pod Logs	None	Raw logs created	Raw logs stored in Elasticsearch	Available for Kibana queries
Fluentd Buffer	Empty	Logs buffered	Empty after forwarding	Empty after forwarding
Elasticsearch Index	Empty	Empty	Logs indexed	Logs searchable
Kibana Display	Empty	Empty	Empty	Logs displayed

Key Moments - 3 Insights

Why does Fluentd buffer logs before sending to Elasticsearch?

Fluentd buffers logs to handle bursts and network delays, ensuring no logs are lost before forwarding, as shown in step 2 and 3 of the execution table.

What happens if Elasticsearch is down when Fluentd tries to send logs?

Fluentd will keep logs in its buffer and retry sending later, preventing log loss. This is implied by Fluentd's buffering behavior in steps 2 and 3.

How does Kibana get the logs to display?

Kibana queries Elasticsearch for indexed logs (step 5), then displays them in the dashboard (step 6), as shown in the execution table.

Visual Quiz - 3 Questions

Look at the execution table, at which step does Fluentd send logs to Elasticsearch?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Check the 'Action' and 'Component' columns for Fluentd forwarding logs.

According to the variable tracker, what is the state of Fluentd Buffer after step 4?

A Logs buffered

B Logs indexed

C Empty after forwarding

D Raw logs created

💡 Hint

Look at the 'Fluentd Buffer' row under 'After Step 4' column.

If Elasticsearch fails to store logs, which component's state will show logs still buffered?

A Fluentd Buffer

B Pod Logs

C Kibana Display

D Elasticsearch Index

💡 Hint

Refer to key moment about Fluentd buffering and retrying logs.

Concept Snapshot

Centralized logging with EFK stack:
- Fluentd collects logs from pods
- Fluentd buffers and forwards logs to Elasticsearch
- Elasticsearch stores and indexes logs
- Kibana queries Elasticsearch and displays logs
- Ensures logs are centralized and easy to search

Full Transcript

In centralized logging using the EFK stack, logs are first generated by pods running in Kubernetes. Fluentd collects these logs from the pods and buffers them to handle network delays or bursts. Then Fluentd forwards the buffered logs to Elasticsearch, which stores and indexes them for fast searching. Kibana queries Elasticsearch to retrieve logs and displays them in a user-friendly dashboard. This flow ensures all logs are collected in one place, stored safely, and easy to view. If Elasticsearch is down, Fluentd keeps logs buffered and retries sending later, preventing data loss.