Centralized logging (EFK stack) in Kubernetes - Step-by-Step Execution

Process Flow - Centralized logging (EFK stack)

Logs generated by pods

↓

Fluentd collects logs

↓

Fluentd forwards logs to Elasticsearch

↓

Elasticsearch stores and indexes logs

↓

Kibana queries Elasticsearch

↓

User views logs in Kibana dashboard

Logs flow from pods to Fluentd, then to Elasticsearch for storage, and finally Kibana displays them.

Execution Sample

Kubernetes

kubectl apply -f fluentd-config.yaml
kubectl apply -f elasticsearch-deployment.yaml
kubectl apply -f kibana-deployment.yaml

Deploy Fluentd, Elasticsearch, and Kibana components to collect, store, and visualize logs.

Process Table

Step	Action	Component	Log Data State	System State Change
1	Pod generates logs	Pod	Raw logs created	Logs available in pod filesystem
2	Fluentd collects logs	Fluentd	Reads raw logs	Logs buffered in Fluentd
3	Fluentd forwards logs	Fluentd	Logs sent	Logs sent to Elasticsearch service
4	Elasticsearch stores logs	Elasticsearch	Logs indexed	Logs stored and searchable
5	Kibana queries logs	Kibana	Query sent	Logs retrieved for display
6	User views logs	Kibana UI	Logs displayed	User sees logs in dashboard
7	End	-	-	Logging cycle complete

💡 All logs collected, stored, and displayed successfully.

Status Tracker

Variable	Start	After Step 2	After Step 4	Final
Pod Logs	None	Raw logs created	Raw logs stored in Elasticsearch	Available for Kibana queries
Fluentd Buffer	Empty	Logs buffered	Empty after forwarding	Empty after forwarding
Elasticsearch Index	Empty	Empty	Logs indexed	Logs searchable
Kibana Display	Empty	Empty	Empty	Logs displayed

Key Moments - 3 Insights

Why does Fluentd buffer logs before sending to Elasticsearch?

Fluentd buffers logs to handle bursts and network delays, ensuring no logs are lost before forwarding, as shown in step 2 and 3 of the execution table.

What happens if Elasticsearch is down when Fluentd tries to send logs?

Fluentd will keep logs in its buffer and retry sending later, preventing log loss. This is implied by Fluentd's buffering behavior in steps 2 and 3.

How does Kibana get the logs to display?

Kibana queries Elasticsearch for indexed logs (step 5), then displays them in the dashboard (step 6), as shown in the execution table.

Visual Quiz - 3 Questions

Look at the execution table, at which step does Fluentd send logs to Elasticsearch?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Check the 'Action' and 'Component' columns for Fluentd forwarding logs.

According to the variable tracker, what is the state of Fluentd Buffer after step 4?

A Logs buffered

B Logs indexed

C Empty after forwarding

D Raw logs created

💡 Hint

Look at the 'Fluentd Buffer' row under 'After Step 4' column.

If Elasticsearch fails to store logs, which component's state will show logs still buffered?

A Fluentd Buffer

B Pod Logs

C Kibana Display

D Elasticsearch Index

💡 Hint

Refer to key moment about Fluentd buffering and retrying logs.

Concept Snapshot

Centralized logging with EFK stack:
- Fluentd collects logs from pods
- Fluentd buffers and forwards logs to Elasticsearch
- Elasticsearch stores and indexes logs
- Kibana queries Elasticsearch and displays logs
- Ensures logs are centralized and easy to search

Full Transcript

In centralized logging using the EFK stack, logs are first generated by pods running in Kubernetes. Fluentd collects these logs from the pods and buffers them to handle network delays or bursts. Then Fluentd forwards the buffered logs to Elasticsearch, which stores and indexes them for fast searching. Kibana queries Elasticsearch to retrieve logs and displays them in a user-friendly dashboard. This flow ensures all logs are collected in one place, stored safely, and easy to view. If Elasticsearch is down, Fluentd keeps logs buffered and retries sending later, preventing data loss.

Practice

1. What is the main purpose of the EFK stack in Kubernetes?

easy

A. To collect, store, and visualize logs from all pods centrally

B. To manage Kubernetes cluster networking

C. To automate deployment of applications

D. To monitor CPU and memory usage only

Solution

1. Step 1: Understand EFK components

   The EFK stack consists of Fluentd (log collector), Elasticsearch (log storage), and Kibana (log viewer).

2. Step 2: Identify the main goal

   Its main goal is to centralize logs from all Kubernetes pods for easier troubleshooting and monitoring.

3. Final Answer:

   To collect, store, and visualize logs from all pods centrally -> Option A

4. Quick Check:

   EFK = Centralized logging [OK]

Hint: EFK means Fluentd, Elasticsearch, Kibana for logs [OK]

Common Mistakes:

• Confusing EFK with monitoring CPU/memory
• Thinking EFK manages networking
• Assuming EFK automates deployments

2. Which Kubernetes resource is typically used to deploy Fluentd as a log collector in the EFK stack?

easy

A. ServiceAccount

B. DaemonSet

C. Deployment

D. ConfigMap

Solution

1. Step 1: Understand Fluentd deployment needs

   Fluentd must run on every node to collect logs from all pods on that node.

2. Step 2: Choose correct Kubernetes resource

   DaemonSet ensures one pod per node, perfect for log collectors like Fluentd.

3. Final Answer:

   DaemonSet -> Option B

4. Quick Check:

   Fluentd runs as DaemonSet [OK]

Hint: DaemonSet runs pods on all nodes [OK]

Common Mistakes:

• Using Deployment which may not run on all nodes
• Confusing ConfigMap with deployment type
• Thinking ServiceAccount deploys pods

3. Given this Fluentd config snippet in Kubernetes:

match ** {
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
}

What is the main effect of this configuration?

medium

A. Fluentd sends all logs to Elasticsearch service at port 9200

B. Fluentd collects logs only from pods named elasticsearch

C. Fluentd stores logs locally on each node

D. Fluentd forwards logs to Kibana directly

Solution

1. Step 1: Analyze Fluentd match directive

   The match ** means all logs are matched and processed by this output plugin.

2. Step 2: Understand output plugin settings

   @type elasticsearch with host and port means logs are sent to Elasticsearch service at that address.

3. Final Answer:

   Fluentd sends all logs to Elasticsearch service at port 9200 -> Option A

4. Quick Check:

   match ** + elasticsearch output = send all logs to ES [OK]

Hint: match ** means all logs sent to Elasticsearch [OK]

Common Mistakes:

• Thinking logs go directly to Kibana
• Assuming logs are stored locally
• Confusing match pattern with pod names

4. You deployed the EFK stack but Kibana shows no logs. Which of these is the most likely cause?

medium

A. Kibana is configured to use wrong Elasticsearch URL

B. Elasticsearch service port is set to 8080 instead of 9200

C. All of the above

D. Fluentd DaemonSet is not running on nodes

Solution

1. Step 1: Check Fluentd status

   If Fluentd pods are not running, logs won't be collected or sent.

2. Step 2: Verify Elasticsearch connectivity

   Wrong port on Elasticsearch service means Fluentd can't send logs properly.

3. Step 3: Confirm Kibana configuration

   If Kibana points to wrong Elasticsearch URL, it can't display logs.

4. Final Answer:

   All of the above -> Option C

5. Quick Check:

   Any broken link in EFK stops logs [OK]

Hint: Check Fluentd, Elasticsearch port, Kibana URL [OK]

Common Mistakes:

• Checking only one component
• Ignoring service port mismatch
• Assuming Kibana auto-fixes URLs

5. You want to filter out logs from Kubernetes system namespaces (like kube-system and default) before sending to Elasticsearch in Fluentd. Which configuration snippet achieves this?

hard

A. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^kube-system$

}

B. filter ** {
@type record_transformer
remove_keys kubernetes.namespace_name
}

C. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^kube-system$

}

D. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^(kube-system|default)$

}

Solution

1. Step 1: Understand filtering with Fluentd grep plugin

   The grep plugin can exclude logs matching certain patterns using blocks.

2. Step 2: Identify namespaces to exclude

   We want to exclude system namespaces like kube-system and default, so pattern must match both.

3. Step 3: Compare options

   filter ** {
     @type grep
     
       key kubernetes.namespace_name
       pattern ^(kube-system|default)$
     
   }

   excludes both kube-system and default namespaces correctly; others exclude only one or do wrong action.

4. Final Answer:

   filter ** { @type grep key kubernetes.namespace_name pattern ^(kube-system|default)$ } -> Option D

5. Quick Check:

   Exclude system namespaces with grep exclude pattern [OK]

Hint: Use grep exclude with regex for system namespaces [OK]

Common Mistakes:

• Excluding only one namespace
• Using include instead of exclude
• Removing keys instead of filtering logs