Centralized logging (EFK stack) in Kubernetes - Deep Dive

Overview - Centralized logging (EFK stack)

What is it?

Centralized logging with the EFK stack means collecting all logs from many computers or containers into one place. EFK stands for Elasticsearch, Fluentd, and Kibana. Elasticsearch stores and searches logs, Fluentd gathers and sends logs, and Kibana shows logs in a friendly way. This helps teams see and understand what is happening across their whole system easily.

Why it matters

Without centralized logging, logs are scattered across many machines or containers, making it hard to find problems or understand system behavior. This wastes time and can delay fixing issues. Centralized logging with EFK lets teams quickly search and analyze logs from everywhere, improving reliability and speed of troubleshooting. It also helps with security and compliance by keeping logs safe and organized.

Where it fits

Before learning EFK, you should understand basic Kubernetes concepts like pods and containers, and know what logs are. After mastering EFK, you can explore advanced monitoring tools, alerting systems, and log analysis techniques to improve system health and performance.

Mental Model

Core Idea

Centralized logging with EFK collects logs from many sources, stores them efficiently, and presents them visually to make troubleshooting simple and fast.

Think of it like...

Imagine a big office building where every room has a notebook logging what happens inside. Instead of checking each room's notebook, a helper collects all notes into one big book in the lobby, organizes them, and shows summaries on a screen for everyone to see.

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Kubernetes │     │   Fluentd   │     │ Elasticsearch│
│  Containers │───▶ │ (Log Agent) │───▶ │ (Log Store) │
└─────────────┘     └─────────────┘     └─────────────┘
                                         │
                                         ▼
                                  ┌─────────────┐
                                  │   Kibana    │
                                  │ (Log Viewer)│
                                  └─────────────┘

Build-Up - 7 Steps

1

Foundation

What is logging and why it matters

Concept: Introduce the idea of logs as records of events in software systems.

Logs are like diaries for computers and applications. They record what happened, when, and sometimes why. For example, a web server logs every request it gets. These logs help us understand if things are working or if there are problems.

Result

You understand that logs are essential for knowing what your system is doing and for finding problems.

Knowing that logs are the system’s memory helps you see why collecting and reading them is crucial for managing software.

2

Foundation

Challenges of scattered logs in Kubernetes

Concept: Explain why logs spread across many containers and nodes are hard to manage.

In Kubernetes, many containers run on many machines. Each container creates its own logs. If you want to find a problem, you must look in many places. This is slow and confusing, especially when containers restart or move.

Result

You realize that scattered logs make troubleshooting complex and slow.

Understanding this challenge sets the stage for why centralized logging is needed.

3

Intermediate

Role of Fluentd as log collector

🤔Before reading on: do you think Fluentd stores logs or just moves them? Commit to your answer.

Concept: Fluentd collects logs from containers and sends them to storage but does not store them long-term.

Fluentd runs on each Kubernetes node. It watches container logs and forwards them to Elasticsearch. Fluentd can also filter or modify logs before sending. It acts like a mail carrier, picking up letters (logs) and delivering them to the post office (Elasticsearch).

Result

Logs from all containers start flowing into one place, ready for storage and search.

Knowing Fluentd’s role as a flexible log forwarder helps you understand how logs move reliably in the system.

4

Intermediate

Elasticsearch stores and indexes logs

🤔Before reading on: do you think Elasticsearch stores logs as plain text or indexes them for search? Commit to your answer.

Concept: Elasticsearch stores logs and creates indexes to make searching fast and efficient.

Elasticsearch is a database designed for searching large amounts of text quickly. It stores logs sent by Fluentd and builds indexes so you can find logs by time, error type, or any detail. This makes searching through millions of logs fast.

Result

Logs are stored safely and can be searched instantly by many criteria.

Understanding Elasticsearch’s indexing explains why searching logs is fast even at huge scale.

5

Intermediate

Kibana visualizes and explores logs

Concept: Kibana provides a user-friendly interface to view and analyze logs stored in Elasticsearch.

Kibana connects to Elasticsearch and shows logs in charts, tables, and dashboards. You can filter logs by time or keywords, create alerts, and spot trends. It turns raw logs into clear pictures that help teams understand system health.

Result

Users can easily explore logs visually without writing complex queries.

Knowing Kibana’s role helps you see how logs become actionable insights for teams.

6

Advanced

Deploying EFK stack on Kubernetes

🤔Before reading on: do you think EFK components run inside or outside Kubernetes? Commit to your answer.

Concept: EFK components run as pods inside Kubernetes, integrated with the cluster.

You deploy Fluentd as a DaemonSet so it runs on every node, collecting logs locally. Elasticsearch runs as a StatefulSet with persistent storage to keep logs safe. Kibana runs as a Deployment to provide the web interface. Together, they form a logging system inside Kubernetes.

Result

A working EFK stack collects, stores, and shows logs from all cluster containers.

Understanding deployment patterns ensures you can set up and scale EFK reliably in real clusters.

7

Expert

Handling log volume and retention in EFK

🤔Before reading on: do you think storing all logs forever is practical? Commit to your answer.

Concept: Managing log volume and retention policies is critical to keep Elasticsearch performant and costs manageable.

Logs can grow very fast, filling storage and slowing searches. Experts set retention policies to delete old logs automatically. They use index lifecycle management to move older logs to cheaper storage or delete them. They also tune Fluentd to filter unnecessary logs and compress data.

Result

EFK stack remains fast and cost-effective even with large log volumes.

Knowing how to manage log lifecycle prevents system overload and keeps logging sustainable in production.

Under the Hood

Fluentd runs as an agent on each Kubernetes node, reading container logs from files or stdout. It buffers and processes logs, then sends them via HTTP or TCP to Elasticsearch. Elasticsearch stores logs in indexes, which are optimized data structures for fast search. Kibana queries Elasticsearch and renders logs in dashboards using web technologies. The system uses APIs and persistent storage to ensure logs are durable and searchable.

Why designed this way?

EFK was designed to handle large, distributed systems where logs come from many sources. Fluentd’s pluggable architecture allows flexible log collection and transformation. Elasticsearch’s indexing enables fast search over huge data. Kibana provides a user-friendly way to explore logs without coding. Alternatives like direct log storage or manual collection were too slow or complex for modern cloud-native environments.

┌─────────────┐       ┌─────────────┐       ┌─────────────┐       ┌─────────────┐
│ Kubernetes  │       │  Fluentd    │       │ Elasticsearch│       │   Kibana    │
│ Containers  │──────▶│ (Collector) │──────▶│ (Indexer &   │──────▶│ (Visualizer)│
│ (Log Source)│       │             │       │  Storage)   │       │             │
└─────────────┘       └─────────────┘       └─────────────┘       └─────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Fluentd store logs permanently or just forward them? Commit to your answer.

Common Belief: Fluentd stores all logs permanently on each node.

Reality: Fluentd only collects and forwards logs; it does not store them long-term.

Why it matters: Believing Fluentd stores logs can lead to missing logs if nodes fail, causing data loss.

Quick: Is Kibana a log storage system or just a visualization tool? Commit to your answer.

Common Belief: Kibana stores logs and manages their retention.

Reality: Kibana only visualizes logs stored in Elasticsearch; it does not store logs itself.

Why it matters: Misunderstanding this can cause confusion about where to manage log data and retention.

Quick: Can you keep all logs forever without issues? Commit to your answer.

Common Belief: Storing all logs forever is always best for troubleshooting.

Reality: Storing all logs indefinitely causes storage overload and slows down search performance.

Why it matters: Ignoring log retention leads to system crashes and high costs.

Quick: Does centralized logging solve all monitoring problems alone? Commit to your answer.

Common Belief: Centralized logging replaces the need for other monitoring tools.

Reality: Centralized logging complements but does not replace metrics and tracing tools.

Why it matters: Relying only on logs can miss performance issues that metrics or tracing reveal.

Expert Zone

1

Fluentd’s buffering and retry mechanisms prevent log loss during network or Elasticsearch outages, but misconfiguration can cause delays or duplicates.

2

Elasticsearch index sharding and replication settings greatly affect performance and fault tolerance; tuning these is key for large clusters.

3

Kibana’s saved searches and dashboards can be shared and versioned, enabling team collaboration and consistent incident response.

When NOT to use

EFK is not ideal for extremely high log volumes without scaling Elasticsearch properly; alternatives like Loki or Splunk may be better. For simple or small setups, lightweight log collectors or cloud-managed logging services might be easier.

Production Patterns

In production, teams deploy Fluentd as a DaemonSet with custom filters to reduce noise. Elasticsearch uses index lifecycle management to automate retention. Kibana dashboards are customized per team needs. Logs are secured with role-based access control and encrypted transport.

Connections

Distributed Tracing

Complementary tool for understanding request flows alongside logs

Knowing centralized logging helps you appreciate how tracing fills gaps by showing how requests move through services, not just what happened.

Database Indexing

Similar concept of indexing data for fast search

Understanding Elasticsearch indexing is easier if you know how databases create indexes to speed up queries.

Library Cataloging Systems

Both organize large amounts of information for quick retrieval

Centralized logging’s indexing and search is like a library catalog helping find books quickly among thousands.

Common Pitfalls

#1 Not configuring Fluentd to handle log rotation causes missing logs.

Wrong approach:

fluentd.conf without proper file input settings:

@type tail
path /var/log/containers/*.log
pos_file /var/log/fluentd.pos
tag kubernetes.*

Correct approach:

fluentd.conf with log rotation handling:

@type tail
path /var/log/containers/*.log
pos_file /var/log/fluentd.pos
tag kubernetes.*
read_from_head true
refresh_interval 5

Root cause:

Ignoring how container logs rotate leads Fluentd to miss new logs or read old ones repeatedly.

#2 Storing all logs forever without retention causes Elasticsearch to slow down and run out of disk.

Wrong approach:

No index lifecycle management configured; all indexes kept indefinitely.

Correct approach:

Set index lifecycle policy to delete or archive logs older than 30 days.

Root cause:

Not planning log retention leads to resource exhaustion and degraded cluster performance.

#3 Running Kibana without securing access exposes sensitive logs to anyone.

Wrong approach:

Kibana deployed with default settings and no authentication.

Correct approach:

Enable authentication and role-based access control in Kibana and Elasticsearch.

Root cause:

Overlooking security risks exposes logs containing sensitive information.

Key Takeaways

Centralized logging with the EFK stack collects logs from many containers into one searchable place.

Fluentd gathers and forwards logs, Elasticsearch stores and indexes them, and Kibana visualizes logs for easy analysis.

Without centralized logging, troubleshooting distributed systems is slow and error-prone.

Proper deployment and management of EFK components ensure reliable, scalable, and secure logging.

Managing log volume and retention is essential to keep the system performant and cost-effective.

Practice

1. What is the main purpose of the EFK stack in Kubernetes?

easy

A. To collect, store, and visualize logs from all pods centrally

B. To manage Kubernetes cluster networking

C. To automate deployment of applications

D. To monitor CPU and memory usage only

Solution

1. Step 1: Understand EFK components

   The EFK stack consists of Fluentd (log collector), Elasticsearch (log storage), and Kibana (log viewer).

2. Step 2: Identify the main goal

   Its main goal is to centralize logs from all Kubernetes pods for easier troubleshooting and monitoring.

3. Final Answer:

   To collect, store, and visualize logs from all pods centrally -> Option A

4. Quick Check:

   EFK = Centralized logging [OK]

Hint: EFK means Fluentd, Elasticsearch, Kibana for logs [OK]

Common Mistakes:

• Confusing EFK with monitoring CPU/memory
• Thinking EFK manages networking
• Assuming EFK automates deployments

2. Which Kubernetes resource is typically used to deploy Fluentd as a log collector in the EFK stack?

easy

A. ServiceAccount

B. DaemonSet

C. Deployment

D. ConfigMap

Solution

1. Step 1: Understand Fluentd deployment needs

   Fluentd must run on every node to collect logs from all pods on that node.

2. Step 2: Choose correct Kubernetes resource

   DaemonSet ensures one pod per node, perfect for log collectors like Fluentd.

3. Final Answer:

   DaemonSet -> Option B

4. Quick Check:

   Fluentd runs as DaemonSet [OK]

Hint: DaemonSet runs pods on all nodes [OK]

Common Mistakes:

• Using Deployment which may not run on all nodes
• Confusing ConfigMap with deployment type
• Thinking ServiceAccount deploys pods

3. Given this Fluentd config snippet in Kubernetes:

match ** {
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
}

What is the main effect of this configuration?

medium

A. Fluentd sends all logs to Elasticsearch service at port 9200

B. Fluentd collects logs only from pods named elasticsearch

C. Fluentd stores logs locally on each node

D. Fluentd forwards logs to Kibana directly

Solution

1. Step 1: Analyze Fluentd match directive

   The match ** means all logs are matched and processed by this output plugin.

2. Step 2: Understand output plugin settings

   @type elasticsearch with host and port means logs are sent to Elasticsearch service at that address.

3. Final Answer:

   Fluentd sends all logs to Elasticsearch service at port 9200 -> Option A

4. Quick Check:

   match ** + elasticsearch output = send all logs to ES [OK]

Hint: match ** means all logs sent to Elasticsearch [OK]

Common Mistakes:

• Thinking logs go directly to Kibana
• Assuming logs are stored locally
• Confusing match pattern with pod names

4. You deployed the EFK stack but Kibana shows no logs. Which of these is the most likely cause?

medium

A. Kibana is configured to use wrong Elasticsearch URL

B. Elasticsearch service port is set to 8080 instead of 9200

C. All of the above

D. Fluentd DaemonSet is not running on nodes

Solution

1. Step 1: Check Fluentd status

   If Fluentd pods are not running, logs won't be collected or sent.

2. Step 2: Verify Elasticsearch connectivity

   Wrong port on Elasticsearch service means Fluentd can't send logs properly.

3. Step 3: Confirm Kibana configuration

   If Kibana points to wrong Elasticsearch URL, it can't display logs.

4. Final Answer:

   All of the above -> Option C

5. Quick Check:

   Any broken link in EFK stops logs [OK]

Hint: Check Fluentd, Elasticsearch port, Kibana URL [OK]

Common Mistakes:

• Checking only one component
• Ignoring service port mismatch
• Assuming Kibana auto-fixes URLs

5. You want to filter out logs from Kubernetes system namespaces (like kube-system and default) before sending to Elasticsearch in Fluentd. Which configuration snippet achieves this?

hard

A. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^kube-system$

}

B. filter ** {
@type record_transformer
remove_keys kubernetes.namespace_name
}

C. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^kube-system$

}

D. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^(kube-system|default)$

}

Solution

1. Step 1: Understand filtering with Fluentd grep plugin

   The grep plugin can exclude logs matching certain patterns using blocks.

2. Step 2: Identify namespaces to exclude

   We want to exclude system namespaces like kube-system and default, so pattern must match both.

3. Step 3: Compare options

   filter ** {
     @type grep
     
       key kubernetes.namespace_name
       pattern ^(kube-system|default)$
     
   }

   excludes both kube-system and default namespaces correctly; others exclude only one or do wrong action.

4. Final Answer:

   filter ** { @type grep key kubernetes.namespace_name pattern ^(kube-system|default)$ } -> Option D

5. Quick Check:

   Exclude system namespaces with grep exclude pattern [OK]

Hint: Use grep exclude with regex for system namespaces [OK]

Common Mistakes:

• Excluding only one namespace
• Using include instead of exclude
• Removing keys instead of filtering logs