Centralized logging (EFK stack) in Kubernetes - Mini Project: Build & Apply

Centralized logging with EFK stack on Kubernetes

📖 Scenario: You are managing a Kubernetes cluster for a small company. You want to collect logs from all running applications in one place to easily search and analyze them.To do this, you will set up the EFK stack: Elasticsearch to store logs, Fluentd to collect and forward logs, and Kibana to view logs in a web interface.

🎯 Goal: Build a simple EFK stack on Kubernetes that collects logs from all pods and allows viewing them in Kibana.

📋 What You'll Learn

Create a Kubernetes namespace called logging

Deploy Elasticsearch StatefulSet with 1 replica in logging namespace

Deploy Fluentd DaemonSet in logging namespace to collect logs from all nodes

Deploy Kibana Deployment with 1 replica in logging namespace

Expose Kibana with a ClusterIP service

Verify logs are collected and visible in Kibana

💡 Why This Matters

🌍 Real World

Centralized logging helps teams monitor and troubleshoot applications by collecting logs from many sources into one place.

💼 Career

DevOps engineers often set up logging stacks like EFK on Kubernetes clusters to improve observability and support incident response.

1

Create the logging namespace

Write a YAML manifest to create a Kubernetes namespace called logging. Use kubectl apply -f to apply it.

Kubernetes

# Create a namespace named logging
# Your code here

Hint

A namespace manifest uses apiVersion: v1 and kind: Namespace. The name goes under metadata.

2

Deploy Elasticsearch StatefulSet in logging namespace

Write a YAML manifest to deploy Elasticsearch as a StatefulSet with 1 replica in the logging namespace. Use the image docker.elastic.co/elasticsearch/elasticsearch:7.17.0. Set environment variable discovery.type to single-node. Use port 9200. Apply the manifest with kubectl apply -f.

Kubernetes

apiVersion: v1
kind: Namespace
metadata:
  name: logging

# Add Elasticsearch StatefulSet manifest below
# Your code here

Hint

Use kind: StatefulSet with replicas: 1. Set environment variable discovery.type: single-node for Elasticsearch single node mode.

3

Deploy Fluentd DaemonSet to collect logs

Write a YAML manifest to deploy Fluentd as a DaemonSet in the logging namespace. Use the image fluent/fluentd:v1.14-debian-1. Mount the host's /var/log directory to /var/log inside the container. Apply the manifest with kubectl apply -f.

Kubernetes

apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
        ports:
        - containerPort: 9200
          name: http
        env:
        - name: discovery.type
          value: single-node

# Add Fluentd DaemonSet manifest below
# Your code here

Hint

Use kind: DaemonSet to run Fluentd on all nodes. Mount /var/log from host to container.

4

Deploy Kibana and expose it with a ClusterIP service

Write a YAML manifest to deploy Kibana as a Deployment with 1 replica in the logging namespace. Use the image docker.elastic.co/kibana/kibana:7.17.0. Expose port 5601. Create a ClusterIP service named kibana in the logging namespace exposing port 5601. Apply the manifest with kubectl apply -f. Then run kubectl get pods -n logging and kubectl get svc -n logging to verify pods and service are running.

Kubernetes

apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
        ports:
        - containerPort: 9200
          name: http
        env:
        - name: discovery.type
          value: single-node
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.14-debian-1
        volumeMounts:
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: varlog
        hostPath:
          path: /var/log

# Add Kibana Deployment and Service manifests below
# Your code here

Hint

Deploy Kibana as a Deployment with 1 replica. Expose it with a ClusterIP service on port 5601.

Practice

1. What is the main purpose of the EFK stack in Kubernetes?

easy

A. To collect, store, and visualize logs from all pods centrally

B. To manage Kubernetes cluster networking

C. To automate deployment of applications

D. To monitor CPU and memory usage only

Solution

1. Step 1: Understand EFK components

   The EFK stack consists of Fluentd (log collector), Elasticsearch (log storage), and Kibana (log viewer).

2. Step 2: Identify the main goal

   Its main goal is to centralize logs from all Kubernetes pods for easier troubleshooting and monitoring.

3. Final Answer:

   To collect, store, and visualize logs from all pods centrally -> Option A

4. Quick Check:

   EFK = Centralized logging [OK]

Hint: EFK means Fluentd, Elasticsearch, Kibana for logs [OK]

Common Mistakes:

• Confusing EFK with monitoring CPU/memory
• Thinking EFK manages networking
• Assuming EFK automates deployments

2. Which Kubernetes resource is typically used to deploy Fluentd as a log collector in the EFK stack?

easy

A. ServiceAccount

B. DaemonSet

C. Deployment

D. ConfigMap

Solution

1. Step 1: Understand Fluentd deployment needs

   Fluentd must run on every node to collect logs from all pods on that node.

2. Step 2: Choose correct Kubernetes resource

   DaemonSet ensures one pod per node, perfect for log collectors like Fluentd.

3. Final Answer:

   DaemonSet -> Option B

4. Quick Check:

   Fluentd runs as DaemonSet [OK]

Hint: DaemonSet runs pods on all nodes [OK]

Common Mistakes:

• Using Deployment which may not run on all nodes
• Confusing ConfigMap with deployment type
• Thinking ServiceAccount deploys pods

3. Given this Fluentd config snippet in Kubernetes:

match ** {
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
}

What is the main effect of this configuration?

medium

A. Fluentd sends all logs to Elasticsearch service at port 9200

B. Fluentd collects logs only from pods named elasticsearch

C. Fluentd stores logs locally on each node

D. Fluentd forwards logs to Kibana directly

Solution

1. Step 1: Analyze Fluentd match directive

   The match ** means all logs are matched and processed by this output plugin.

2. Step 2: Understand output plugin settings

   @type elasticsearch with host and port means logs are sent to Elasticsearch service at that address.

3. Final Answer:

   Fluentd sends all logs to Elasticsearch service at port 9200 -> Option A

4. Quick Check:

   match ** + elasticsearch output = send all logs to ES [OK]

Hint: match ** means all logs sent to Elasticsearch [OK]

Common Mistakes:

• Thinking logs go directly to Kibana
• Assuming logs are stored locally
• Confusing match pattern with pod names

4. You deployed the EFK stack but Kibana shows no logs. Which of these is the most likely cause?

medium

A. Kibana is configured to use wrong Elasticsearch URL

B. Elasticsearch service port is set to 8080 instead of 9200

C. All of the above

D. Fluentd DaemonSet is not running on nodes

Solution

1. Step 1: Check Fluentd status

   If Fluentd pods are not running, logs won't be collected or sent.

2. Step 2: Verify Elasticsearch connectivity

   Wrong port on Elasticsearch service means Fluentd can't send logs properly.

3. Step 3: Confirm Kibana configuration

   If Kibana points to wrong Elasticsearch URL, it can't display logs.

4. Final Answer:

   All of the above -> Option C

5. Quick Check:

   Any broken link in EFK stops logs [OK]

Hint: Check Fluentd, Elasticsearch port, Kibana URL [OK]

Common Mistakes:

• Checking only one component
• Ignoring service port mismatch
• Assuming Kibana auto-fixes URLs

5. You want to filter out logs from Kubernetes system namespaces (like kube-system and default) before sending to Elasticsearch in Fluentd. Which configuration snippet achieves this?

hard

A. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^kube-system$

}

B. filter ** {
@type record_transformer
remove_keys kubernetes.namespace_name
}

C. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^kube-system$

}

D. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^(kube-system|default)$

}

Solution

1. Step 1: Understand filtering with Fluentd grep plugin

   The grep plugin can exclude logs matching certain patterns using blocks.

2. Step 2: Identify namespaces to exclude

   We want to exclude system namespaces like kube-system and default, so pattern must match both.

3. Step 3: Compare options

   filter ** {
     @type grep
     
       key kubernetes.namespace_name
       pattern ^(kube-system|default)$
     
   }

   excludes both kube-system and default namespaces correctly; others exclude only one or do wrong action.

4. Final Answer:

   filter ** { @type grep key kubernetes.namespace_name pattern ^(kube-system|default)$ } -> Option D

5. Quick Check:

   Exclude system namespaces with grep exclude pattern [OK]

Hint: Use grep exclude with regex for system namespaces [OK]

Common Mistakes:

• Excluding only one namespace
• Using include instead of exclude
• Removing keys instead of filtering logs