Centralized logging (EFK stack) in Kubernetes - Commands & Configuration

Introduction

When you run many applications on Kubernetes, logs are scattered across many places. Centralized logging collects all logs in one place so you can easily search and analyze them. The EFK stack uses Elasticsearch to store logs, Fluentd to collect and send logs, and Kibana to view logs in a friendly way.

When you want to see logs from all your Kubernetes pods in one dashboard.

When you need to quickly find errors or issues across multiple containers.

When you want to keep logs for a long time and search them efficiently.

When you want to monitor your applications without logging into each pod.

When you want to share logs with your team using a web interface.

Config File - efk-stack.yaml

efk-stack.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.6.3
        ports:
        - containerPort: 9200
        env:
        - name: discovery.type
          value: single-node
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: logging
spec:
  ports:
  - port: 9200
    targetPort: 9200
  selector:
    app: elasticsearch
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.14.5-debian-1.0
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: elasticsearch.logging.svc.cluster.local
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:8.6.3
        ports:
        - containerPort: 5601
        env:
        - name: ELASTICSEARCH_HOSTS
          value: http://elasticsearch.logging.svc.cluster.local:9200
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: logging
spec:
  ports:
  - port: 5601
    targetPort: 5601
  selector:
    app: kibana

This file creates a logging namespace to keep logging components separate. It deploys Elasticsearch as a single-node cluster to store logs. A Fluentd DaemonSet runs on every node to collect logs from containers and send them to Elasticsearch. Kibana is deployed to provide a web interface to search and view logs. Services expose Elasticsearch and Kibana inside the cluster.

Commands

This command creates all the EFK stack components in Kubernetes: Elasticsearch, Fluentd, and Kibana with their services and namespace.

Terminal

kubectl apply -f efk-stack.yaml

Expected Output

namespace/logging created

deployment.apps/elasticsearch created

service/elasticsearch created

daemonset.apps/fluentd created

deployment.apps/kibana created

service/kibana created

Check that all pods for Elasticsearch, Fluentd, and Kibana are running in the logging namespace.

Terminal

kubectl get pods -n logging

Expected Output

NAME                             READY   STATUS    RESTARTS   AGE

elasticsearch-xxxxxxxxxx-xxxxx   1/1     Running   0          1m

fluentd-xxxxx                   1/1     Running   0          1m

kibana-xxxxxxxxxx-xxxxx          1/1     Running   0          1m

Forward local port 5601 to Kibana service so you can open Kibana dashboard in your browser at http://localhost:5601.

Terminal

kubectl port-forward svc/kibana 5601:5601 -n logging

Expected Output

Forwarding from 127.0.0.1:5601 -> 5601

Forwarding from [::1]:5601 -> 5601

→

-n - Specify the namespace where Kibana service is running

View the last 10 log lines from Fluentd pods to verify it is collecting and forwarding logs.

Terminal

kubectl logs -l app=fluentd -n logging --tail=10

Expected Output

[2024-06-01 12:00:00] Fluentd started

[2024-06-01 12:00:05] Sending logs to Elasticsearch

[2024-06-01 12:00:10] Successfully sent batch of logs

→

-l - Select pods by label

→

--tail - Show only last N lines

Key Concept

If you remember nothing else from this pattern, remember: Fluentd collects logs from all nodes and sends them to Elasticsearch, where Kibana lets you search and view them easily.

Common Mistakes

Not creating the logging namespace before applying the EFK stack.

Resources fail to create because the namespace does not exist.

Apply the full efk-stack.yaml which includes the namespace creation or create the namespace first with kubectl create namespace logging.

Trying to access Kibana without port-forwarding or exposing the service.

Kibana is not accessible outside the cluster by default, so the browser cannot connect.

Use kubectl port-forward to access Kibana locally or create an ingress or LoadBalancer service.

Not mounting the correct log directories in Fluentd DaemonSet.

Fluentd cannot read container logs, so no logs are collected or sent.

Mount /var/log and /var/lib/docker/containers as shown in the config to allow Fluentd to read logs.

Summary

Apply the efk-stack.yaml file to deploy Elasticsearch, Fluentd, and Kibana in the logging namespace.

Verify pods are running with kubectl get pods -n logging.

Use kubectl port-forward to access Kibana dashboard locally.

Check Fluentd logs to confirm it is collecting and forwarding logs.

Practice

1. What is the main purpose of the EFK stack in Kubernetes?

easy

A. To collect, store, and visualize logs from all pods centrally

B. To manage Kubernetes cluster networking

C. To automate deployment of applications

D. To monitor CPU and memory usage only

Solution

1. Step 1: Understand EFK components

   The EFK stack consists of Fluentd (log collector), Elasticsearch (log storage), and Kibana (log viewer).

2. Step 2: Identify the main goal

   Its main goal is to centralize logs from all Kubernetes pods for easier troubleshooting and monitoring.

3. Final Answer:

   To collect, store, and visualize logs from all pods centrally -> Option A

4. Quick Check:

   EFK = Centralized logging [OK]

Hint: EFK means Fluentd, Elasticsearch, Kibana for logs [OK]

Common Mistakes:

• Confusing EFK with monitoring CPU/memory
• Thinking EFK manages networking
• Assuming EFK automates deployments

2. Which Kubernetes resource is typically used to deploy Fluentd as a log collector in the EFK stack?

easy

A. ServiceAccount

B. DaemonSet

C. Deployment

D. ConfigMap

Solution

1. Step 1: Understand Fluentd deployment needs

   Fluentd must run on every node to collect logs from all pods on that node.

2. Step 2: Choose correct Kubernetes resource

   DaemonSet ensures one pod per node, perfect for log collectors like Fluentd.

3. Final Answer:

   DaemonSet -> Option B

4. Quick Check:

   Fluentd runs as DaemonSet [OK]

Hint: DaemonSet runs pods on all nodes [OK]

Common Mistakes:

• Using Deployment which may not run on all nodes
• Confusing ConfigMap with deployment type
• Thinking ServiceAccount deploys pods

3. Given this Fluentd config snippet in Kubernetes:

match ** {
  @type elasticsearch
  host elasticsearch.logging.svc.cluster.local
  port 9200
}

What is the main effect of this configuration?

medium

A. Fluentd sends all logs to Elasticsearch service at port 9200

B. Fluentd collects logs only from pods named elasticsearch

C. Fluentd stores logs locally on each node

D. Fluentd forwards logs to Kibana directly

Solution

1. Step 1: Analyze Fluentd match directive

   The match ** means all logs are matched and processed by this output plugin.

2. Step 2: Understand output plugin settings

   @type elasticsearch with host and port means logs are sent to Elasticsearch service at that address.

3. Final Answer:

   Fluentd sends all logs to Elasticsearch service at port 9200 -> Option A

4. Quick Check:

   match ** + elasticsearch output = send all logs to ES [OK]

Hint: match ** means all logs sent to Elasticsearch [OK]

Common Mistakes:

• Thinking logs go directly to Kibana
• Assuming logs are stored locally
• Confusing match pattern with pod names

4. You deployed the EFK stack but Kibana shows no logs. Which of these is the most likely cause?

medium

A. Kibana is configured to use wrong Elasticsearch URL

B. Elasticsearch service port is set to 8080 instead of 9200

C. All of the above

D. Fluentd DaemonSet is not running on nodes

Solution

1. Step 1: Check Fluentd status

   If Fluentd pods are not running, logs won't be collected or sent.

2. Step 2: Verify Elasticsearch connectivity

   Wrong port on Elasticsearch service means Fluentd can't send logs properly.

3. Step 3: Confirm Kibana configuration

   If Kibana points to wrong Elasticsearch URL, it can't display logs.

4. Final Answer:

   All of the above -> Option C

5. Quick Check:

   Any broken link in EFK stops logs [OK]

Hint: Check Fluentd, Elasticsearch port, Kibana URL [OK]

Common Mistakes:

• Checking only one component
• Ignoring service port mismatch
• Assuming Kibana auto-fixes URLs

5. You want to filter out logs from Kubernetes system namespaces (like kube-system and default) before sending to Elasticsearch in Fluentd. Which configuration snippet achieves this?

hard

A. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^kube-system$

}

B. filter ** {
@type record_transformer
remove_keys kubernetes.namespace_name
}

C. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^kube-system$

}

D. filter ** {
@type grep

key kubernetes.namespace_name
pattern ^(kube-system|default)$

}

Solution

1. Step 1: Understand filtering with Fluentd grep plugin

   The grep plugin can exclude logs matching certain patterns using blocks.

2. Step 2: Identify namespaces to exclude

   We want to exclude system namespaces like kube-system and default, so pattern must match both.

3. Step 3: Compare options

   filter ** {
     @type grep
     
       key kubernetes.namespace_name
       pattern ^(kube-system|default)$
     
   }

   excludes both kube-system and default namespaces correctly; others exclude only one or do wrong action.

4. Final Answer:

   filter ** { @type grep key kubernetes.namespace_name pattern ^(kube-system|default)$ } -> Option D

5. Quick Check:

   Exclude system namespaces with grep exclude pattern [OK]

Hint: Use grep exclude with regex for system namespaces [OK]

Common Mistakes:

• Excluding only one namespace
• Using include instead of exclude
• Removing keys instead of filtering logs