How to Use ELK Stack with Kubernetes: Setup and Best Practices

To use the ELK stack with Kubernetes, deploy Elasticsearch, Logstash, and Kibana as pods or stateful sets in your cluster using YAML manifests or Helm charts. Configure Logstash to collect logs from Kubernetes pods and send them to Elasticsearch, then use Kibana to visualize and analyze the logs.

Syntax

The ELK stack on Kubernetes typically involves deploying three main components:

• Elasticsearch: Stores and indexes logs.
• Logstash: Collects, processes, and forwards logs.
• Kibana: Visualizes logs and dashboards.

You deploy these using Kubernetes manifests or Helm charts. Each component runs as a pod or stateful set with services exposing them internally or externally.

Basic YAML manifest parts include:

• apiVersion: Kubernetes API version.
• kind: Resource type (Pod, Deployment, StatefulSet, Service).
• metadata: Name and labels.
• spec: Pod template, containers, volumes, ports.

yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.7.0
        ports:
        - containerPort: 9200
          name: http
        env:
        - name: discovery.type
          value: single-node

Example

This example shows a simple deployment of Elasticsearch, Logstash, and Kibana on Kubernetes using YAML manifests. It demonstrates how to deploy Elasticsearch as a StatefulSet, Logstash as a Deployment configured to receive logs, and Kibana as a Deployment to visualize logs.

yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.7.0
        ports:
        - containerPort: 9200
          name: http
        env:
        - name: discovery.type
          value: single-node
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
spec:
  ports:
  - port: 9200
    name: http
  selector:
    app: elasticsearch
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:8.7.0
        ports:
        - containerPort: 5044
        volumeMounts:
        - name: config-volume
          mountPath: /usr/share/logstash/pipeline/
      volumes:
      - name: config-volume
        configMap:
          name: logstash-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-config
data:
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }
    output {
      elasticsearch {
        hosts => ["http://elasticsearch:9200"]
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:8.7.0
        ports:
        - containerPort: 5601
        env:
        - name: ELASTICSEARCH_HOSTS
          value: "http://elasticsearch:9200"
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
spec:
  ports:
  - port: 5601
    name: http
  selector:
    app: kibana

Output

kubectl get pods
NAME              READY   STATUS    RESTARTS   AGE
elasticsearch-0   1/1     Running   0          2m
logstash-xxxxx    1/1     Running   0          1m
kibana-xxxxx      1/1     Running   0          1m

Common Pitfalls

Common mistakes when using ELK stack with Kubernetes include:

• Not configuring persistent storage for Elasticsearch, causing data loss on pod restarts.
• Incorrect Logstash pipeline configuration, leading to no logs being forwarded.
• Missing service discovery or wrong service names causing Kibana or Logstash to fail connecting to Elasticsearch.
• Exposing Kibana without authentication, risking security.

Always use persistent volumes for Elasticsearch and validate your Logstash config before deployment.

conf

Wrong Logstash output config example:
output {
  elasticsearch {
    hosts => ["http://wrong-service:9200"]
  }
}

Right Logstash output config example:
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
  }
}

Quick Reference

Tips for using ELK stack on Kubernetes:

• Use Helm charts like elastic/elastic-stack for easier deployment.
• Configure persistent volumes for Elasticsearch data.
• Use Kubernetes logging agents like Filebeat to send pod logs to Logstash.
• Secure Kibana with authentication and network policies.
• Monitor resource usage to scale ELK components properly.

Key Takeaways

Deploy Elasticsearch, Logstash, and Kibana as Kubernetes pods or stateful sets with proper service exposure.

Configure Logstash pipelines correctly to forward Kubernetes logs to Elasticsearch.

Use persistent storage for Elasticsearch to avoid data loss on pod restarts.

Secure Kibana access and monitor ELK resource usage for stable operation.

Consider using Helm charts and Kubernetes logging agents like Filebeat for easier setup.