How to Use Loki for Logging Kubernetes Clusters

To use Loki for logging in Kubernetes, deploy Loki as a logging backend and configure Promtail as an agent on your nodes to collect logs. Promtail sends logs to Loki, which stores and indexes them for querying via Grafana.

Syntax

The main components to use Loki for Kubernetes logging are:

• Loki: The log aggregation system that stores logs.
• Promtail: The agent that runs on Kubernetes nodes to collect logs and send them to Loki.
• Grafana: The UI to query and visualize logs stored in Loki.

Typical usage involves deploying Loki and Promtail using Kubernetes manifests or Helm charts, then configuring Promtail to watch log files and send them to Loki's HTTP API.

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: logging
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
    clients:
      - url: http://loki:3100/loki/api/v1/push
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_name]
            target_label: job
          - source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
          - source_labels: [__meta_kubernetes_pod_container_name]
            target_label: container
        pipeline_stages:
          - docker: {}
        static_configs:
          - targets:
              - localhost
            labels:
              job: kubernetes-pods
              __path__: /var/log/containers/*.log

Example

This example shows how to deploy Loki and Promtail in a Kubernetes cluster using Helm, then query logs in Grafana.

bash

# Add Loki Helm repo
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Loki
helm install loki grafana/loki-stack --namespace logging --create-namespace --set promtail.enabled=true

# Install Grafana
helm install grafana grafana/grafana --namespace logging

# Port-forward Grafana to access UI
kubectl port-forward svc/grafana 3000:80 -n logging

# Login to Grafana at http://localhost:3000 (default admin/admin)
# Add Loki as a data source in Grafana
# Use Explore tab to query logs with Loki query language

Output

NAME: loki
LAST DEPLOYED: <date>
NAMESPACE: logging
STATUS: deployed

NAME: grafana
LAST DEPLOYED: <date>
NAMESPACE: logging
STATUS: deployed

Forwarding from 127.0.0.1:3000 -> 80

Common Pitfalls

• Incorrect Promtail config: Not setting the correct __path__ label to match container log files causes no logs to be collected.
• Network issues: Promtail must reach Loki's HTTP endpoint; network policies or service misconfigurations can block this.
• Resource limits: Loki and Promtail need enough CPU and memory; insufficient resources cause log loss or crashes.
• Time synchronization: Logs with wrong timestamps due to unsynced clocks make querying confusing.

yaml

Wrong Promtail snippet:
static_configs:
  - targets:
      - localhost
    labels:
      __path__: /var/log/*.log  # Too broad, misses container logs

Correct Promtail snippet:
static_configs:
  - targets:
      - localhost
    labels:
      __path__: /var/log/containers/*.log  # Matches Kubernetes container logs

Quick Reference

Component	Purpose	Common Commands/Config
Loki	Stores and indexes logs	Helm install: helm install loki grafana/loki-stack
Promtail	Collects logs from nodes and sends to Loki	ConfigMap with promtail.yaml, watch /var/log/containers/*.log
Grafana	Visualizes logs from Loki	Add Loki data source, use Explore tab for queries

Key Takeaways

Deploy Loki and Promtail in Kubernetes to collect and store logs centrally.

Configure Promtail to watch Kubernetes container log paths and send logs to Loki.

Use Grafana connected to Loki to query and visualize logs easily.

Ensure network access and correct Promtail paths to avoid missing logs.

Monitor resource usage and time sync for reliable logging.