Prometheus and Grafana integration in Kafka - Commands & Configuration

Introduction

When running Kafka, you want to see how well it works and catch problems early. Prometheus collects data about Kafka's health and performance. Grafana shows this data in easy-to-understand graphs and dashboards.

When you want to watch Kafka's message rates and lag in real time.

When you need alerts if Kafka brokers or topics have issues.

When you want to understand resource use like CPU and memory for Kafka.

When you want to share Kafka performance reports with your team visually.

When you want to troubleshoot Kafka problems by looking at historical data.

Config File - kafka-prometheus.yml

kafka-prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'kafka'
    static_configs:
      - targets: ['localhost:9308']

This is the Prometheus configuration file to collect Kafka metrics.

global: sets how often Prometheus collects data.

scrape_configs: tells Prometheus where to get Kafka metrics, here from localhost port 9308 where Kafka exporter runs.

Commands

This command runs the Kafka exporter container that collects Kafka metrics and exposes them on port 9308 for Prometheus to scrape.

Terminal

docker run -d --name kafka-exporter -p 9308:9308 danielqsj/kafka-exporter --kafka.server=localhost:9092

Expected Output

a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

→

-d - Run container in background

→

-p 9308:9308 - Expose exporter port for Prometheus

→

--kafka.server=localhost:9092 - Connect exporter to Kafka broker

Starts Prometheus with the configuration file that tells it to collect Kafka metrics from the exporter.

Terminal

prometheus --config.file=kafka-prometheus.yml

Expected Output

level=info ts=2024-06-01T12:00:00.000Z caller=main.go:123 msg="Starting Prometheus" version="2.45.0"

→

--config.file - Specify Prometheus config file

Runs Grafana container to visualize metrics. It listens on port 3000 for the web interface.

Terminal

docker run -d --name grafana -p 3000:3000 grafana/grafana

Expected Output

b1c2d3e4f5g6h7i8j9k0l1m2n3o4p5q6

→

-d - Run container in background

→

-p 3000:3000 - Expose Grafana web UI port

Checks if Grafana is running and healthy by requesting its health API.

Terminal

curl http://localhost:3000/api/health

Expected Output

{"database":"ok","version":"9.5.2","commit":"abc123"}

Key Concept

If you remember nothing else from this pattern, remember: Prometheus collects Kafka metrics and Grafana shows them visually for easy monitoring.

Common Mistakes

Not running the Kafka exporter before starting Prometheus

Prometheus cannot collect Kafka metrics if the exporter is not running and exposing them.

Always start the Kafka exporter container first so Prometheus can scrape metrics.

Using wrong port or address in Prometheus config for Kafka exporter

Prometheus will fail to scrape metrics if the target address or port is incorrect.

Verify the exporter runs on the configured host and port, here localhost:9308.

Not exposing Grafana port to access the web UI

Without port mapping, you cannot open Grafana dashboard in your browser.

Use -p 3000:3000 flag when running Grafana container.

Summary

Run Kafka exporter to expose Kafka metrics on port 9308.

Configure Prometheus to scrape metrics from Kafka exporter.

Start Prometheus with the config file to collect metrics.

Run Grafana to visualize the collected Kafka metrics.

Check Grafana health to confirm it is running.