Prometheus and Grafana integration in Kafka - Commands & Configuration

Introduction

When running Kafka, you want to see how well it works and catch problems early. Prometheus collects data about Kafka's health and performance. Grafana shows this data in easy-to-understand graphs and dashboards.

When you want to watch Kafka's message rates and lag in real time.

When you need alerts if Kafka brokers or topics have issues.

When you want to understand resource use like CPU and memory for Kafka.

When you want to share Kafka performance reports with your team visually.

When you want to troubleshoot Kafka problems by looking at historical data.

Config File - kafka-prometheus.yml

kafka-prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'kafka'
    static_configs:
      - targets: ['localhost:9308']

This is the Prometheus configuration file to collect Kafka metrics.

global: sets how often Prometheus collects data.

scrape_configs: tells Prometheus where to get Kafka metrics, here from localhost port 9308 where Kafka exporter runs.

Commands

This command runs the Kafka exporter container that collects Kafka metrics and exposes them on port 9308 for Prometheus to scrape.

Terminal

docker run -d --name kafka-exporter -p 9308:9308 danielqsj/kafka-exporter --kafka.server=localhost:9092

Expected Output

a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

→

-d - Run container in background

→

-p 9308:9308 - Expose exporter port for Prometheus

→

--kafka.server=localhost:9092 - Connect exporter to Kafka broker

Starts Prometheus with the configuration file that tells it to collect Kafka metrics from the exporter.

Terminal

prometheus --config.file=kafka-prometheus.yml

Expected Output

level=info ts=2024-06-01T12:00:00.000Z caller=main.go:123 msg="Starting Prometheus" version="2.45.0"

→

--config.file - Specify Prometheus config file

Runs Grafana container to visualize metrics. It listens on port 3000 for the web interface.

Terminal

docker run -d --name grafana -p 3000:3000 grafana/grafana

Expected Output

b1c2d3e4f5g6h7i8j9k0l1m2n3o4p5q6

→

-d - Run container in background

→

-p 3000:3000 - Expose Grafana web UI port

Checks if Grafana is running and healthy by requesting its health API.

Terminal

curl http://localhost:3000/api/health

Expected Output

{"database":"ok","version":"9.5.2","commit":"abc123"}

Key Concept

If you remember nothing else from this pattern, remember: Prometheus collects Kafka metrics and Grafana shows them visually for easy monitoring.

Common Mistakes

Not running the Kafka exporter before starting Prometheus

Prometheus cannot collect Kafka metrics if the exporter is not running and exposing them.

Always start the Kafka exporter container first so Prometheus can scrape metrics.

Using wrong port or address in Prometheus config for Kafka exporter

Prometheus will fail to scrape metrics if the target address or port is incorrect.

Verify the exporter runs on the configured host and port, here localhost:9308.

Not exposing Grafana port to access the web UI

Without port mapping, you cannot open Grafana dashboard in your browser.

Use -p 3000:3000 flag when running Grafana container.

Summary

Run Kafka exporter to expose Kafka metrics on port 9308.

Configure Prometheus to scrape metrics from Kafka exporter.

Start Prometheus with the config file to collect metrics.

Run Grafana to visualize the collected Kafka metrics.

Check Grafana health to confirm it is running.

Practice

1. What is the main role of Prometheus in Kafka monitoring when integrated with Grafana?

easy

A. To collect Kafka metrics by scraping the metrics endpoint

B. To display visual dashboards of Kafka metrics

C. To send alerts directly to Kafka topics

D. To store Kafka messages for backup

Solution

1. Step 1: Understand Prometheus function

   Prometheus collects metrics by scraping endpoints exposed by applications like Kafka.

2. Step 2: Differentiate roles of Prometheus and Grafana

   Grafana is used for visualization, not data collection.

3. Final Answer:

   To collect Kafka metrics by scraping the metrics endpoint -> Option A

4. Quick Check:

   Prometheus collects metrics = A [OK]

Hint: Remember: Prometheus collects, Grafana displays [OK]

Common Mistakes:

• Confusing Grafana as the collector
• Thinking Prometheus stores Kafka messages
• Assuming Prometheus sends alerts to Kafka

2. Which of the following is the correct Prometheus scrape configuration snippet to monitor Kafka metrics exposed at port 9090?

easy

A. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:9090']

B. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:2181']

C. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:8080']

D. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:1234']

Solution

1. Step 1: Identify Kafka metrics port

   Kafka metrics exporter commonly exposes metrics on port 9090.

2. Step 2: Match correct target in Prometheus config

   Only - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:9090'] uses port 9090, which is correct for scraping Kafka metrics.

3. Final Answer:

   - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:9090'] -> Option A

4. Quick Check:

   Port 9090 for Kafka metrics = A [OK]

Hint: Kafka metrics usually on port 9090, check targets carefully [OK]

Common Mistakes:

• Using Kafka broker port 2181 (Zookeeper port) instead of metrics port
• Confusing HTTP ports like 8080 for metrics
• Using random ports not related to Kafka metrics

3. Given this Prometheus query to monitor Kafka consumer lag:
sum(kafka_consumer_group_lag) by (group)
What output will this query produce?

medium

A. Current CPU usage of Kafka brokers

B. Total lag per consumer group as a number

C. Number of active Kafka brokers

D. List of all Kafka topics with their lag

Solution

1. Step 1: Understand the metric and aggregation

   The metric kafka_consumer_group_lag measures lag per consumer group and partition. Summing by group aggregates lag per group.

2. Step 2: Interpret the query output

   The query returns total lag number for each consumer group, not topics or CPU usage.

3. Final Answer:

   Total lag per consumer group as a number -> Option B

4. Quick Check:

   sum by group = total lag per group [OK]

Hint: sum(... by (group)) means total per group [OK]

Common Mistakes:

• Thinking it lists topics instead of groups
• Confusing lag with broker count
• Assuming CPU metrics from Kafka lag metric

4. You configured Prometheus to scrape Kafka metrics but Grafana shows no data. Which is the most likely cause?

medium

A. Prometheus scrape interval is too short

B. Grafana dashboard is not connected to Prometheus

C. Kafka metrics endpoint is not exposed or unreachable

D. Kafka broker is down

Solution

1. Step 1: Check Prometheus scraping setup

   If Kafka metrics endpoint is not exposed or unreachable, Prometheus cannot collect data.

2. Step 2: Understand impact on Grafana

   Without data in Prometheus, Grafana dashboards show no metrics.

3. Final Answer:

   Kafka metrics endpoint is not exposed or unreachable -> Option C

4. Quick Check:

   No data means no scrape target reachable [OK]

Hint: No data? Check if metrics endpoint is reachable first [OK]

Common Mistakes:

• Blaming Grafana connection before checking Prometheus data
• Assuming Kafka broker down always causes no data
• Thinking scrape interval too short causes no data

5. You want to create a Grafana dashboard showing Kafka broker CPU usage and consumer lag side by side. Which data sources and queries should you use?

hard

A. Connect Grafana to Prometheus; use queries kafka_cpu_usage and kafka_topic_lag

B. Connect Grafana directly to Kafka; use queries kafka_cpu and kafka_lag

C. Connect Grafana to Kafka Connect; use queries cpu_usage and consumer_lag

D. Connect Grafana to Prometheus; use queries kafka_broker_cpu_usage and sum(kafka_consumer_group_lag) by (group)

Solution

1. Step 1: Identify correct data source for metrics

   Prometheus collects Kafka metrics; Grafana connects to Prometheus for queries.

2. Step 2: Choose correct metric names and queries

   Common metrics are kafka_broker_cpu_usage and kafka_consumer_group_lag aggregated by group.

3. Final Answer:

   Connect Grafana to Prometheus; use queries kafka_broker_cpu_usage and sum(kafka_consumer_group_lag) by (group) -> Option D

4. Quick Check:

   Prometheus + correct queries = C [OK]

Hint: Grafana queries Prometheus metrics, not Kafka directly [OK]

Common Mistakes:

• Trying to connect Grafana directly to Kafka brokers
• Using wrong metric names or unaggregated lag
• Confusing Kafka Connect with Kafka broker metrics