Prometheus and Grafana integration in Kafka - Deep Dive

Overview - Prometheus and Grafana integration

What is it?

Prometheus is a tool that collects and stores data about how software and systems perform. Grafana is a tool that takes this data and shows it in easy-to-understand charts and dashboards. Integrating Prometheus with Grafana means connecting them so you can see live and historical performance data visually. This helps teams watch their systems and fix problems quickly.

Why it matters

Without Prometheus and Grafana working together, teams would have to look at raw numbers or logs, which is slow and confusing. This integration makes it simple to spot issues before they become big problems, improving system reliability and user experience. It saves time and reduces downtime, which is critical for businesses that depend on their software running smoothly.

Where it fits

Before learning this, you should understand basic monitoring concepts and how data collection works. After this, you can explore alerting systems that notify teams when problems happen and advanced dashboard customization for better insights.

Mental Model

Core Idea

Prometheus collects performance data, and Grafana turns that data into clear, visual dashboards for easy monitoring.

Think of it like...

It's like having a weather station (Prometheus) that measures temperature and wind, and a TV weather channel (Grafana) that shows you the forecast with maps and graphs.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   Prometheus  │─────▶│   Data Store  │─────▶│    Grafana    │
│ (Data Source) │      │ (Time Series) │      │ (Dashboard)   │
└───────────────┘      └───────────────┘      └───────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding Prometheus Basics

Concept: Learn what Prometheus does and how it collects data.

Prometheus works by asking software or systems for their current status at regular times. It stores this information as numbers over time, called time series data. For example, it might record how many messages Kafka processes every second.

Result

You know that Prometheus gathers data by regularly checking systems and saves it for later use.

Understanding that Prometheus actively pulls data helps you see why systems need to expose their status in a way Prometheus can read.

2

Foundation

Getting to Know Grafana Dashboards

Concept: Learn how Grafana shows data visually.

Grafana connects to data sources like Prometheus and lets you create charts and graphs. You can build dashboards that update live, showing trends and current values. For example, you can see Kafka's message rate as a line graph.

Result

You understand that Grafana turns numbers into pictures that are easier to understand.

Knowing that Grafana is a visualization tool clarifies why it needs data from somewhere else like Prometheus.

3

Intermediate

Configuring Prometheus to Monitor Kafka

🤔Before reading on: do you think Prometheus needs special setup to monitor Kafka metrics? Commit to your answer.

Concept: Learn how to set up Prometheus to collect Kafka's performance data.

Kafka exposes metrics using a tool called JMX exporter. You install this exporter on Kafka servers, which makes Kafka's data available in a format Prometheus understands. Then, you add Kafka's address to Prometheus's configuration file under 'scrape_configs' so Prometheus knows where to collect data.

Result

Prometheus starts collecting Kafka metrics like message throughput and consumer lag.

Understanding that Prometheus needs exporters to read Kafka's data explains why monitoring setup involves multiple tools working together.

4

Intermediate

Connecting Grafana to Prometheus Data

🤔Before reading on: do you think Grafana can automatically find Prometheus data sources or do you need to configure it manually? Commit to your answer.

Concept: Learn how to link Grafana with Prometheus to visualize data.

In Grafana, you add Prometheus as a data source by entering its URL and testing the connection. Once connected, you can create dashboards using Prometheus queries to select which Kafka metrics to display. Grafana updates these dashboards live as Prometheus collects new data.

Result

You can see Kafka metrics in Grafana dashboards with live updates.

Knowing that Grafana queries Prometheus directly helps you understand how data flows from collection to visualization.

5

Advanced

Creating Custom Kafka Dashboards in Grafana

🤔Before reading on: do you think you need to write complex queries to build useful dashboards or can simple queries suffice? Commit to your answer.

Concept: Learn how to write Prometheus queries to build meaningful Kafka dashboards.

Grafana uses PromQL, a query language, to select and manipulate data. For Kafka, you might write queries to show message rates, consumer lag, or partition status. You can combine multiple queries in one dashboard and use alerts to notify when metrics cross thresholds.

Result

You create dashboards that give clear insights into Kafka's health and performance.

Understanding PromQL empowers you to tailor dashboards to exactly what your team needs to monitor.

6

Expert

Optimizing and Scaling Monitoring Setup

🤔Before reading on: do you think monitoring many Kafka clusters requires separate Prometheus servers or can one handle all? Commit to your answer.

Concept: Learn best practices for scaling Prometheus and Grafana with Kafka in production.

For large Kafka deployments, you might run multiple Prometheus instances to reduce load and improve reliability. You can use federation to aggregate data from several Prometheus servers. Grafana supports multiple data sources and can combine data from different Prometheus servers. Also, use recording rules in Prometheus to precompute expensive queries for faster dashboards.

Result

Your monitoring system remains fast and reliable even as Kafka grows.

Knowing how to scale monitoring prevents slow dashboards and missed alerts in real-world systems.

Under the Hood

Prometheus works by scraping HTTP endpoints that expose metrics in a specific format. Kafka does not expose metrics directly, so a JMX exporter acts as a bridge, converting Kafka's Java Management Extensions (JMX) data into Prometheus format. Prometheus stores this data as time series in its database. Grafana queries this database using PromQL and renders the results as visual panels on dashboards.

Why designed this way?

Prometheus was designed for reliability and simplicity, pulling data instead of waiting for it to be pushed. This reduces complexity and improves fault tolerance. Grafana was built separately to focus on visualization, allowing it to support many data sources beyond Prometheus. This separation of concerns makes the system flexible and modular.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│   Kafka JVM   │─────▶│  JMX Exporter │─────▶│  Prometheus   │─────▶│   Grafana     │
│ (Metrics Data)│      │ (Metrics HTTP)│      │ (Data Storage)│      │ (Dashboard UI)│
└───────────────┘      └───────────────┘      └───────────────┘      └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Prometheus push data to Grafana automatically? Commit to yes or no.

Common Belief: Prometheus sends data to Grafana by pushing it whenever new data arrives.

Reality: Prometheus stores data and Grafana pulls data by querying Prometheus when rendering dashboards.

Why it matters: Thinking Prometheus pushes data can lead to confusion about how to troubleshoot data delays or dashboard refresh issues.

Quick: Can Grafana collect metrics directly from Kafka without Prometheus? Commit to yes or no.

Common Belief: Grafana can directly collect Kafka metrics without needing Prometheus or exporters.

Reality: Grafana only visualizes data; it cannot collect metrics. Prometheus and exporters are needed to gather Kafka metrics.

Why it matters: Believing Grafana collects data leads to missing the necessary setup steps and monitoring gaps.

Quick: Is it okay to monitor Kafka with Prometheus without using JMX exporters? Commit to yes or no.

Common Belief: Prometheus can scrape Kafka metrics directly without any exporters.

Reality: Kafka metrics are exposed via JMX, so a JMX exporter is required to convert them into Prometheus format.

Why it matters: Skipping the exporter causes no metrics to be collected, leaving Kafka unmonitored.

Quick: Does adding more dashboards in Grafana slow down Prometheus data collection? Commit to yes or no.

Common Belief: More Grafana dashboards directly slow down Prometheus's ability to collect data.

Reality: Grafana queries Prometheus but does not affect how Prometheus scrapes data from targets.

Why it matters: Misunderstanding this can lead to unnecessary changes in Prometheus scraping intervals instead of optimizing Grafana queries.

Expert Zone

1

Prometheus's pull model allows it to detect when a target is down by missing scrapes, which is harder with push models.

2

Grafana's support for templating lets you build dynamic dashboards that adjust to different Kafka clusters or topics without rewriting queries.

3

Recording rules in Prometheus can precompute complex queries, reducing load and improving Grafana dashboard responsiveness.

When NOT to use

For very high-frequency metrics or logs, specialized tools like Kafka's own monitoring or ELK stack might be better. Also, if you need real-time alerting with minimal delay, consider combining Prometheus with alert managers or other streaming analytics tools.

Production Patterns

In production, teams often run Prometheus in a highly available setup with multiple instances and use federation to aggregate data. Grafana dashboards are shared via version control and automated deployment. Alertmanager integrates with Prometheus to send alerts based on Kafka metrics, enabling proactive incident response.

Connections

Time Series Databases

Prometheus is a type of time series database specialized for monitoring data.

Understanding time series databases helps grasp how Prometheus stores and queries data efficiently over time.

Event-Driven Architecture

Kafka is an event streaming platform whose metrics reflect event flow, which Prometheus monitors.

Knowing event-driven systems clarifies why monitoring message rates and consumer lag is critical for Kafka health.

Human Visual Perception

Grafana leverages how humans recognize patterns visually to make complex data understandable.

Understanding visual perception principles helps design better dashboards that highlight important trends and anomalies.

Common Pitfalls

#1 Not configuring the JMX exporter on Kafka, so Prometheus gets no metrics.

Wrong approach:

prometheus.yml:
scrape_configs:
- job_name: 'kafka'
static_configs:
- targets: ['kafka-server:9090']

Correct approach:

prometheus.yml:
scrape_configs:
- job_name: 'kafka'
static_configs:
- targets: ['kafka-server:5556'] # JMX exporter port

# JMX exporter runs on port 5556 exposing metrics

Root cause:

Assuming Kafka exposes Prometheus metrics directly without the JMX exporter.

#2 Adding Prometheus as a data source in Grafana with the wrong URL or port.

Wrong approach:

Grafana data source URL: http://localhost:1234 (wrong port)

Correct approach:

Grafana data source URL: http://localhost:9090 (default Prometheus port)

Root cause:

Not verifying Prometheus server address and port before configuring Grafana.

#3 Writing PromQL queries without understanding metric names or labels, resulting in empty or wrong graphs.

Wrong approach:

sum(rate(kafka_messages_total[5m]))

Correct approach:

sum(rate(kafka_server_brokertopicmetrics_messagesin_total[5m]))

Root cause:

Using generic or incorrect metric names without checking actual exported metrics.

Key Takeaways

Prometheus collects metrics by regularly scraping targets that expose data in a specific format.

Grafana visualizes data by querying Prometheus and displaying it in customizable dashboards.

Kafka metrics require a JMX exporter to convert Java metrics into Prometheus format for scraping.

Proper configuration and understanding of PromQL are essential to build meaningful monitoring dashboards.

Scaling monitoring setups and using recording rules improve performance and reliability in production.

Practice

1. What is the main role of Prometheus in Kafka monitoring when integrated with Grafana?

easy

A. To collect Kafka metrics by scraping the metrics endpoint

B. To display visual dashboards of Kafka metrics

C. To send alerts directly to Kafka topics

D. To store Kafka messages for backup

Solution

1. Step 1: Understand Prometheus function

   Prometheus collects metrics by scraping endpoints exposed by applications like Kafka.

2. Step 2: Differentiate roles of Prometheus and Grafana

   Grafana is used for visualization, not data collection.

3. Final Answer:

   To collect Kafka metrics by scraping the metrics endpoint -> Option A

4. Quick Check:

   Prometheus collects metrics = A [OK]

Hint: Remember: Prometheus collects, Grafana displays [OK]

Common Mistakes:

• Confusing Grafana as the collector
• Thinking Prometheus stores Kafka messages
• Assuming Prometheus sends alerts to Kafka

2. Which of the following is the correct Prometheus scrape configuration snippet to monitor Kafka metrics exposed at port 9090?

easy

A. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:9090']

B. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:2181']

C. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:8080']

D. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:1234']

Solution

1. Step 1: Identify Kafka metrics port

   Kafka metrics exporter commonly exposes metrics on port 9090.

2. Step 2: Match correct target in Prometheus config

   Only - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:9090'] uses port 9090, which is correct for scraping Kafka metrics.

3. Final Answer:

   - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:9090'] -> Option A

4. Quick Check:

   Port 9090 for Kafka metrics = A [OK]

Hint: Kafka metrics usually on port 9090, check targets carefully [OK]

Common Mistakes:

• Using Kafka broker port 2181 (Zookeeper port) instead of metrics port
• Confusing HTTP ports like 8080 for metrics
• Using random ports not related to Kafka metrics

3. Given this Prometheus query to monitor Kafka consumer lag:
sum(kafka_consumer_group_lag) by (group)
What output will this query produce?

medium

A. Current CPU usage of Kafka brokers

B. Total lag per consumer group as a number

C. Number of active Kafka brokers

D. List of all Kafka topics with their lag

Solution

1. Step 1: Understand the metric and aggregation

   The metric kafka_consumer_group_lag measures lag per consumer group and partition. Summing by group aggregates lag per group.

2. Step 2: Interpret the query output

   The query returns total lag number for each consumer group, not topics or CPU usage.

3. Final Answer:

   Total lag per consumer group as a number -> Option B

4. Quick Check:

   sum by group = total lag per group [OK]

Hint: sum(... by (group)) means total per group [OK]

Common Mistakes:

• Thinking it lists topics instead of groups
• Confusing lag with broker count
• Assuming CPU metrics from Kafka lag metric

4. You configured Prometheus to scrape Kafka metrics but Grafana shows no data. Which is the most likely cause?

medium

A. Prometheus scrape interval is too short

B. Grafana dashboard is not connected to Prometheus

C. Kafka metrics endpoint is not exposed or unreachable

D. Kafka broker is down

Solution

1. Step 1: Check Prometheus scraping setup

   If Kafka metrics endpoint is not exposed or unreachable, Prometheus cannot collect data.

2. Step 2: Understand impact on Grafana

   Without data in Prometheus, Grafana dashboards show no metrics.

3. Final Answer:

   Kafka metrics endpoint is not exposed or unreachable -> Option C

4. Quick Check:

   No data means no scrape target reachable [OK]

Hint: No data? Check if metrics endpoint is reachable first [OK]

Common Mistakes:

• Blaming Grafana connection before checking Prometheus data
• Assuming Kafka broker down always causes no data
• Thinking scrape interval too short causes no data

5. You want to create a Grafana dashboard showing Kafka broker CPU usage and consumer lag side by side. Which data sources and queries should you use?

hard

A. Connect Grafana to Prometheus; use queries kafka_cpu_usage and kafka_topic_lag

B. Connect Grafana directly to Kafka; use queries kafka_cpu and kafka_lag

C. Connect Grafana to Kafka Connect; use queries cpu_usage and consumer_lag

D. Connect Grafana to Prometheus; use queries kafka_broker_cpu_usage and sum(kafka_consumer_group_lag) by (group)

Solution

1. Step 1: Identify correct data source for metrics

   Prometheus collects Kafka metrics; Grafana connects to Prometheus for queries.

2. Step 2: Choose correct metric names and queries

   Common metrics are kafka_broker_cpu_usage and kafka_consumer_group_lag aggregated by group.

3. Final Answer:

   Connect Grafana to Prometheus; use queries kafka_broker_cpu_usage and sum(kafka_consumer_group_lag) by (group) -> Option D

4. Quick Check:

   Prometheus + correct queries = C [OK]

Hint: Grafana queries Prometheus metrics, not Kafka directly [OK]

Common Mistakes:

• Trying to connect Grafana directly to Kafka brokers
• Using wrong metric names or unaggregated lag
• Confusing Kafka Connect with Kafka broker metrics