Prometheus and Grafana integration in Kafka - Mini Project: Build & Apply

Prometheus and Grafana Integration with Kafka Metrics

📖 Scenario: You are working in a team that manages Kafka clusters. Your team wants to monitor Kafka metrics to ensure the system runs smoothly. You will set up Prometheus to scrape Kafka metrics and configure Grafana to visualize these metrics.

🎯 Goal: Build a simple Kafka metrics monitoring setup by configuring Prometheus to scrape Kafka JMX exporter metrics and create a Grafana dashboard to visualize the metrics.

📋 What You'll Learn

Create a Prometheus scrape configuration for Kafka JMX exporter

Define a Kafka JMX exporter endpoint variable

Write the Prometheus scrape job configuration for Kafka metrics

Print the final Prometheus scrape configuration

💡 Why This Matters

🌍 Real World

Monitoring Kafka clusters is essential to detect issues early and maintain system health. Prometheus and Grafana are popular tools for collecting and visualizing these metrics.

💼 Career

DevOps engineers and site reliability engineers often configure Prometheus and Grafana to monitor Kafka and other services to ensure uptime and performance.

1

Create Kafka JMX Exporter Endpoint Variable

Create a variable called kafka_jmx_exporter_endpoint and set it to the string "localhost:9404" which is the Kafka JMX exporter metrics endpoint host and port.

Kafka

# Your code here

Need a hint?

This variable holds the host and port where Prometheus will scrape Kafka metrics.

2

Define Prometheus Scrape Job Configuration

Create a variable called prometheus_scrape_config and assign it a dictionary with a key scrape_configs that contains a list with one dictionary. This dictionary should have job_name set to "kafka" and static_configs set to a list with one dictionary containing targets set to a list with the kafka_jmx_exporter_endpoint variable.

Kafka

kafka_jmx_exporter_endpoint = "localhost:9404"
# Your code here

Need a hint?

This dictionary configures Prometheus to scrape metrics from the Kafka JMX exporter endpoint.

3

Add Prometheus Scrape Interval Configuration

Add a key scrape_interval with the value "15s" to the prometheus_scrape_config dictionary to set the scrape interval to 15 seconds.

Kafka

kafka_jmx_exporter_endpoint = "localhost:9404"
prometheus_scrape_config = {
    "scrape_configs": [
        {
            "job_name": "kafka",
            "static_configs": [
                {"targets": [kafka_jmx_exporter_endpoint]}
            ]
        }
    ]
}
# Your code here

Need a hint?

The scrape interval tells Prometheus how often to collect metrics.

4

Print the Prometheus Scrape Configuration

Write a print statement to display the prometheus_scrape_config dictionary.

Kafka

kafka_jmx_exporter_endpoint = "localhost:9404"
prometheus_scrape_config = {
    "scrape_interval": "15s",
    "scrape_configs": [
        {
            "job_name": "kafka",
            "static_configs": [
                {"targets": [kafka_jmx_exporter_endpoint]}
            ]
        }
    ]
}
# Your code here

Need a hint?

Printing the configuration helps verify it is correct before using it in Prometheus.