Prometheus and Grafana integration in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Prometheus scrape configuration output

Given this Prometheus scrape config snippet, what will be the value of the 'job' label for metrics scraped from Kafka exporters?

Kafka

scrape_configs:
  - job_name: 'kafka'
    static_configs:
      - targets: ['localhost:9308']

A kafka

B localhost

C exporter

D 9308

💡 Hint

Look at the job_name field in the scrape_configs section.

The 'job' label in Prometheus metrics is set from the 'job_name' field in the scrape_configs. Here, it is 'kafka'.

🔀 Workflow

intermediate

Steps to visualize Kafka metrics in Grafana

Which sequence correctly describes the steps to visualize Kafka metrics in Grafana after Prometheus is scraping Kafka exporter metrics?

A 1,4,2,3

B 4,1,2,3

C 1,2,4,3

D 4,2,1,3

💡 Hint

Start by ensuring Prometheus is scraping metrics before adding Grafana data source.

First verify Prometheus scrapes metrics (4), then add Prometheus as Grafana data source (1), create dashboard (2), then add panels (3).

❓ Troubleshoot

advanced

Prometheus not scraping Kafka exporter metrics

Prometheus is not collecting metrics from Kafka exporter at localhost:9308. Which configuration mistake causes this?

Kafka

scrape_configs:
  - job_name: 'kafka'
    static_configs:
      - targets: ['localhost:9307']

A Target port is incorrect; should be 9308 not 9307.

B Job name must be 'kafka_exporter' instead of 'kafka'.

C Static configs must use IP address, not 'localhost'.

D Missing scheme: 'http://' before target address.

💡 Hint

Check the port number in the targets list.

The Kafka exporter runs on port 9308 by default. Using 9307 causes Prometheus to fail scraping.

🧠 Conceptual

advanced

Grafana alerting on Kafka lag

Which Prometheus query expression correctly triggers an alert when Kafka consumer lag exceeds 1000 messages?

A avg(kafka_consumergroup_lag) > 1000

B max(kafka_consumergroup_lag) < 1000

C sum(kafka_consumergroup_lag) by (group) > 1000

D kafka_consumergroup_lag > 1000

💡 Hint

Consider grouping lag by consumer group to alert per group.

Summing lag by group and checking if it exceeds 1000 triggers alert for groups with high lag.

✅ Best Practice

expert

Optimizing Prometheus scrape intervals for Kafka exporters

What is the best practice for setting Prometheus scrape interval for Kafka exporters to balance data freshness and system load?

A Set scrape_interval to 5m to reduce load on Kafka exporters.

B Set scrape_interval to 1s for maximum data freshness.

C Set scrape_interval to 1h to minimize Prometheus storage usage.

D Set scrape_interval to 15s to get near real-time metrics without high load.

💡 Hint

Consider typical Kafka metric update frequency and system resource usage.

A 15-second interval balances timely metrics and reasonable load on exporters and Prometheus.

Practice

1. What is the main role of Prometheus in Kafka monitoring when integrated with Grafana?

easy

A. To collect Kafka metrics by scraping the metrics endpoint

B. To display visual dashboards of Kafka metrics

C. To send alerts directly to Kafka topics

D. To store Kafka messages for backup

Solution

1. Step 1: Understand Prometheus function

   Prometheus collects metrics by scraping endpoints exposed by applications like Kafka.

2. Step 2: Differentiate roles of Prometheus and Grafana

   Grafana is used for visualization, not data collection.

3. Final Answer:

   To collect Kafka metrics by scraping the metrics endpoint -> Option A

4. Quick Check:

   Prometheus collects metrics = A [OK]

Hint: Remember: Prometheus collects, Grafana displays [OK]

Common Mistakes:

• Confusing Grafana as the collector
• Thinking Prometheus stores Kafka messages
• Assuming Prometheus sends alerts to Kafka

2. Which of the following is the correct Prometheus scrape configuration snippet to monitor Kafka metrics exposed at port 9090?

easy

A. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:9090']

B. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:2181']

C. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:8080']

D. - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:1234']

Solution

1. Step 1: Identify Kafka metrics port

   Kafka metrics exporter commonly exposes metrics on port 9090.

2. Step 2: Match correct target in Prometheus config

   Only - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:9090'] uses port 9090, which is correct for scraping Kafka metrics.

3. Final Answer:

   - job_name: 'kafka'\n static_configs:\n - targets: ['localhost:9090'] -> Option A

4. Quick Check:

   Port 9090 for Kafka metrics = A [OK]

Hint: Kafka metrics usually on port 9090, check targets carefully [OK]

Common Mistakes:

• Using Kafka broker port 2181 (Zookeeper port) instead of metrics port
• Confusing HTTP ports like 8080 for metrics
• Using random ports not related to Kafka metrics

3. Given this Prometheus query to monitor Kafka consumer lag:
sum(kafka_consumer_group_lag) by (group)
What output will this query produce?

medium

A. Current CPU usage of Kafka brokers

B. Total lag per consumer group as a number

C. Number of active Kafka brokers

D. List of all Kafka topics with their lag

Solution

1. Step 1: Understand the metric and aggregation

   The metric kafka_consumer_group_lag measures lag per consumer group and partition. Summing by group aggregates lag per group.

2. Step 2: Interpret the query output

   The query returns total lag number for each consumer group, not topics or CPU usage.

3. Final Answer:

   Total lag per consumer group as a number -> Option B

4. Quick Check:

   sum by group = total lag per group [OK]

Hint: sum(... by (group)) means total per group [OK]

Common Mistakes:

• Thinking it lists topics instead of groups
• Confusing lag with broker count
• Assuming CPU metrics from Kafka lag metric

4. You configured Prometheus to scrape Kafka metrics but Grafana shows no data. Which is the most likely cause?

medium

A. Prometheus scrape interval is too short

B. Grafana dashboard is not connected to Prometheus

C. Kafka metrics endpoint is not exposed or unreachable

D. Kafka broker is down

Solution

1. Step 1: Check Prometheus scraping setup

   If Kafka metrics endpoint is not exposed or unreachable, Prometheus cannot collect data.

2. Step 2: Understand impact on Grafana

   Without data in Prometheus, Grafana dashboards show no metrics.

3. Final Answer:

   Kafka metrics endpoint is not exposed or unreachable -> Option C

4. Quick Check:

   No data means no scrape target reachable [OK]

Hint: No data? Check if metrics endpoint is reachable first [OK]

Common Mistakes:

• Blaming Grafana connection before checking Prometheus data
• Assuming Kafka broker down always causes no data
• Thinking scrape interval too short causes no data

5. You want to create a Grafana dashboard showing Kafka broker CPU usage and consumer lag side by side. Which data sources and queries should you use?

hard

A. Connect Grafana to Prometheus; use queries kafka_cpu_usage and kafka_topic_lag

B. Connect Grafana directly to Kafka; use queries kafka_cpu and kafka_lag

C. Connect Grafana to Kafka Connect; use queries cpu_usage and consumer_lag

D. Connect Grafana to Prometheus; use queries kafka_broker_cpu_usage and sum(kafka_consumer_group_lag) by (group)

Solution

1. Step 1: Identify correct data source for metrics

   Prometheus collects Kafka metrics; Grafana connects to Prometheus for queries.

2. Step 2: Choose correct metric names and queries

   Common metrics are kafka_broker_cpu_usage and kafka_consumer_group_lag aggregated by group.

3. Final Answer:

   Connect Grafana to Prometheus; use queries kafka_broker_cpu_usage and sum(kafka_consumer_group_lag) by (group) -> Option D

4. Quick Check:

   Prometheus + correct queries = C [OK]

Hint: Grafana queries Prometheus metrics, not Kafka directly [OK]

Common Mistakes:

• Trying to connect Grafana directly to Kafka brokers
• Using wrong metric names or unaggregated lag
• Confusing Kafka Connect with Kafka broker metrics