Consumer lag monitoring in Kafka - Mini Project: Build & Apply

Consumer lag monitoring

📖 Scenario: You work in a team managing a Kafka messaging system. To keep the system healthy, you need to monitor how far behind each consumer group is from the latest messages. This delay is called consumer lag. Monitoring lag helps prevent message loss and delays in processing.

🎯 Goal: Build a simple script that tracks consumer lag for a Kafka topic by comparing the latest message offsets with the consumer group's committed offsets.

📋 What You'll Learn

Create a dictionary with Kafka topic partitions and their latest offsets

Create a dictionary with consumer group committed offsets for each partition

Calculate the lag for each partition by subtracting committed offset from latest offset

Print the lag for each partition clearly

💡 Why This Matters

🌍 Real World

Monitoring consumer lag is critical in real Kafka deployments to ensure timely processing of messages and to avoid data loss or delays.

💼 Career

Understanding consumer lag and how to monitor it is a key skill for DevOps engineers and site reliability engineers working with Kafka.

1

Create latest offsets data

Create a dictionary called latest_offsets with these exact entries: 0: 150, 1: 200, 2: 180. This represents the latest message offset for each partition of the Kafka topic.

Kafka

# Create the latest_offsets dictionary with partition keys and offset values
# Your code here

Need a hint?

Use curly braces to create a dictionary with partition numbers as keys and offsets as values.

2

Create consumer group committed offsets

Create a dictionary called committed_offsets with these exact entries: 0: 145, 1: 198, 2: 175. This represents the last processed offset by the consumer group for each partition.

Kafka

latest_offsets = {0: 150, 1: 200, 2: 180}
# Create the committed_offsets dictionary with partition keys and committed offset values
# Your code here

Need a hint?

Use a dictionary similar to latest_offsets but with committed offset values.

3

Calculate consumer lag per partition

Create a dictionary called consumer_lag that calculates lag for each partition by subtracting the committed offset from the latest offset using a for loop with variables partition and latest iterating over latest_offsets.items().

Kafka

latest_offsets = {0: 150, 1: 200, 2: 180}
committed_offsets = {0: 145, 1: 198, 2: 175}
# Calculate consumer lag for each partition
# Your code here

Need a hint?

Use a for loop to go through each partition and calculate lag by subtracting committed offset from latest offset.

4

Print the consumer lag

Use a for loop with variables partition and lag iterating over consumer_lag.items() to print the lag in the format: Partition {partition} lag: {lag}.

Kafka

latest_offsets = {0: 150, 1: 200, 2: 180}
committed_offsets = {0: 145, 1: 198, 2: 175}
consumer_lag = {}
for partition, latest in latest_offsets.items():
    consumer_lag[partition] = latest - committed_offsets[partition]
# Print the lag for each partition
# Your code here

Need a hint?

Use a for loop to print each partition's lag with the exact format.