Consumer lag monitoring in Kafka - Step-by-Step Execution

Process Flow - Consumer lag monitoring

Kafka Topic Produces Messages

↓

Consumer Reads Messages

↓

Calculate Consumer Offset

↓

Calculate Latest Topic Offset

↓

Compute Lag = Latest Offset - Consumer Offset

↓

Alert or Visualize Lag if Above Threshold

↓

Repeat Monitoring

This flow shows how consumer lag is monitored by comparing the latest message offset with the consumer's current offset, then alerting if lag is too high.

Execution Sample

Kafka

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group

This command shows the current consumer offsets and lag for the group 'my-group'.

Process Table

Step	Action	Latest Offset	Consumer Offset	Lag (Latest - Consumer)	Result
1	Fetch latest topic offset	1500	-	-	Latest offset is 1500
2	Fetch consumer offset for partition 0	-	1480	-	Consumer offset is 1480
3	Calculate lag	1500	1480	20	Lag is 20 messages
4	Compare lag with threshold (10)	1500	1480	20	Lag exceeds threshold, alert triggered
5	Fetch consumer offset for partition 1	-	1500	-	Consumer offset is 1500
6	Calculate lag	1500	1500	0	Lag is 0 messages
7	Compare lag with threshold (10)	1500	1500	0	Lag below threshold, no alert
8	End monitoring cycle	-	-	-	Monitoring cycle complete

💡 Monitoring cycle ends after checking all partitions and comparing lag to threshold.

Status Tracker

Variable	Start	After Step 2	After Step 5	Final
Latest Offset	-	1500	1500	1500
Consumer Offset Partition 0	-	1480	1480	1480
Consumer Offset Partition 1	-	-	1500	1500
Lag Partition 0	-	20	20	20
Lag Partition 1	-	-	0	0

Key Moments - 3 Insights

Why is lag calculated as Latest Offset minus Consumer Offset?

Lag shows how many messages the consumer has not processed yet. The execution_table rows 3 and 6 show lag calculation as the difference between latest and consumer offsets.

What happens if lag is zero?

If lag is zero, the consumer is caught up with the topic. Rows 6 and 7 show lag zero leads to no alert.

Why do we check lag for each partition separately?

Each partition has its own offset and lag. Rows 2-7 show separate calculations per partition to monitor all parts of the topic.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what is the lag value?

A 20

B 1480

C 1500

D 0

💡 Hint

Check the 'Lag (Latest - Consumer)' column at step 3 in the execution_table.

At which step does the lag exceed the threshold and trigger an alert?

A Step 7

B Step 6

C Step 4

D Step 2

💡 Hint

Look at the 'Result' column for alert messages in the execution_table.

If the consumer offset for partition 0 was 1495 instead of 1480, what would the lag be at step 3?

A 15

B 5

C 20

D 0

💡 Hint

Lag = Latest Offset - Consumer Offset; see variable_tracker and execution_table step 3.

Concept Snapshot

Consumer lag monitoring compares the latest message offset in a Kafka topic with the consumer's current offset.
Lag = Latest Offset - Consumer Offset.
Lag per partition is checked separately.
Alerts trigger if lag exceeds a set threshold.
Use 'kafka-consumer-groups --describe' to view offsets and lag.
Regular monitoring helps ensure consumers keep up with message flow.

Full Transcript

Consumer lag monitoring in Kafka involves checking how far behind a consumer group is from the latest messages in a topic. The process starts by fetching the latest offset for each partition of the topic. Then, the consumer's current offset for each partition is retrieved. Lag is calculated by subtracting the consumer offset from the latest offset. If the lag exceeds a predefined threshold, an alert is triggered to notify that the consumer is falling behind. This check is repeated for all partitions. The command 'kafka-consumer-groups --describe' helps visualize these offsets and lag values. Monitoring lag helps maintain smooth data processing and avoid delays.