Consumer lag monitoring in Kafka - Step-by-Step Execution

Process Flow - Consumer lag monitoring

Kafka Topic Produces Messages

↓

Consumer Reads Messages

↓

Calculate Consumer Offset

↓

Calculate Latest Topic Offset

↓

Compute Lag = Latest Offset - Consumer Offset

↓

Alert or Visualize Lag if Above Threshold

↓

Repeat Monitoring

This flow shows how consumer lag is monitored by comparing the latest message offset with the consumer's current offset, then alerting if lag is too high.

Execution Sample

Kafka

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group

This command shows the current consumer offsets and lag for the group 'my-group'.

Process Table

Step	Action	Latest Offset	Consumer Offset	Lag (Latest - Consumer)	Result
1	Fetch latest topic offset	1500	-	-	Latest offset is 1500
2	Fetch consumer offset for partition 0	-	1480	-	Consumer offset is 1480
3	Calculate lag	1500	1480	20	Lag is 20 messages
4	Compare lag with threshold (10)	1500	1480	20	Lag exceeds threshold, alert triggered
5	Fetch consumer offset for partition 1	-	1500	-	Consumer offset is 1500
6	Calculate lag	1500	1500	0	Lag is 0 messages
7	Compare lag with threshold (10)	1500	1500	0	Lag below threshold, no alert
8	End monitoring cycle	-	-	-	Monitoring cycle complete

💡 Monitoring cycle ends after checking all partitions and comparing lag to threshold.

Status Tracker

Variable	Start	After Step 2	After Step 5	Final
Latest Offset	-	1500	1500	1500
Consumer Offset Partition 0	-	1480	1480	1480
Consumer Offset Partition 1	-	-	1500	1500
Lag Partition 0	-	20	20	20
Lag Partition 1	-	-	0	0

Key Moments - 3 Insights

Why is lag calculated as Latest Offset minus Consumer Offset?

Lag shows how many messages the consumer has not processed yet. The execution_table rows 3 and 6 show lag calculation as the difference between latest and consumer offsets.

What happens if lag is zero?

If lag is zero, the consumer is caught up with the topic. Rows 6 and 7 show lag zero leads to no alert.

Why do we check lag for each partition separately?

Each partition has its own offset and lag. Rows 2-7 show separate calculations per partition to monitor all parts of the topic.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what is the lag value?

A 20

B 1480

C 1500

D 0

💡 Hint

Check the 'Lag (Latest - Consumer)' column at step 3 in the execution_table.

At which step does the lag exceed the threshold and trigger an alert?

A Step 7

B Step 6

C Step 4

D Step 2

💡 Hint

Look at the 'Result' column for alert messages in the execution_table.

If the consumer offset for partition 0 was 1495 instead of 1480, what would the lag be at step 3?

A 15

B 5

C 20

D 0

💡 Hint

Lag = Latest Offset - Consumer Offset; see variable_tracker and execution_table step 3.

Concept Snapshot

Consumer lag monitoring compares the latest message offset in a Kafka topic with the consumer's current offset.
Lag = Latest Offset - Consumer Offset.
Lag per partition is checked separately.
Alerts trigger if lag exceeds a set threshold.
Use 'kafka-consumer-groups --describe' to view offsets and lag.
Regular monitoring helps ensure consumers keep up with message flow.

Full Transcript

Consumer lag monitoring in Kafka involves checking how far behind a consumer group is from the latest messages in a topic. The process starts by fetching the latest offset for each partition of the topic. Then, the consumer's current offset for each partition is retrieved. Lag is calculated by subtracting the consumer offset from the latest offset. If the lag exceeds a predefined threshold, an alert is triggered to notify that the consumer is falling behind. This check is repeated for all partitions. The command 'kafka-consumer-groups --describe' helps visualize these offsets and lag values. Monitoring lag helps maintain smooth data processing and avoid delays.

Practice

1. What does consumer lag in Kafka indicate?

easy

A. The delay between the latest message produced and the message consumed

B. The number of consumers in a consumer group

C. The total number of messages produced to a topic

D. The size of the Kafka topic in bytes

Solution

1. Step 1: Understand consumer lag meaning

   Consumer lag measures how far behind a consumer is from the latest message produced in the topic partition.

2. Step 2: Differentiate from other metrics

   It is not about consumer count, total messages, or topic size, but about delay in consumption.

3. Final Answer:

   The delay between the latest message produced and the message consumed -> Option A

4. Quick Check:

   Consumer lag = delay behind latest message [OK]

Hint: Lag means delay between produced and consumed messages [OK]

Common Mistakes:

• Confusing lag with consumer count
• Thinking lag is topic size
• Mixing lag with total messages produced

2. Which command correctly shows consumer lag for a group named my-group?

easy

A. kafka-consumer-groups.sh --describe --group my-group

B. kafka-consumer-groups.sh --list --group my-group

C. kafka-topics.sh --describe --group my-group

D. kafka-producer-perf-test.sh --group my-group

Solution

1. Step 1: Identify correct command for consumer lag

   The command to describe consumer group lag is kafka-consumer-groups.sh --describe --group <group-name>.

2. Step 2: Eliminate incorrect commands

   --list lists groups but does not show lag; kafka-topics.sh is for topics, not groups; kafka-producer-perf-test.sh is unrelated.

3. Final Answer:

   kafka-consumer-groups.sh --describe --group my-group -> Option A

4. Quick Check:

   Describe group shows lag [OK]

Hint: Use --describe with --group to see lag [OK]

Common Mistakes:

• Using --list instead of --describe
• Confusing topic commands with consumer group commands
• Using producer commands for consumer lag

3. Given this output from kafka-consumer-groups.sh --describe --group my-group:

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID
my-topic       0          100             105             5    consumer-1
my-topic       1          200             200             0    consumer-1

What is the total lag for the consumer group?

medium

A. 105

B. 5

C. 0

D. 205

Solution

1. Step 1: Identify lag per partition

   Partition 0 lag is 5, partition 1 lag is 0.

2. Step 2: Calculate total lag

   Total lag = 5 + 0 = 5 messages behind.

3. Final Answer:

   5 -> Option B

4. Quick Check:

   Total lag = sum of partition lags = 5 [OK]

Hint: Add lag values from all partitions for total lag [OK]

Common Mistakes:

• Using LOG-END-OFFSET as lag
• Confusing CURRENT-OFFSET with lag
• Adding offsets instead of lag values

4. You run kafka-consumer-groups.sh --describe --group my-group but see no lag column. What is the likely cause?

medium

A. The Kafka broker is down

B. The topic does not exist

C. The command syntax is incorrect

D. The consumer group has no committed offsets yet

Solution

1. Step 1: Understand missing lag column reason

   If the consumer group has no committed offsets, lag cannot be calculated and may not show.

2. Step 2: Exclude other causes

   Broker down or wrong syntax usually cause errors, not missing lag column; topic existence affects consumption but not lag display here.

3. Final Answer:

   The consumer group has no committed offsets yet -> Option D

4. Quick Check:

   No committed offsets = no lag shown [OK]

Hint: No committed offsets means lag won't appear [OK]

Common Mistakes:

• Assuming broker down causes missing lag column
• Thinking syntax error causes missing lag column
• Confusing topic existence with lag display

5. You notice high consumer lag for a group processing a topic with 3 partitions. The lag per partition is 100, 0, and 50 respectively. What is the best approach to reduce lag quickly?

hard

A. Delete and recreate the consumer group

B. Increase the topic partition count to 10

C. Add more consumers to the group to increase parallel processing

D. Restart the Kafka broker to clear lag

Solution

1. Step 1: Analyze lag distribution

   Partitions have uneven lag: 100, 0, 50. Some partitions are behind.

2. Step 2: Choose action to reduce lag

   Adding consumers can help process partitions faster by parallel consumption.

3. Step 3: Evaluate other options

   Increasing partitions alone won't reduce lag immediately; restarting broker or deleting group won't fix lag directly.

4. Final Answer:

   Add more consumers to the group to increase parallel processing -> Option C

5. Quick Check:

   More consumers = faster lag reduction [OK]

Hint: Add consumers to process partitions faster [OK]

Common Mistakes:

• Thinking restarting broker clears lag
• Believing increasing partitions instantly reduces lag
• Deleting group removes offsets but not lag solution