How to Monitor Consumer Lag in Kafka: Simple Guide

To monitor consumer lag in Kafka, use the kafka-consumer-groups.sh command with the --describe option to see the difference between the latest offset and the consumer's committed offset. This lag shows how far behind a consumer is from the producer's latest messages.

Syntax

The main command to check consumer lag is kafka-consumer-groups.sh --describe. It shows details about consumer groups, partitions, current offsets, log end offsets, and lag.

• --bootstrap-server <broker>: Kafka broker address
• --group <group_id>: Consumer group to check
• --describe: Show detailed info including lag

bash

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-consumer-group --describe

Example

This example shows how to check lag for a consumer group named my-consumer-group connected to a Kafka broker on localhost.

bash

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-consumer-group --describe

Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
my-topic       0          105             110             5               consumer-1-12345                               /127.0.0.1      consumer-1
my-topic       1          200             200             0               consumer-1-12345                               /127.0.0.1      consumer-1

Common Pitfalls

Common mistakes when monitoring consumer lag include:

• Using the wrong --group name, which shows no data.
• Checking lag on the wrong Kafka broker or cluster.
• Not having consumer offsets committed, so lag appears incorrect.
• Confusing CURRENT-OFFSET with LOG-END-OFFSET — lag is the difference.

Always ensure your consumers commit offsets and you use the correct group and broker.

bash

## Wrong group example (no output)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group wrong-group --describe

## Correct group example
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-consumer-group --describe

Output

Error: Group 'wrong-group' does not exist or has no active members.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
my-topic       0          105             110             5               consumer-1-12345                               /127.0.0.1      consumer-1

Quick Reference

Tips for effective consumer lag monitoring:

• Run kafka-consumer-groups.sh --describe regularly to track lag.
• Use monitoring tools like Kafka Manager or Burrow for automated alerts.
• Lag > 0 means consumers are behind; investigate slow consumers or processing issues.
• Ensure consumers commit offsets properly to get accurate lag data.

Key Takeaways

Use kafka-consumer-groups.sh --describe with the correct group to see consumer lag.

Lag is the difference between log end offset and current consumer offset.

Ensure consumers commit offsets to get accurate lag information.

Monitor lag regularly to detect slow consumers and avoid data backlog.