Consumer lag monitoring in Kafka - Commands & Configuration

Introduction

When you use Kafka to send messages, consumers read these messages. Consumer lag monitoring helps you see if consumers are falling behind and not reading messages fast enough. This prevents delays and data loss.

When you want to check if your app reading messages from Kafka is keeping up with new messages.

When you notice delays in processing data from Kafka topics and want to find the cause.

When you want to alert your team if consumers stop reading messages for too long.

When you deploy a new consumer and want to verify it catches up with existing messages.

When you want to optimize consumer performance by understanding lag patterns.

Commands

This command shows the current lag of the consumer group named 'my-consumer-group'. It tells how many messages behind the consumer is for each partition.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-consumer-group

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                    HOST            CLIENT-ID

my-topic       0          105             110             5    consumer-1-12345-67890                        /127.0.0.1      consumer-1

→

--bootstrap-server - Specifies the Kafka server address to connect to.

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies which consumer group to check.

This command lists all consumer groups in the Kafka cluster so you can find the group name to monitor.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --list

Expected Output

my-consumer-group

another-group

analytics-group

→

--bootstrap-server - Specifies the Kafka server address to connect to.

→

--list - Lists all consumer groups.

Key Concept

If you remember nothing else from this pattern, remember: consumer lag shows how far behind your app is in reading messages, so monitoring it helps keep data fresh and processing timely.

Common Mistakes

Using the wrong consumer group name in the command.

The command will show no data or errors because it cannot find that group.

Use 'kafka-consumer-groups --list' first to find the exact group name.

Checking lag without specifying the correct Kafka server address.

The command cannot connect and will fail or show no output.

Always use the correct --bootstrap-server flag with your Kafka server address.

Summary

Use 'kafka-consumer-groups --list' to find consumer groups in your Kafka cluster.

Use 'kafka-consumer-groups --describe --group <group-name>' to see lag details for a consumer group.

Monitoring consumer lag helps ensure your consumers keep up with message production and avoid delays.