Offset management in Kafka - Commands & Configuration

Introduction

When you read messages from Kafka, you need to remember where you left off. Offset management helps track this position so you don't read the same message twice or miss any messages.

When you want to process messages from a Kafka topic without losing or repeating data.

When your consumer application crashes and you want to resume reading from the last processed message.

When multiple consumers share the same group and you want to balance message processing.

When you want to manually control which messages your consumer reads next.

When you want to reset or rewind your consumer to reprocess messages.

Commands

This command shows the current offset positions for each partition in the consumer group 'my-group'. It helps you see where your consumers are reading from.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --describe

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

my-topic       0          15              20              5               consumer-1-1234abcd-5678-90ef-ghij-klmnopqrstuv /127.0.0.1      consumer-1

→

--bootstrap-server - Specifies the Kafka server address to connect to.

→

--group - Specifies the consumer group to inspect.

→

--describe - Shows detailed offset and lag information.

This command resets the offsets for 'my-group' on 'my-topic' to the earliest available messages, so the consumer will reprocess all messages from the start.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --reset-offsets --to-earliest --topic my-topic --execute

Expected Output

Offsets reset to earliest for group my-group on topic my-topic.



Consumer group offsets have been reset successfully.

→

--reset-offsets - Starts the offset reset operation.

→

--to-earliest - Resets offsets to the earliest message available.

→

--execute - Applies the reset changes. Without this, the command only shows what would happen.

After resetting offsets, this command verifies the new offset positions to confirm the reset worked.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --describe

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

my-topic       0          0               20              20              -                                           -               -

→

--bootstrap-server - Specifies the Kafka server address to connect to.

→

--group - Specifies the consumer group to inspect.

→

--describe - Shows detailed offset and lag information.

Key Concept

If you remember nothing else from this pattern, remember: offsets tell your consumer where to start reading next to avoid missing or repeating messages.

Common Mistakes

Not using the --execute flag when resetting offsets.

Without --execute, the reset command only simulates the change and does not apply it.

Always add --execute to apply offset reset changes.

Resetting offsets without stopping the consumer group.

Active consumers may interfere with offset reset, causing inconsistent behavior.

Stop consumers in the group before resetting offsets, then restart them.

Confusing consumer group offsets with topic log end offsets.

Consumer offsets track reading position; log end offsets show total messages. Mixing them leads to wrong assumptions about lag.

Use the describe command to see both and understand lag as the difference.

Summary

Use kafka-consumer-groups --describe to check current offsets and lag for a consumer group.

Use kafka-consumer-groups --reset-offsets with --execute to change where a consumer group starts reading.

Always verify offset changes by describing the consumer group again after reset.