Group coordinator in Kafka - Commands & Configuration

Introduction

When multiple consumers read messages from Kafka, they form a group to share the work. The group coordinator is a Kafka broker that manages this group, making sure each consumer gets its share of messages and handles changes smoothly.

When you have multiple consumers reading from the same Kafka topic and want to balance the load automatically.

When a consumer joins or leaves a group and you want Kafka to reassign message partitions without manual intervention.

When you want to track which consumers are active and manage their offsets centrally.

When you want to ensure fault tolerance by automatically handling consumer failures in a group.

When you want to commit offsets in a coordinated way to avoid message duplication or loss.

Commands

This command shows the current state of the consumer group named 'example-group', including which partitions each consumer is assigned to. It helps verify the group coordinator's assignment.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

example-topic   0          15              20              5               consumer-1-12345-67890-abcde                  /127.0.0.1      consumer-1

example-topic   1          20              20              0               consumer-2-12345-67890-fghij                  /127.0.0.1      consumer-2

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies the consumer group to inspect.

Lists all consumer groups currently managed by Kafka, showing which groups the group coordinator is tracking.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --list

Expected Output

example-group

analytics-group

logging-group

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

Resets the offsets for the 'example-group' on 'example-topic' to the earliest available messages. This operation is coordinated by the group coordinator to ensure consistency.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --reset-offsets --group example-group --topic example-topic --to-earliest --execute

Expected Output

Reset offset for partition example-topic-0 to offset 0.

Reset offset for partition example-topic-1 to offset 0.

Successfully reset offsets for group example-group.

→

--reset-offsets - Indicates that offsets should be reset.

→

--to-earliest - Resets offsets to the earliest available message.

→

--execute - Applies the reset operation.

Key Concept

The group coordinator is the Kafka broker responsible for managing consumer group membership and partition assignments to ensure balanced and fault-tolerant message consumption.

Common Mistakes

Trying to describe a consumer group that does not exist or has no active members.

Kafka cannot find the group coordinator or group information, so the command returns an error or empty output.

Ensure the consumer group is active by starting consumers before running the describe command.

Resetting offsets without the --execute flag.

The reset command only simulates the offset changes and does not apply them, causing confusion when offsets do not actually change.

Always include the --execute flag to apply offset resets.

Using an incorrect bootstrap server address.

The CLI cannot connect to Kafka, so it fails to find the group coordinator or group data.

Verify the Kafka broker address and port before running commands.

Summary

Use kafka-consumer-groups CLI to interact with the group coordinator and manage consumer groups.

Describe consumer groups to see partition assignments and lag, which the group coordinator manages.

Reset offsets carefully with the --execute flag to coordinate changes across the group.