Rebalancing behavior in Kafka - Commands & Configuration

Introduction

When multiple consumers read from the same Kafka group, Kafka needs to assign partitions evenly. Rebalancing is the process where Kafka redistributes these partitions among consumers to keep the load balanced and handle changes like new consumers joining or existing ones leaving.

When you add a new consumer to a consumer group and want Kafka to redistribute partitions automatically.

When a consumer in the group crashes or disconnects, and Kafka needs to reassign its partitions to others.

When you want to scale your application by increasing or decreasing the number of consumers in a group.

When you want to ensure that no single consumer is overloaded with too many partitions.

When you want to maintain fault tolerance by redistributing partitions after failures.

Commands

Starts a consumer that joins the group 'example-group' and reads messages from 'example-topic'. This triggers Kafka to assign partitions to this consumer.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group

Expected Output

Starting consumer for group example-group

[Consumer clientId=consumer-1, groupId=example-group] Subscribed to topic(s): example-topic

[Consumer clientId=consumer-1, groupId=example-group] Assigned partitions: [example-topic-0, example-topic-1]

→

--bootstrap-server - Specifies the Kafka server address

→

--topic - Specifies the topic to consume from

→

--group - Specifies the consumer group to join

Shows the current partition assignments and consumer details for the group 'example-group'. Useful to verify how partitions are distributed after rebalancing.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

example-group   example-topic   0          15              20              5               consumer-1-12345-67890-abcde                  /127.0.0.1      consumer-1

example-group   example-topic   1          10              20              10              consumer-1-12345-67890-abcde                  /127.0.0.1      consumer-1

→

--bootstrap-server - Specifies the Kafka server address

→

--describe - Shows detailed group info

→

--group - Specifies the consumer group to inspect

Starts a second consumer joining the same group 'example-group'. Kafka triggers a rebalance to redistribute partitions between the two consumers.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group

Expected Output

Starting consumer for group example-group

[Consumer clientId=consumer-2, groupId=example-group] Subscribed to topic(s): example-topic

[Consumer clientId=consumer-2, groupId=example-group] Assigned partitions: [example-topic-0, example-topic-1]

→

--bootstrap-server - Specifies the Kafka server address

→

--topic - Specifies the topic to consume from

→

--group - Specifies the consumer group to join

Verifies the new partition assignments after the rebalance caused by the second consumer joining the group.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

example-group   example-topic   0          15              20              5               consumer-1-12345-67890-abcde                  /127.0.0.1      consumer-1

example-group   example-topic   1          10              20              10              consumer-2-54321-09876-zyxwv                  /127.0.0.1      consumer-2

→

--bootstrap-server - Specifies the Kafka server address

→

--describe - Shows detailed group info

→

--group - Specifies the consumer group to inspect

Key Concept

If you remember nothing else from this pattern, remember: Kafka automatically redistributes topic partitions among consumers in a group to balance load whenever consumers join or leave.

Common Mistakes

Starting multiple consumers with the same group ID but different topics.

Kafka only balances partitions among consumers subscribed to the same topic within the group; different topics mean no rebalancing happens.

Ensure all consumers in the group subscribe to the same topic(s) to trigger rebalancing.

Not waiting for the rebalance to complete before producing or consuming messages.

During rebalance, consumers may not receive messages or may get duplicates, causing processing delays or errors.

Wait for rebalance to finish, indicated by partition assignment logs, before processing messages.

Using static partition assignments manually without understanding rebalance triggers.

Manual assignments bypass Kafka's automatic balancing and can cause uneven load or missed messages.

Use Kafka's consumer group management to let it handle partition assignments automatically.

Summary

Start consumers with the same group ID to join a consumer group and trigger partition assignment.

Use kafka-consumer-groups command to check how partitions are assigned and monitor rebalancing.

Adding or removing consumers causes Kafka to rebalance partitions automatically to keep load balanced.