How to Rebalance Consumer Group in Kafka: Simple Steps

To rebalance a Kafka consumer group, you can either restart the consumers or trigger a rebalance by changing the group membership, such as adding or removing a consumer. Kafka automatically redistributes partitions among active consumers in the group during rebalance using the kafka-consumer-groups.sh tool or programmatically via the consumer API.

Syntax

The main command to manage and observe consumer groups is kafka-consumer-groups.sh. To trigger a rebalance, you typically restart consumers or change group membership. The syntax to describe a consumer group is:

kafka-consumer-groups.sh --bootstrap-server <broker-list> --describe --group <group-id>

This shows the current partition assignments. To force rebalance, you can restart consumers or use the --reset-offsets option carefully.

bash

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-consumer-group

Example

This example shows how to check the current consumer group state and trigger a rebalance by restarting consumers.

bash

# Check current consumer group assignment
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-consumer-group

# Restart consumer application to trigger rebalance
# (Stop and start your consumer process or container)

# After restart, check the new assignment
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-consumer-group

Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
my-consumer-group  my-topic       0          1234            1234            0               consumer-1-1234abcd-5678-90ef-ghij-klmnopqrstuv /127.0.0.1      consumer-1
my-consumer-group  my-topic       1          5678            5678            0               consumer-2-2345bcde-6789-01fg-hijk-lmnopqrstuvw /127.0.0.1      consumer-2

Common Pitfalls

• Not restarting consumers: Kafka only rebalances when group membership changes, so doing nothing won't rebalance.
• Using --reset-offsets incorrectly: This command changes offsets but does not trigger rebalance by itself.
• Long session timeouts: If session timeout is too long, rebalance can be delayed.
• Uneven partition count: If partitions are fewer than consumers, some consumers stay idle.

bash

## Wrong: Trying to rebalance without changing group membership
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group my-consumer-group --to-earliest --all-topics --execute

## Right: Restart consumers to trigger rebalance
# Stop consumer process
# Start consumer process again

Quick Reference

Action	Command / Description
Check consumer group status	kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group
Trigger rebalance	Restart consumers or add/remove consumers in the group
Reset offsets (does not rebalance)	kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group --to-earliest --all-topics --execute
Avoid long session timeouts	Set consumer config 'session.timeout.ms' to a reasonable value (default 10000 ms)

Key Takeaways

Kafka rebalances consumer groups automatically when consumers join or leave the group.

To force rebalance, restart consumers or change group membership; offset reset alone does not trigger rebalance.

Use kafka-consumer-groups.sh to monitor group partition assignments and consumer status.

Ensure session timeouts are configured properly to avoid delayed rebalances.

Partition count should be equal or greater than consumers for effective load balancing.