How to Fix 'Rebalancing in Progress' Issue in Kafka

The rebalancing in progress message in Kafka occurs when consumer group members are changing, causing temporary unavailability. To fix it, ensure consumers have stable session timeouts, avoid frequent restarts, and check for network issues that delay heartbeats.

Why This Happens

Kafka consumer groups rebalance when members join, leave, or fail. During this process, consumers stop consuming temporarily to redistribute partitions. If rebalancing takes too long or happens repeatedly, it causes the rebalancing in progress state.

This often happens due to short session timeouts, slow processing, or network delays causing missed heartbeats.

bash

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --group my-group
# Consumer logs:
[Consumer clientId=consumer-1, groupId=my-group] Rebalancing in progress
[Consumer clientId=consumer-1, groupId=my-group] Revoking partitions []
[Consumer clientId=consumer-1, groupId=my-group] Assigning partitions [my-topic-0]

Output

[Consumer clientId=consumer-1, groupId=my-group] Rebalancing in progress
[Consumer clientId=consumer-1, groupId=my-group] Revoking partitions []
[Consumer clientId=consumer-1, groupId=my-group] Assigning partitions [my-topic-0]

The Fix

Increase the session.timeout.ms and heartbeat.interval.ms in your consumer configuration to allow more time for heartbeats and reduce unnecessary rebalances.

Also, avoid restarting consumers too often and ensure your consumer processing is efficient to prevent delays.

properties

bootstrap.servers=localhost:9092
group.id=my-group
session.timeout.ms=30000
heartbeat.interval.ms=10000
enable.auto.commit=true
auto.commit.interval.ms=5000
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer

Output

[Consumer clientId=consumer-1, groupId=my-group] Successfully joined group with assigned partitions [my-topic-0]
[Consumer clientId=consumer-1, groupId=my-group] Started consuming

Prevention

To avoid frequent rebalancing:

• Set appropriate session.timeout.ms and heartbeat.interval.ms values based on your network and processing speed.
• Keep consumer processing fast and avoid blocking operations.
• Use static group membership if supported to reduce rebalances on restarts.
• Monitor consumer logs and Kafka broker metrics for early signs of instability.

Related Errors

Other common Kafka consumer errors include:

• Offset commit failures: Fix by ensuring enable.auto.commit is properly configured.
• Consumer group coordinator not available: Check broker connectivity and broker logs.
• Timeout exceptions: Tune timeout settings and check network latency.

Key Takeaways

Increase session timeout and heartbeat interval to reduce unnecessary rebalances.

Avoid frequent consumer restarts and keep processing fast to prevent delays.

Use static group membership to stabilize consumer group membership if possible.

Monitor consumer and broker logs to detect and fix rebalance issues early.

Tune Kafka consumer configs based on your environment's network and workload.