Cooperative vs eager rebalancing in Kafka - CLI Comparison

Introduction

Kafka uses rebalancing to assign partitions to consumers in a group. Cooperative and eager rebalancing are two ways Kafka manages this process to keep data flowing smoothly when consumers join or leave.

When you add a new consumer to a Kafka consumer group and want to minimize message processing pauses.

When a consumer leaves or crashes and you need to redistribute partitions quickly.

When you want to avoid duplicate message processing during rebalancing.

When you want to reduce the time your application is unavailable due to rebalancing.

When you want to choose between faster rebalancing or smoother transitions with less disruption.

Commands

Starts a Kafka consumer using cooperative rebalancing to reduce disruption during partition reassignment.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --consumer-property partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Expected Output

Subscribed to topic(s): example-topic

[Consumer] Starting consumption with cooperative rebalancing

→

--consumer-property partition.assignment.strategy - Sets the partition assignment strategy to cooperative sticky assignor

Starts a Kafka consumer using eager rebalancing which immediately revokes all partitions and reassigns them.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group --consumer-property partition.assignment.strategy=org.apache.kafka.clients.consumer.StickyAssignor

Expected Output

Subscribed to topic(s): example-topic

[Consumer] Starting consumption with eager rebalancing

→

--consumer-property partition.assignment.strategy - Sets the partition assignment strategy to sticky assignor (eager rebalancing)

Shows the current partition assignments and consumer group status to verify rebalancing results.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

TOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                   HOST            CLIENT-ID

example-topic  0          100             100             0    consumer-1-1234abcd-5678-90ef-ghij-klmnopqrst  /127.0.0.1      consumer-1

example-topic  1          50              50              0    consumer-2-1234abcd-5678-90ef-ghij-klmnopqrst  /127.0.0.1      consumer-2

→

--describe - Shows detailed information about the consumer group

Key Concept

If you remember nothing else, remember: cooperative rebalancing reduces downtime by gradually moving partitions, while eager rebalancing moves all partitions at once causing longer pauses.

Common Mistakes

Using eager rebalancing when you have many consumers and want minimal processing pauses.

Eager rebalancing revokes all partitions immediately, causing longer pauses and possible duplicate processing.

Use cooperative rebalancing by setting partition.assignment.strategy to CooperativeStickyAssignor for smoother transitions.

Not setting the partition.assignment.strategy property explicitly and assuming the default fits all cases.

Kafka defaults to eager rebalancing which may cause unnecessary downtime in some applications.

Explicitly set the assignment strategy based on your application's tolerance for downtime and complexity.

Summary

Cooperative rebalancing moves partitions gradually to reduce consumer downtime.

Eager rebalancing revokes all partitions immediately causing longer pauses.

Set partition.assignment.strategy to control the rebalancing behavior in Kafka consumers.