Cooperative vs eager rebalancing in Kafka - Performance Comparison

Time Complexity: Cooperative vs eager rebalancing

O(n)

Understanding Time Complexity

When Kafka consumers rebalance, the way partitions are reassigned affects how long the process takes.

We want to understand how the time to rebalance grows as the number of consumers and partitions increases.

Scenario Under Consideration

Analyze the time complexity of these simplified rebalance steps.

// Eager rebalance example
onRebalance() {
  revokeAllPartitions();  // revoke all partitions from this consumer
  assignPartitions();    // assign partitions to this consumer
}

// Cooperative rebalance example
onRebalance() {
  revokePartitionsIncrementally();  // revoke partitions gradually
  assignPartitionsIncrementally();  // assign partitions gradually
}
    

This code shows two ways Kafka rebalances partitions among consumers: all at once (eager) or step-by-step (cooperative).

Identify Repeating Operations

Look at what repeats during rebalance.

• Primary operation: Reassigning partitions to consumers.
• How many times: For eager, all partitions are revoked and reassigned in one big step. For cooperative, partitions are revoked and assigned in smaller batches over multiple steps.

How Execution Grows With Input

As the number of partitions and consumers grows, the work to rebalance changes.

Input Size (partitions)	Approx. Operations (eager)	Approx. Operations (cooperative)
10	~10 revoke + 10 assign = 20	~10 revoke + 10 assign in steps = 20
100	~100 revoke + 100 assign = 200	~100 revoke + 100 assign in smaller steps = 200
1000	~1000 revoke + 1000 assign = 2000	~1000 revoke + 1000 assign in smaller steps = 2000

Pattern observation: Both methods do similar total work, but cooperative spreads it out, avoiding big pauses.

Final Time Complexity

Time Complexity: O(n)

This means the time to rebalance grows linearly with the number of partitions involved.

Common Mistake

[X] Wrong: "Cooperative rebalancing is slower because it does more steps."

[OK] Correct: Cooperative rebalancing spreads work over time but does not increase total operations; it reduces pause time and improves smoothness.

Interview Connect

Understanding how Kafka rebalancing scales helps you explain system behavior and design choices clearly, a valuable skill in many real-world roles.

Self-Check

"What if the number of consumers grows faster than partitions? How would that affect the rebalance time complexity?"