Cooperative vs eager rebalancing in Kafka - Visual Side-by-Side Comparison

Process Flow - Cooperative vs eager rebalancing

Consumer Group Start

↓

Assign Partitions

↓

Detect Group Change?

No→Continue Processing

Yes↓

Rebalancing Triggered

↓

Eager Rebalancing

↓

Revoke All Partitions

↓

Reassign All Partitions

↓

Resume Processing

↓

Wait for Next Group Change

When a consumer group changes, Kafka triggers rebalancing. Eager rebalancing revokes and reassigns all partitions at once, pausing processing fully. Cooperative rebalancing revokes and assigns partitions incrementally, allowing smoother transitions.

Execution Sample

Kafka

1. Consumer group detects member leave
2. Eager: revoke all partitions immediately
3. Eager: assign all partitions to new members
4. Cooperative: revoke only partitions needed
5. Cooperative: assign partitions incrementally
6. Consumers resume processing

This sequence shows how eager rebalancing revokes all partitions at once, while cooperative rebalancing does it step-by-step.

Process Table

Step	Event	Eager Rebalancing Action	Cooperative Rebalancing Action	Processing Status
1	Member leaves group	Trigger full rebalance	Trigger incremental rebalance	Processing paused (both)
2	Revoke partitions	Revoke all partitions immediately	Revoke only partitions needed	Eager: paused fully, Cooperative: partial pause
3	Assign partitions	Assign all partitions to new members	Assign partitions incrementally	Eager: no processing, Cooperative: partial processing
4	Resume processing	Resume after full rebalance	Resume as partitions assigned	Eager: resumes later, Cooperative: resumes sooner
5	Wait for next change	Idle until next rebalance	Idle until next rebalance	Processing ongoing

💡 Rebalancing ends when all partitions are assigned and consumers resume processing

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
Partitions Assigned (Eager)	All assigned	None assigned (revoked all)	All assigned	All assigned	All assigned
Partitions Assigned (Cooperative)	All assigned	Some revoked	Some assigned	All assigned	All assigned
Processing Status (Eager)	Running	Paused fully	Paused fully	Running	Running
Processing Status (Cooperative)	Running	Partially paused	Partially running	Running	Running

Key Moments - 3 Insights

Why does eager rebalancing pause all processing at once?

Because eager rebalancing revokes all partitions immediately (see execution_table step 2), consumers cannot process any data until reassignment finishes.

How does cooperative rebalancing allow some processing during rebalance?

Cooperative rebalancing revokes only needed partitions incrementally (execution_table step 2), so consumers keep processing partitions not revoked yet.

What triggers rebalancing in both modes?

Any group membership change like a consumer leaving or joining triggers rebalancing (execution_table step 1).

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what happens in eager rebalancing?

A No partitions are revoked

B Only some partitions are revoked

C All partitions are revoked immediately

D Partitions are assigned incrementally

💡 Hint

Check the 'Eager Rebalancing Action' column at step 2 in execution_table

According to variable_tracker, after step 3, what is the processing status in cooperative rebalancing?

A Partially running

B Paused fully

C Running fully

D Stopped

💡 Hint

Look at 'Processing Status (Cooperative)' after step 3 in variable_tracker

If a new consumer joins, which step in execution_table shows the rebalance trigger?

A Step 5

B Step 1

C Step 3

D Step 4

💡 Hint

Rebalancing triggers on group membership changes shown in step 1

Concept Snapshot

Cooperative vs eager rebalancing in Kafka:
- Eager revokes all partitions at once, pausing processing fully
- Cooperative revokes partitions incrementally, allowing partial processing
- Rebalancing triggers on group membership changes
- Cooperative reduces downtime and improves stability
- Eager is simpler but causes full pause during rebalance

Full Transcript

In Kafka consumer groups, rebalancing happens when group membership changes. Eager rebalancing revokes all partitions immediately, causing all consumers to pause processing until reassignment completes. Cooperative rebalancing revokes partitions incrementally, allowing consumers to keep processing partitions not yet revoked. This reduces downtime and improves stability. The execution table shows step-by-step actions for both modes, and the variable tracker shows how partition assignments and processing status change over time. Key moments clarify why eager pauses fully and cooperative allows partial processing. The visual quiz tests understanding of these steps and states.

Practice

1. What is the main advantage of cooperative rebalancing in Kafka consumer groups?

easy

A. It reduces consumer pause time by reassigning partitions gradually.

B. It reassigns all partitions at once for faster processing.

C. It eliminates the need for consumer group coordination.

D. It increases the number of partitions automatically.

Solution

1. Step 1: Understand cooperative rebalancing behavior

   Cooperative rebalancing allows consumers to leave partitions gradually, which reduces the pause time during rebalancing.

2. Step 2: Compare with eager rebalancing

   Eager rebalancing reassigns all partitions at once, causing longer pauses, unlike cooperative rebalancing.

3. Final Answer:

   It reduces consumer pause time by reassigning partitions gradually. -> Option A

4. Quick Check:

   Cooperative rebalancing = gradual partition reassignment [OK]

Hint: Cooperative means gradual, eager means all at once [OK]

Common Mistakes:

• Confusing eager with cooperative rebalancing
• Thinking cooperative rebalancing eliminates coordination
• Assuming rebalancing increases partitions

2. Which configuration property enables cooperative rebalancing in Kafka consumers?

easy

A. group.rebalance.strategy = 'eager'

B. rebalance.policy = 'cooperative'

C. enable.cooperative.rebalance = true

D. partition.assignment.strategy = 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor'

Solution

1. Step 1: Identify the correct property for assignment strategy

   Kafka uses partition.assignment.strategy to set the rebalance strategy class.

2. Step 2: Recognize the cooperative assignor class

   The cooperative rebalancing strategy is implemented by org.apache.kafka.clients.consumer.CooperativeStickyAssignor.

3. Final Answer:

   partition.assignment.strategy = 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor' -> Option D

4. Quick Check:

   Cooperative assignor class = partition.assignment.strategy = 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor' [OK]

Hint: Look for 'partition.assignment.strategy' with cooperative class [OK]

Common Mistakes:

• Using incorrect property names
• Confusing eager and cooperative strategy names
• Assuming a boolean flag enables cooperative rebalancing

3. Given two consumers in a group using eager rebalancing, what happens when one consumer leaves?

medium

A. All partitions in the group are reassigned immediately, causing a pause.

B. Only the partitions of the leaving consumer are reassigned gradually.

C. The remaining consumer continues without any rebalancing.

D. Partitions are reassigned only after a timeout period.

Solution

1. Step 1: Understand eager rebalancing behavior on consumer leave

   In eager rebalancing, when any consumer leaves, all partitions are reassigned immediately to maintain balance.

2. Step 2: Recognize the impact on consumer pause

   This immediate reassignment causes a pause for all consumers during the rebalance.

3. Final Answer:

   All partitions in the group are reassigned immediately, causing a pause. -> Option A

4. Quick Check:

   Eager rebalancing = full immediate reassignment [OK]

Hint: Eager means all partitions reassign at once [OK]

Common Mistakes:

• Thinking only leaving consumer's partitions reassign
• Assuming no pause occurs
• Believing reassignment waits for timeout

4. You have a Kafka consumer group using cooperative rebalancing but notice long pauses during rebalances. What is a likely cause?

medium

A. Partitions are being reassigned gradually as expected.

B. The consumer group has only one consumer.

C. Consumers are using the eager assignor instead of cooperative assignor.

D. The rebalance timeout is set too low.

Solution

1. Step 1: Check the assignor configuration

   If long pauses occur despite expecting cooperative rebalancing, the consumers might be using the eager assignor by mistake.

2. Step 2: Understand the impact of eager assignor

   Eager assignor causes all partitions to reassign at once, leading to longer pauses.

3. Final Answer:

   Consumers are using the eager assignor instead of cooperative assignor. -> Option C

4. Quick Check:

   Long pauses + cooperative expected = eager assignor used [OK]

Hint: Long pauses mean eager assignor, not cooperative [OK]

Common Mistakes:

• Assuming gradual reassignment causes long pauses
• Ignoring assignor configuration
• Blaming rebalance timeout without checking assignor

5. You want to minimize downtime in a Kafka consumer group with frequent consumer joins and leaves. Which strategy and configuration should you choose?

hard

A. Use eager rebalancing with default assignor for simplicity.

B. Use cooperative rebalancing with CooperativeStickyAssignor to reduce pause time.

C. Disable rebalancing to avoid any pauses.

D. Use eager rebalancing but increase session timeout to reduce rebalances.

Solution

1. Step 1: Identify the goal

   The goal is to minimize downtime during frequent consumer group changes.

2. Step 2: Choose the appropriate rebalance strategy

   Cooperative rebalancing reduces pause time by reassigning partitions gradually, which suits frequent joins/leaves.

3. Step 3: Select the correct configuration

   Using CooperativeStickyAssignor enables cooperative rebalancing in Kafka consumers.

4. Final Answer:

   Use cooperative rebalancing with CooperativeStickyAssignor to reduce pause time. -> Option B

5. Quick Check:

   Minimize downtime = cooperative rebalancing + correct assignor [OK]

Hint: CooperativeStickyAssignor minimizes downtime with frequent changes [OK]

Common Mistakes:

• Choosing eager rebalancing for low downtime
• Disabling rebalancing causing stale assignments
• Misconfiguring session timeout instead of strategy