Rebalancing behavior in Kafka - Step-by-Step Execution

Process Flow - Rebalancing behavior

Consumer Group Starts

↓

Coordinator Assigns Partitions

↓

Consumers Receive Assignments

↓

Consumers Process Messages

↓

Consumer Joins/Leaves or Session Timeout

↓

Trigger Rebalance

↓

Coordinator Reassigns Partitions

↓

Consumers Receive New Assignments

↓

Continue Processing

This flow shows how Kafka consumer groups rebalance partitions when consumers join, leave, or fail.

Execution Sample

Kafka

consumer1 joins group
coordinator assigns partitions
consumer1 processes assigned partitions
consumer2 joins group
rebalance triggered
coordinator reassigns partitions
consumers receive new assignments

Simulates consumers joining a group and Kafka rebalancing partition assignments.

Process Table

Step	Event	Coordinator Action	Consumer State	Partition Assignment
1	consumer1 joins group	Assign all partitions to consumer1	consumer1 connected	consumer1: [partition0, partition1, partition2]
2	consumer1 processes messages	No change	consumer1 processing	consumer1: [partition0, partition1, partition2]
3	consumer2 joins group	Trigger rebalance	consumer1 and consumer2 connected	none (rebalancing)
4	rebalance triggered	Reassign partitions evenly	consumer1 and consumer2 paused	none (rebalancing)
5	coordinator reassigns partitions	Assign partitions to consumers	consumer1 and consumer2 ready	consumer1: [partition0, partition1], consumer2: [partition2]
6	consumers resume processing	No change	consumer1 and consumer2 processing	consumer1: [partition0, partition1], consumer2: [partition2]
7	consumer1 leaves group	Trigger rebalance	consumer2 connected	none (rebalancing)
8	rebalance triggered	Assign all partitions to consumer2	consumer2 ready	consumer2: [partition0, partition1, partition2]
9	consumer2 processes messages	No change	consumer2 processing	consumer2: [partition0, partition1, partition2]

💡 Rebalancing stops when all consumers have stable partition assignments.

Status Tracker

Variable	Start	After Step 3	After Step 5	After Step 7	Final
consumer1_state	not connected	connected	processing	left group	left group
consumer2_state	not connected	connected	processing	processing	processing
partition_assignment	none	none (rebalancing)	consumer1: [partition0, partition1], consumer2: [partition2]	none (rebalancing)	consumer2: [partition0, partition1, partition2]

Key Moments - 3 Insights

Why do consumers stop processing during rebalance?

During rebalance (see steps 3-5 in execution_table), consumers pause processing to avoid duplicate processing and ensure partitions are reassigned cleanly.

What triggers a rebalance in a consumer group?

A rebalance is triggered when a consumer joins or leaves the group or when a consumer session times out, as shown at steps 3 and 7.

How are partitions assigned after rebalance?

The coordinator evenly distributes partitions among active consumers, as seen in step 5 where partitions are split between consumer1 and consumer2.

Visual Quiz - 3 Questions

Look at the execution_table, what is the partition assignment after step 5?

A consumer2: [partition0, partition1, partition2]

B consumer1: [partition0, partition1, partition2]

C consumer1: [partition0, partition1], consumer2: [partition2]

D No partitions assigned

💡 Hint

Check the 'Partition Assignment' column at step 5 in execution_table.

At which step does consumer2 start processing messages?

A Step 6

B Step 3

C Step 7

D Step 9

💡 Hint

Look for when consumer2 state changes to processing in execution_table.

If consumer1 never leaves the group, what would happen to partition assignments after step 7?

A All partitions assigned to consumer2

B Partitions remain assigned as in step 5

C Rebalance triggered again

D No partitions assigned

💡 Hint

Refer to variable_tracker for consumer1_state and partition_assignment after step 7.

Concept Snapshot

Kafka consumer groups assign partitions to consumers.
When consumers join or leave, a rebalance happens.
During rebalance, consumers pause processing.
Coordinator reassigns partitions evenly.
Consumers resume processing after rebalance.

Full Transcript

This visual execution shows how Kafka consumer groups handle rebalancing. When a consumer joins or leaves, the coordinator triggers a rebalance. During rebalance, consumers pause processing to avoid duplicates. The coordinator then reassigns partitions evenly among active consumers. After assignments, consumers resume processing. The execution table traces each step, showing consumer states and partition assignments. Key moments clarify why processing pauses and what triggers rebalances. The quiz tests understanding of partition assignments and rebalance triggers.

Practice

1. What triggers a rebalance in a Kafka consumer group?

easy

A. When a consumer commits an offset

B. When a message is consumed

C. When a topic is deleted

D. When a consumer joins or leaves the group

Solution

1. Step 1: Understand rebalance triggers

   Rebalancing happens when the group membership changes or partitions change.

2. Step 2: Identify correct trigger

   Joining or leaving a consumer causes the group to rebalance to share partitions evenly.

3. Final Answer:

   When a consumer joins or leaves the group -> Option D

4. Quick Check:

   Consumer join/leave triggers rebalance [OK]

Hint: Rebalance happens on group membership changes [OK]

Common Mistakes:

• Thinking rebalance happens on every message
• Confusing topic deletion with rebalance
• Assuming offset commit triggers rebalance

2. Which of the following is the correct way to listen for rebalance events in Kafka consumer code?

easy

A. consumer.subscribe(topics, new ConsumerRebalanceListener() {...})

B. consumer.on('rebalance', callback)

C. consumer.addRebalanceListener(callback)

D. consumer.listenRebalance(callback)

Solution

1. Step 1: Recall Kafka consumer API

   Kafka's Java consumer uses subscribe with a ConsumerRebalanceListener to handle rebalance events.

2. Step 2: Match correct syntax

   Only consumer.subscribe(topics, new ConsumerRebalanceListener() {...}) shows the correct method signature for rebalance listener.

3. Final Answer:

   consumer.subscribe(topics, new ConsumerRebalanceListener() {...}) -> Option A

4. Quick Check:

   Use subscribe with ConsumerRebalanceListener [OK]

Hint: Use subscribe with ConsumerRebalanceListener for rebalance [OK]

Common Mistakes:

• Using non-existent methods like on or addRebalanceListener
• Confusing event listeners with other APIs
• Missing the listener parameter in subscribe

3. Given a consumer group with 3 consumers and 6 partitions, what happens after one consumer leaves the group?

medium

A. The consumer group stops consuming messages

B. The remaining 2 consumers rebalance and each consumes 3 partitions

C. The partitions are left unassigned until the group is restarted

D. The remaining consumers continue consuming their original partitions

Solution

1. Step 1: Understand partition assignment

   Partitions are divided evenly among active consumers in the group.

2. Step 2: Calculate new assignment after consumer leaves

   With 6 partitions and 2 consumers left, each gets 3 partitions after rebalance.

3. Final Answer:

   The remaining 2 consumers rebalance and each consumes 3 partitions -> Option B

4. Quick Check:

   Partitions evenly split after consumer leaves [OK]

Hint: Partitions redistribute evenly after consumer leaves [OK]

Common Mistakes:

• Assuming partitions stay with old consumers
• Thinking partitions become unassigned
• Believing the group stops consuming

4. You wrote a rebalance listener but your consumer crashes with a NullPointerException during rebalance. What is the likely cause?

medium

A. Not subscribing to the topic before starting the consumer

B. Using the wrong Kafka version

C. Not handling partition assignment and revocation properly in the listener

D. Committing offsets manually instead of auto-commit

Solution

1. Step 1: Analyze rebalance listener role

   The listener must handle partition revocation and assignment to avoid null references.

2. Step 2: Identify cause of NullPointerException

   Failing to check or update state during rebalance causes null pointer errors.

3. Final Answer:

   Not handling partition assignment and revocation properly in the listener -> Option C

4. Quick Check:

   Proper rebalance handling prevents NullPointerException [OK]

Hint: Handle partition changes carefully in rebalance listener [OK]

Common Mistakes:

• Ignoring partition revocation steps
• Assuming auto-commit fixes rebalance issues
• Blaming Kafka version without checking code

5. You want to ensure no duplicate message processing during rebalance in a Kafka consumer group. Which approach is best?

hard

A. Implement a ConsumerRebalanceListener to commit offsets before partition revocation

B. Disable rebalancing by fixing the number of consumers

C. Use auto-commit offsets with a short interval

D. Restart consumers manually after each rebalance

Solution

1. Step 1: Understand duplicate processing risk

   During rebalance, uncommitted offsets can cause reprocessing of messages.

2. Step 2: Use rebalance listener to commit offsets

   Committing offsets in onPartitionsRevoked ensures processed messages are saved before rebalance.

3. Step 3: Evaluate other options

   Disabling rebalance or relying on auto-commit does not guarantee no duplicates; manual restarts are inefficient.

4. Final Answer:

   Implement a ConsumerRebalanceListener to commit offsets before partition revocation -> Option A

5. Quick Check:

   Commit offsets on revoke to avoid duplicates [OK]

Hint: Commit offsets in rebalance listener before revocation [OK]

Common Mistakes:

• Disabling rebalance which is not practical
• Relying only on auto-commit offsets
• Restarting consumers manually wastes resources