Rebalancing behavior in Kafka - Commands & Configuration

Introduction

When multiple consumers read from the same Kafka group, Kafka needs to assign partitions evenly. Rebalancing is the process where Kafka redistributes these partitions among consumers to keep the load balanced and handle changes like new consumers joining or existing ones leaving.

When you add a new consumer to a consumer group and want Kafka to redistribute partitions automatically.

When a consumer in the group crashes or disconnects, and Kafka needs to reassign its partitions to others.

When you want to scale your application by increasing or decreasing the number of consumers in a group.

When you want to ensure that no single consumer is overloaded with too many partitions.

When you want to maintain fault tolerance by redistributing partitions after failures.

Commands

Starts a consumer that joins the group 'example-group' and reads messages from 'example-topic'. This triggers Kafka to assign partitions to this consumer.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group

Expected Output

Starting consumer for group example-group

[Consumer clientId=consumer-1, groupId=example-group] Subscribed to topic(s): example-topic

[Consumer clientId=consumer-1, groupId=example-group] Assigned partitions: [example-topic-0, example-topic-1]

→

--bootstrap-server - Specifies the Kafka server address

→

--topic - Specifies the topic to consume from

→

--group - Specifies the consumer group to join

Shows the current partition assignments and consumer details for the group 'example-group'. Useful to verify how partitions are distributed after rebalancing.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

example-group   example-topic   0          15              20              5               consumer-1-12345-67890-abcde                  /127.0.0.1      consumer-1

example-group   example-topic   1          10              20              10              consumer-1-12345-67890-abcde                  /127.0.0.1      consumer-1

→

--bootstrap-server - Specifies the Kafka server address

→

--describe - Shows detailed group info

→

--group - Specifies the consumer group to inspect

Starts a second consumer joining the same group 'example-group'. Kafka triggers a rebalance to redistribute partitions between the two consumers.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --group example-group

Expected Output

Starting consumer for group example-group

[Consumer clientId=consumer-2, groupId=example-group] Subscribed to topic(s): example-topic

[Consumer clientId=consumer-2, groupId=example-group] Assigned partitions: [example-topic-0, example-topic-1]

→

--bootstrap-server - Specifies the Kafka server address

→

--topic - Specifies the topic to consume from

→

--group - Specifies the consumer group to join

Verifies the new partition assignments after the rebalance caused by the second consumer joining the group.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID

example-group   example-topic   0          15              20              5               consumer-1-12345-67890-abcde                  /127.0.0.1      consumer-1

example-group   example-topic   1          10              20              10              consumer-2-54321-09876-zyxwv                  /127.0.0.1      consumer-2

→

--bootstrap-server - Specifies the Kafka server address

→

--describe - Shows detailed group info

→

--group - Specifies the consumer group to inspect

Key Concept

If you remember nothing else from this pattern, remember: Kafka automatically redistributes topic partitions among consumers in a group to balance load whenever consumers join or leave.

Common Mistakes

Starting multiple consumers with the same group ID but different topics.

Kafka only balances partitions among consumers subscribed to the same topic within the group; different topics mean no rebalancing happens.

Ensure all consumers in the group subscribe to the same topic(s) to trigger rebalancing.

Not waiting for the rebalance to complete before producing or consuming messages.

During rebalance, consumers may not receive messages or may get duplicates, causing processing delays or errors.

Wait for rebalance to finish, indicated by partition assignment logs, before processing messages.

Using static partition assignments manually without understanding rebalance triggers.

Manual assignments bypass Kafka's automatic balancing and can cause uneven load or missed messages.

Use Kafka's consumer group management to let it handle partition assignments automatically.

Summary

Start consumers with the same group ID to join a consumer group and trigger partition assignment.

Use kafka-consumer-groups command to check how partitions are assigned and monitor rebalancing.

Adding or removing consumers causes Kafka to rebalance partitions automatically to keep load balanced.

Practice

1. What triggers a rebalance in a Kafka consumer group?

easy

A. When a consumer commits an offset

B. When a message is consumed

C. When a topic is deleted

D. When a consumer joins or leaves the group

Solution

1. Step 1: Understand rebalance triggers

   Rebalancing happens when the group membership changes or partitions change.

2. Step 2: Identify correct trigger

   Joining or leaving a consumer causes the group to rebalance to share partitions evenly.

3. Final Answer:

   When a consumer joins or leaves the group -> Option D

4. Quick Check:

   Consumer join/leave triggers rebalance [OK]

Hint: Rebalance happens on group membership changes [OK]

Common Mistakes:

• Thinking rebalance happens on every message
• Confusing topic deletion with rebalance
• Assuming offset commit triggers rebalance

2. Which of the following is the correct way to listen for rebalance events in Kafka consumer code?

easy

A. consumer.subscribe(topics, new ConsumerRebalanceListener() {...})

B. consumer.on('rebalance', callback)

C. consumer.addRebalanceListener(callback)

D. consumer.listenRebalance(callback)

Solution

1. Step 1: Recall Kafka consumer API

   Kafka's Java consumer uses subscribe with a ConsumerRebalanceListener to handle rebalance events.

2. Step 2: Match correct syntax

   Only consumer.subscribe(topics, new ConsumerRebalanceListener() {...}) shows the correct method signature for rebalance listener.

3. Final Answer:

   consumer.subscribe(topics, new ConsumerRebalanceListener() {...}) -> Option A

4. Quick Check:

   Use subscribe with ConsumerRebalanceListener [OK]

Hint: Use subscribe with ConsumerRebalanceListener for rebalance [OK]

Common Mistakes:

• Using non-existent methods like on or addRebalanceListener
• Confusing event listeners with other APIs
• Missing the listener parameter in subscribe

3. Given a consumer group with 3 consumers and 6 partitions, what happens after one consumer leaves the group?

medium

A. The consumer group stops consuming messages

B. The remaining 2 consumers rebalance and each consumes 3 partitions

C. The partitions are left unassigned until the group is restarted

D. The remaining consumers continue consuming their original partitions

Solution

1. Step 1: Understand partition assignment

   Partitions are divided evenly among active consumers in the group.

2. Step 2: Calculate new assignment after consumer leaves

   With 6 partitions and 2 consumers left, each gets 3 partitions after rebalance.

3. Final Answer:

   The remaining 2 consumers rebalance and each consumes 3 partitions -> Option B

4. Quick Check:

   Partitions evenly split after consumer leaves [OK]

Hint: Partitions redistribute evenly after consumer leaves [OK]

Common Mistakes:

• Assuming partitions stay with old consumers
• Thinking partitions become unassigned
• Believing the group stops consuming

4. You wrote a rebalance listener but your consumer crashes with a NullPointerException during rebalance. What is the likely cause?

medium

A. Not subscribing to the topic before starting the consumer

B. Using the wrong Kafka version

C. Not handling partition assignment and revocation properly in the listener

D. Committing offsets manually instead of auto-commit

Solution

1. Step 1: Analyze rebalance listener role

   The listener must handle partition revocation and assignment to avoid null references.

2. Step 2: Identify cause of NullPointerException

   Failing to check or update state during rebalance causes null pointer errors.

3. Final Answer:

   Not handling partition assignment and revocation properly in the listener -> Option C

4. Quick Check:

   Proper rebalance handling prevents NullPointerException [OK]

Hint: Handle partition changes carefully in rebalance listener [OK]

Common Mistakes:

• Ignoring partition revocation steps
• Assuming auto-commit fixes rebalance issues
• Blaming Kafka version without checking code

5. You want to ensure no duplicate message processing during rebalance in a Kafka consumer group. Which approach is best?

hard

A. Implement a ConsumerRebalanceListener to commit offsets before partition revocation

B. Disable rebalancing by fixing the number of consumers

C. Use auto-commit offsets with a short interval

D. Restart consumers manually after each rebalance

Solution

1. Step 1: Understand duplicate processing risk

   During rebalance, uncommitted offsets can cause reprocessing of messages.

2. Step 2: Use rebalance listener to commit offsets

   Committing offsets in onPartitionsRevoked ensures processed messages are saved before rebalance.

3. Step 3: Evaluate other options

   Disabling rebalance or relying on auto-commit does not guarantee no duplicates; manual restarts are inefficient.

4. Final Answer:

   Implement a ConsumerRebalanceListener to commit offsets before partition revocation -> Option A

5. Quick Check:

   Commit offsets on revoke to avoid duplicates [OK]

Hint: Commit offsets in rebalance listener before revocation [OK]

Common Mistakes:

• Disabling rebalance which is not practical
• Relying only on auto-commit offsets
• Restarting consumers manually wastes resources