Rebalancing behavior in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output when a consumer group rebalances?

Consider a Kafka consumer group with 3 consumers subscribed to a topic with 6 partitions. When one consumer leaves the group, what happens to the partition assignments?

Kafka

Consumer group: 3 consumers
Topic partitions: 6
Event: One consumer leaves

What is the new partition assignment behavior?

A Remaining consumers evenly redistribute all 6 partitions among themselves.

B Partitions are reassigned randomly without considering previous assignments.

C All partitions are assigned to a single remaining consumer.

D Partitions assigned to the leaving consumer remain unassigned until manual reassignment.

💡 Hint

Think about how Kafka ensures load balancing among active consumers.

When a consumer leaves, Kafka triggers a rebalance. The partitions previously assigned to the leaving consumer are redistributed evenly among the remaining consumers to maintain balanced consumption.

❓ Predict Output

intermediate

What error occurs if a consumer does not commit offsets before rebalancing?

A Kafka consumer processes messages but does not commit offsets before a rebalance occurs. What is the expected behavior?

Kafka

Consumer processes messages
No offset commit
Rebalance triggered

What happens next?

A Consumer will reprocess messages from the last committed offset, causing duplicates.

B Consumer loses all messages and skips to the latest offset.

C Consumer crashes with a CommitFailedException.

D Consumer automatically commits offsets before rebalance without explicit call.

💡 Hint

Offsets determine where consumption resumes after rebalance.

If offsets are not committed before rebalance, the consumer restarts from the last committed offset, causing some messages to be processed again, resulting in duplicates.

🔧 Debug

advanced

Why does this Kafka consumer group experience frequent rebalances?

Given the following consumer configuration, identify the cause of frequent rebalances:

group.id=my-group
session.timeout.ms=10000
auto.offset.reset=earliest
max.poll.interval.ms=3000

A session.timeout.ms is too high, causing delayed detection of dead consumers.

B max.poll.interval.ms is less than session.timeout.ms, causing session expiration.

C auto.offset.reset=earliest causes rebalances on startup.

D group.id is not unique, causing conflicts.

💡 Hint

Check the relationship between poll interval and session timeout.

If max.poll.interval.ms is less than session.timeout.ms, the consumer may not poll frequently enough, causing the broker to think it is dead and triggering rebalances.

🧠 Conceptual

advanced

How does Kafka ensure partition ownership consistency during rebalancing?

Which mechanism does Kafka use to guarantee that only one consumer owns a partition at a time during rebalancing?

A Kafka uses a first-come, first-served approach without coordination.

B Kafka relies on Zookeeper locks to assign partitions exclusively.

C Kafka uses a group coordinator and heartbeat protocol to manage partition ownership.

D Kafka assigns partitions randomly without ownership guarantees.

💡 Hint

Think about how Kafka tracks consumer liveness and ownership.

Kafka uses a group coordinator that tracks consumer heartbeats. If a consumer fails to send heartbeats, the coordinator triggers a rebalance to reassign partitions, ensuring exclusive ownership.

🚀 Application

expert

How to minimize downtime during Kafka consumer group rebalancing?

You have a critical Kafka consumer group that must minimize downtime during rebalances. Which approach best achieves this?

A Increase max.poll.records to process more messages per poll.

B Set session.timeout.ms to a very low value to detect failures faster.

C Disable heartbeats to prevent rebalances triggered by missed heartbeats.

D Use static group membership with assigned instance IDs to avoid full rebalances on consumer restarts.

💡 Hint

Consider Kafka features that reduce rebalance scope.

Static group membership allows consumers to keep their partition assignments across restarts, reducing the scope and frequency of rebalances and minimizing downtime.

Practice

1. What triggers a rebalance in a Kafka consumer group?

easy

A. When a consumer commits an offset

B. When a message is consumed

C. When a topic is deleted

D. When a consumer joins or leaves the group

Solution

1. Step 1: Understand rebalance triggers

   Rebalancing happens when the group membership changes or partitions change.

2. Step 2: Identify correct trigger

   Joining or leaving a consumer causes the group to rebalance to share partitions evenly.

3. Final Answer:

   When a consumer joins or leaves the group -> Option D

4. Quick Check:

   Consumer join/leave triggers rebalance [OK]

Hint: Rebalance happens on group membership changes [OK]

Common Mistakes:

• Thinking rebalance happens on every message
• Confusing topic deletion with rebalance
• Assuming offset commit triggers rebalance

2. Which of the following is the correct way to listen for rebalance events in Kafka consumer code?

easy

A. consumer.subscribe(topics, new ConsumerRebalanceListener() {...})

B. consumer.on('rebalance', callback)

C. consumer.addRebalanceListener(callback)

D. consumer.listenRebalance(callback)

Solution

1. Step 1: Recall Kafka consumer API

   Kafka's Java consumer uses subscribe with a ConsumerRebalanceListener to handle rebalance events.

2. Step 2: Match correct syntax

   Only consumer.subscribe(topics, new ConsumerRebalanceListener() {...}) shows the correct method signature for rebalance listener.

3. Final Answer:

   consumer.subscribe(topics, new ConsumerRebalanceListener() {...}) -> Option A

4. Quick Check:

   Use subscribe with ConsumerRebalanceListener [OK]

Hint: Use subscribe with ConsumerRebalanceListener for rebalance [OK]

Common Mistakes:

• Using non-existent methods like on or addRebalanceListener
• Confusing event listeners with other APIs
• Missing the listener parameter in subscribe

3. Given a consumer group with 3 consumers and 6 partitions, what happens after one consumer leaves the group?

medium

A. The consumer group stops consuming messages

B. The remaining 2 consumers rebalance and each consumes 3 partitions

C. The partitions are left unassigned until the group is restarted

D. The remaining consumers continue consuming their original partitions

Solution

1. Step 1: Understand partition assignment

   Partitions are divided evenly among active consumers in the group.

2. Step 2: Calculate new assignment after consumer leaves

   With 6 partitions and 2 consumers left, each gets 3 partitions after rebalance.

3. Final Answer:

   The remaining 2 consumers rebalance and each consumes 3 partitions -> Option B

4. Quick Check:

   Partitions evenly split after consumer leaves [OK]

Hint: Partitions redistribute evenly after consumer leaves [OK]

Common Mistakes:

• Assuming partitions stay with old consumers
• Thinking partitions become unassigned
• Believing the group stops consuming

4. You wrote a rebalance listener but your consumer crashes with a NullPointerException during rebalance. What is the likely cause?

medium

A. Not subscribing to the topic before starting the consumer

B. Using the wrong Kafka version

C. Not handling partition assignment and revocation properly in the listener

D. Committing offsets manually instead of auto-commit

Solution

1. Step 1: Analyze rebalance listener role

   The listener must handle partition revocation and assignment to avoid null references.

2. Step 2: Identify cause of NullPointerException

   Failing to check or update state during rebalance causes null pointer errors.

3. Final Answer:

   Not handling partition assignment and revocation properly in the listener -> Option C

4. Quick Check:

   Proper rebalance handling prevents NullPointerException [OK]

Hint: Handle partition changes carefully in rebalance listener [OK]

Common Mistakes:

• Ignoring partition revocation steps
• Assuming auto-commit fixes rebalance issues
• Blaming Kafka version without checking code

5. You want to ensure no duplicate message processing during rebalance in a Kafka consumer group. Which approach is best?

hard

A. Implement a ConsumerRebalanceListener to commit offsets before partition revocation

B. Disable rebalancing by fixing the number of consumers

C. Use auto-commit offsets with a short interval

D. Restart consumers manually after each rebalance

Solution

1. Step 1: Understand duplicate processing risk

   During rebalance, uncommitted offsets can cause reprocessing of messages.

2. Step 2: Use rebalance listener to commit offsets

   Committing offsets in onPartitionsRevoked ensures processed messages are saved before rebalance.

3. Step 3: Evaluate other options

   Disabling rebalance or relying on auto-commit does not guarantee no duplicates; manual restarts are inefficient.

4. Final Answer:

   Implement a ConsumerRebalanceListener to commit offsets before partition revocation -> Option A

5. Quick Check:

   Commit offsets on revoke to avoid duplicates [OK]

Hint: Commit offsets in rebalance listener before revocation [OK]

Common Mistakes:

• Disabling rebalance which is not practical
• Relying only on auto-commit offsets
• Restarting consumers manually wastes resources