Rebalancing behavior in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output when a consumer group rebalances?

Consider a Kafka consumer group with 3 consumers subscribed to a topic with 6 partitions. When one consumer leaves the group, what happens to the partition assignments?

Kafka

Consumer group: 3 consumers
Topic partitions: 6
Event: One consumer leaves

What is the new partition assignment behavior?

A Remaining consumers evenly redistribute all 6 partitions among themselves.

B Partitions are reassigned randomly without considering previous assignments.

C All partitions are assigned to a single remaining consumer.

D Partitions assigned to the leaving consumer remain unassigned until manual reassignment.

💡 Hint

Think about how Kafka ensures load balancing among active consumers.

When a consumer leaves, Kafka triggers a rebalance. The partitions previously assigned to the leaving consumer are redistributed evenly among the remaining consumers to maintain balanced consumption.

❓ Predict Output

intermediate

What error occurs if a consumer does not commit offsets before rebalancing?

A Kafka consumer processes messages but does not commit offsets before a rebalance occurs. What is the expected behavior?

Kafka

Consumer processes messages
No offset commit
Rebalance triggered

What happens next?

A Consumer will reprocess messages from the last committed offset, causing duplicates.

B Consumer loses all messages and skips to the latest offset.

C Consumer crashes with a CommitFailedException.

D Consumer automatically commits offsets before rebalance without explicit call.

💡 Hint

Offsets determine where consumption resumes after rebalance.

If offsets are not committed before rebalance, the consumer restarts from the last committed offset, causing some messages to be processed again, resulting in duplicates.

🔧 Debug

advanced

Why does this Kafka consumer group experience frequent rebalances?

Given the following consumer configuration, identify the cause of frequent rebalances:

group.id=my-group
session.timeout.ms=10000
auto.offset.reset=earliest
max.poll.interval.ms=3000

A session.timeout.ms is too high, causing delayed detection of dead consumers.

B max.poll.interval.ms is less than session.timeout.ms, causing session expiration.

C auto.offset.reset=earliest causes rebalances on startup.

D group.id is not unique, causing conflicts.

💡 Hint

Check the relationship between poll interval and session timeout.

If max.poll.interval.ms is less than session.timeout.ms, the consumer may not poll frequently enough, causing the broker to think it is dead and triggering rebalances.

🧠 Conceptual

advanced

How does Kafka ensure partition ownership consistency during rebalancing?

Which mechanism does Kafka use to guarantee that only one consumer owns a partition at a time during rebalancing?

A Kafka uses a first-come, first-served approach without coordination.

B Kafka relies on Zookeeper locks to assign partitions exclusively.

C Kafka uses a group coordinator and heartbeat protocol to manage partition ownership.

D Kafka assigns partitions randomly without ownership guarantees.

💡 Hint

Think about how Kafka tracks consumer liveness and ownership.

Kafka uses a group coordinator that tracks consumer heartbeats. If a consumer fails to send heartbeats, the coordinator triggers a rebalance to reassign partitions, ensuring exclusive ownership.

🚀 Application

expert

How to minimize downtime during Kafka consumer group rebalancing?

You have a critical Kafka consumer group that must minimize downtime during rebalances. Which approach best achieves this?

A Increase max.poll.records to process more messages per poll.

B Set session.timeout.ms to a very low value to detect failures faster.

C Disable heartbeats to prevent rebalances triggered by missed heartbeats.

D Use static group membership with assigned instance IDs to avoid full rebalances on consumer restarts.

💡 Hint

Consider Kafka features that reduce rebalance scope.

Static group membership allows consumers to keep their partition assignments across restarts, reducing the scope and frequency of rebalances and minimizing downtime.