Rebalancing behavior in Kafka - Mini Project: Build & Apply

Understanding Kafka Consumer Group Rebalancing

📖 Scenario: You are working with Apache Kafka to process messages in real-time. Kafka uses consumer groups to distribute message processing across multiple consumers. When consumers join or leave a group, Kafka triggers a rebalance to redistribute partitions among consumers.Understanding how rebalancing works helps you build reliable and efficient Kafka consumers.

🎯 Goal: Build a simple Kafka consumer group setup that demonstrates rebalance behavior. You will create a consumer group with two consumers, configure rebalance listeners, and observe partition assignment changes when a consumer joins or leaves.

📋 What You'll Learn

Create two Kafka consumer instances with the same group ID

Configure a rebalance listener to handle partition assignment and revocation

Simulate consumer join and leave events to trigger rebalancing

Print partition assignment changes during rebalance events

💡 Why This Matters

🌍 Real World

Kafka consumer groups are widely used in real-time data processing systems to scale message consumption and ensure fault tolerance.

💼 Career

Understanding Kafka rebalancing is essential for roles like backend developer, data engineer, and site reliability engineer working with distributed streaming platforms.

1

Create Kafka consumer properties

Create a dictionary called consumer_props with these exact entries: 'bootstrap.servers': 'localhost:9092', 'group.id': 'test-group', 'key.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer', and 'value.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer'.

Kafka

# Create consumer_props dictionary with Kafka connection and deserializer settings
# Your code here

Hint

Use a Python dictionary with the exact keys and values as shown.

2

Define a rebalance listener class

Create a class called RebalanceListener with two methods: on_partitions_revoked and on_partitions_assigned. Each method should accept a parameter partitions and print 'Partitions revoked: ' or 'Partitions assigned: ' followed by the partitions list.

Kafka

consumer_props = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'key.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer',
    'value.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer'
}

# Define RebalanceListener class with on_partitions_revoked and on_partitions_assigned methods
# Your code here

Hint

Define a class with two methods that print the partitions list with the specified messages.

3

Create two consumers with rebalance listener

Create two consumer instances called consumer1 and consumer2 using KafkaConsumer with consumer_props. Assign an instance of RebalanceListener to both consumers' subscribe method as the listener parameter for topic 'test-topic'.

Kafka

consumer_props = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'key.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer',
    'value.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer'
}

class RebalanceListener:
    def on_partitions_revoked(self, partitions):
        print('Partitions revoked:', partitions)

    def on_partitions_assigned(self, partitions):
        print('Partitions assigned:', partitions)

# Create consumer1 and consumer2 with consumer_props
# Subscribe both to 'test-topic' with RebalanceListener instance
# Your code here

Hint

Use KafkaConsumer with consumer_props and subscribe both consumers to 'test-topic' with the rebalance listener.

4

Print partition assignments on rebalance events

Write code to simulate consumer1 polling messages once, then close consumer2 to trigger rebalance. Print 'Consumer1 polling...' before polling and 'Consumer2 closed to trigger rebalance' after closing consumer2.

Kafka

from kafka import KafkaConsumer

consumer_props = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'key.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer',
    'value.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer'
}

class RebalanceListener:
    def on_partitions_revoked(self, partitions):
        print('Partitions revoked:', partitions)

    def on_partitions_assigned(self, partitions):
        print('Partitions assigned:', partitions)

listener = RebalanceListener()
consumer1 = KafkaConsumer(**consumer_props)
consumer1.subscribe(['test-topic'], listener=listener)
consumer2 = KafkaConsumer(**consumer_props)
consumer2.subscribe(['test-topic'], listener=listener)

# Simulate consumer1 polling and consumer2 closing to trigger rebalance
# Your code here

Hint

Call consumer1.poll() and then consumer2.close() with print statements before and after.

Practice

1. What triggers a rebalance in a Kafka consumer group?

easy

A. When a consumer commits an offset

B. When a message is consumed

C. When a topic is deleted

D. When a consumer joins or leaves the group

Solution

1. Step 1: Understand rebalance triggers

   Rebalancing happens when the group membership changes or partitions change.

2. Step 2: Identify correct trigger

   Joining or leaving a consumer causes the group to rebalance to share partitions evenly.

3. Final Answer:

   When a consumer joins or leaves the group -> Option D

4. Quick Check:

   Consumer join/leave triggers rebalance [OK]

Hint: Rebalance happens on group membership changes [OK]

Common Mistakes:

• Thinking rebalance happens on every message
• Confusing topic deletion with rebalance
• Assuming offset commit triggers rebalance

2. Which of the following is the correct way to listen for rebalance events in Kafka consumer code?

easy

A. consumer.subscribe(topics, new ConsumerRebalanceListener() {...})

B. consumer.on('rebalance', callback)

C. consumer.addRebalanceListener(callback)

D. consumer.listenRebalance(callback)

Solution

1. Step 1: Recall Kafka consumer API

   Kafka's Java consumer uses subscribe with a ConsumerRebalanceListener to handle rebalance events.

2. Step 2: Match correct syntax

   Only consumer.subscribe(topics, new ConsumerRebalanceListener() {...}) shows the correct method signature for rebalance listener.

3. Final Answer:

   consumer.subscribe(topics, new ConsumerRebalanceListener() {...}) -> Option A

4. Quick Check:

   Use subscribe with ConsumerRebalanceListener [OK]

Hint: Use subscribe with ConsumerRebalanceListener for rebalance [OK]

Common Mistakes:

• Using non-existent methods like on or addRebalanceListener
• Confusing event listeners with other APIs
• Missing the listener parameter in subscribe

3. Given a consumer group with 3 consumers and 6 partitions, what happens after one consumer leaves the group?

medium

A. The consumer group stops consuming messages

B. The remaining 2 consumers rebalance and each consumes 3 partitions

C. The partitions are left unassigned until the group is restarted

D. The remaining consumers continue consuming their original partitions

Solution

1. Step 1: Understand partition assignment

   Partitions are divided evenly among active consumers in the group.

2. Step 2: Calculate new assignment after consumer leaves

   With 6 partitions and 2 consumers left, each gets 3 partitions after rebalance.

3. Final Answer:

   The remaining 2 consumers rebalance and each consumes 3 partitions -> Option B

4. Quick Check:

   Partitions evenly split after consumer leaves [OK]

Hint: Partitions redistribute evenly after consumer leaves [OK]

Common Mistakes:

• Assuming partitions stay with old consumers
• Thinking partitions become unassigned
• Believing the group stops consuming

4. You wrote a rebalance listener but your consumer crashes with a NullPointerException during rebalance. What is the likely cause?

medium

A. Not subscribing to the topic before starting the consumer

B. Using the wrong Kafka version

C. Not handling partition assignment and revocation properly in the listener

D. Committing offsets manually instead of auto-commit

Solution

1. Step 1: Analyze rebalance listener role

   The listener must handle partition revocation and assignment to avoid null references.

2. Step 2: Identify cause of NullPointerException

   Failing to check or update state during rebalance causes null pointer errors.

3. Final Answer:

   Not handling partition assignment and revocation properly in the listener -> Option C

4. Quick Check:

   Proper rebalance handling prevents NullPointerException [OK]

Hint: Handle partition changes carefully in rebalance listener [OK]

Common Mistakes:

• Ignoring partition revocation steps
• Assuming auto-commit fixes rebalance issues
• Blaming Kafka version without checking code

5. You want to ensure no duplicate message processing during rebalance in a Kafka consumer group. Which approach is best?

hard

A. Implement a ConsumerRebalanceListener to commit offsets before partition revocation

B. Disable rebalancing by fixing the number of consumers

C. Use auto-commit offsets with a short interval

D. Restart consumers manually after each rebalance

Solution

1. Step 1: Understand duplicate processing risk

   During rebalance, uncommitted offsets can cause reprocessing of messages.

2. Step 2: Use rebalance listener to commit offsets

   Committing offsets in onPartitionsRevoked ensures processed messages are saved before rebalance.

3. Step 3: Evaluate other options

   Disabling rebalance or relying on auto-commit does not guarantee no duplicates; manual restarts are inefficient.

4. Final Answer:

   Implement a ConsumerRebalanceListener to commit offsets before partition revocation -> Option A

5. Quick Check:

   Commit offsets on revoke to avoid duplicates [OK]

Hint: Commit offsets in rebalance listener before revocation [OK]

Common Mistakes:

• Disabling rebalance which is not practical
• Relying only on auto-commit offsets
• Restarting consumers manually wastes resources