Rebalancing behavior in Kafka - Mini Project: Build & Apply

Understanding Kafka Consumer Group Rebalancing

📖 Scenario: You are working with Apache Kafka to process messages in real-time. Kafka uses consumer groups to distribute message processing across multiple consumers. When consumers join or leave a group, Kafka triggers a rebalance to redistribute partitions among consumers.Understanding how rebalancing works helps you build reliable and efficient Kafka consumers.

🎯 Goal: Build a simple Kafka consumer group setup that demonstrates rebalance behavior. You will create a consumer group with two consumers, configure rebalance listeners, and observe partition assignment changes when a consumer joins or leaves.

📋 What You'll Learn

Create two Kafka consumer instances with the same group ID

Configure a rebalance listener to handle partition assignment and revocation

Simulate consumer join and leave events to trigger rebalancing

Print partition assignment changes during rebalance events

💡 Why This Matters

🌍 Real World

Kafka consumer groups are widely used in real-time data processing systems to scale message consumption and ensure fault tolerance.

💼 Career

Understanding Kafka rebalancing is essential for roles like backend developer, data engineer, and site reliability engineer working with distributed streaming platforms.

1

Create Kafka consumer properties

Create a dictionary called consumer_props with these exact entries: 'bootstrap.servers': 'localhost:9092', 'group.id': 'test-group', 'key.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer', and 'value.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer'.

Kafka

# Create consumer_props dictionary with Kafka connection and deserializer settings
# Your code here

Need a hint?

Use a Python dictionary with the exact keys and values as shown.

2

Define a rebalance listener class

Create a class called RebalanceListener with two methods: on_partitions_revoked and on_partitions_assigned. Each method should accept a parameter partitions and print 'Partitions revoked: ' or 'Partitions assigned: ' followed by the partitions list.

Kafka

consumer_props = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'key.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer',
    'value.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer'
}

# Define RebalanceListener class with on_partitions_revoked and on_partitions_assigned methods
# Your code here

Need a hint?

Define a class with two methods that print the partitions list with the specified messages.

3

Create two consumers with rebalance listener

Create two consumer instances called consumer1 and consumer2 using KafkaConsumer with consumer_props. Assign an instance of RebalanceListener to both consumers' subscribe method as the listener parameter for topic 'test-topic'.

Kafka

consumer_props = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'key.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer',
    'value.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer'
}

class RebalanceListener:
    def on_partitions_revoked(self, partitions):
        print('Partitions revoked:', partitions)

    def on_partitions_assigned(self, partitions):
        print('Partitions assigned:', partitions)

# Create consumer1 and consumer2 with consumer_props
# Subscribe both to 'test-topic' with RebalanceListener instance
# Your code here

Need a hint?

Use KafkaConsumer with consumer_props and subscribe both consumers to 'test-topic' with the rebalance listener.

4

Print partition assignments on rebalance events

Write code to simulate consumer1 polling messages once, then close consumer2 to trigger rebalance. Print 'Consumer1 polling...' before polling and 'Consumer2 closed to trigger rebalance' after closing consumer2.

Kafka

from kafka import KafkaConsumer

consumer_props = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'key.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer',
    'value.deserializer': 'org.apache.kafka.common.serialization.StringDeserializer'
}

class RebalanceListener:
    def on_partitions_revoked(self, partitions):
        print('Partitions revoked:', partitions)

    def on_partitions_assigned(self, partitions):
        print('Partitions assigned:', partitions)

listener = RebalanceListener()
consumer1 = KafkaConsumer(**consumer_props)
consumer1.subscribe(['test-topic'], listener=listener)
consumer2 = KafkaConsumer(**consumer_props)
consumer2.subscribe(['test-topic'], listener=listener)

# Simulate consumer1 polling and consumer2 closing to trigger rebalance
# Your code here

Need a hint?

Call consumer1.poll() and then consumer2.close() with print statements before and after.