Cooperative vs eager rebalancing in Kafka - Hands-On Comparison

Understanding Cooperative vs Eager Rebalancing in Kafka Consumer Groups

📖 Scenario: You are working with Apache Kafka, a system that helps many programs share messages efficiently. When multiple programs (called consumers) read messages together in a group, Kafka needs to decide who reads which messages. This decision process is called rebalancing.There are two main ways Kafka does rebalancing: eager and cooperative. Eager rebalancing stops all consumers and quickly assigns new message parts, while cooperative rebalancing tries to make changes smoothly without stopping everything.

🎯 Goal: You will create a simple Kafka consumer group setup in code to see how to configure and understand the difference between cooperative and eager rebalancing.

📋 What You'll Learn

Create a Kafka consumer configuration dictionary with group ID and bootstrap servers

Add a configuration setting to choose the rebalance protocol (cooperative or eager)

Write code to simulate subscribing to a topic with the chosen rebalance protocol

Print the rebalance protocol used to confirm the setup

💡 Why This Matters

🌍 Real World

Kafka consumer groups are used in many applications to process data streams efficiently. Understanding rebalancing helps keep data processing smooth and fast.

💼 Career

Knowing how to configure and manage Kafka consumer groups is important for roles in data engineering, backend development, and system architecture.

1

Create Kafka consumer configuration

Create a dictionary called consumer_config with these exact entries: 'bootstrap.servers': 'localhost:9092' and 'group.id': 'test-group'.

Kafka

# Create the consumer_config dictionary with bootstrap.servers and group.id
# Your code here

Need a hint?

Use curly braces to create a dictionary and include the exact keys and values as strings.

2

Add rebalance protocol configuration

Add a new entry to the consumer_config dictionary with key 'partition.assignment.strategy' and value 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor' to use cooperative rebalancing.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group'
}
# Add the partition.assignment.strategy for cooperative rebalancing
# Your code here

Need a hint?

Add the new key and value inside the dictionary, separated by a comma.

3

Simulate subscribing to a topic

Write a function called simulate_consumer_subscribe that takes config as a parameter and returns the value of config['partition.assignment.strategy']. Then call this function with consumer_config and store the result in a variable called rebalance_protocol.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'partition.assignment.strategy': 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor'
}
# Define simulate_consumer_subscribe function and call it
# Your code here

Need a hint?

Define a function that returns the rebalance strategy from the config dictionary, then call it.

4

Print the rebalance protocol

Write a print statement to display the text "Rebalance protocol used: " followed by the value of rebalance_protocol.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'partition.assignment.strategy': 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor'
}

def simulate_consumer_subscribe(config):
    return config['partition.assignment.strategy']

rebalance_protocol = simulate_consumer_subscribe(consumer_config)
# Print the rebalance protocol used
# Your code here

Need a hint?

Use an f-string to include the variable in the printed message.