Cooperative vs eager rebalancing in Kafka - Hands-On Comparison

Understanding Cooperative vs Eager Rebalancing in Kafka Consumer Groups

📖 Scenario: You are working with Apache Kafka, a system that helps many programs share messages efficiently. When multiple programs (called consumers) read messages together in a group, Kafka needs to decide who reads which messages. This decision process is called rebalancing.There are two main ways Kafka does rebalancing: eager and cooperative. Eager rebalancing stops all consumers and quickly assigns new message parts, while cooperative rebalancing tries to make changes smoothly without stopping everything.

🎯 Goal: You will create a simple Kafka consumer group setup in code to see how to configure and understand the difference between cooperative and eager rebalancing.

📋 What You'll Learn

Create a Kafka consumer configuration dictionary with group ID and bootstrap servers

Add a configuration setting to choose the rebalance protocol (cooperative or eager)

Write code to simulate subscribing to a topic with the chosen rebalance protocol

Print the rebalance protocol used to confirm the setup

💡 Why This Matters

🌍 Real World

Kafka consumer groups are used in many applications to process data streams efficiently. Understanding rebalancing helps keep data processing smooth and fast.

💼 Career

Knowing how to configure and manage Kafka consumer groups is important for roles in data engineering, backend development, and system architecture.

1

Create Kafka consumer configuration

Create a dictionary called consumer_config with these exact entries: 'bootstrap.servers': 'localhost:9092' and 'group.id': 'test-group'.

Kafka

# Create the consumer_config dictionary with bootstrap.servers and group.id
# Your code here

Hint

Use curly braces to create a dictionary and include the exact keys and values as strings.

2

Add rebalance protocol configuration

Add a new entry to the consumer_config dictionary with key 'partition.assignment.strategy' and value 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor' to use cooperative rebalancing.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group'
}
# Add the partition.assignment.strategy for cooperative rebalancing
# Your code here

Hint

Add the new key and value inside the dictionary, separated by a comma.

3

Simulate subscribing to a topic

Write a function called simulate_consumer_subscribe that takes config as a parameter and returns the value of config['partition.assignment.strategy']. Then call this function with consumer_config and store the result in a variable called rebalance_protocol.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'partition.assignment.strategy': 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor'
}
# Define simulate_consumer_subscribe function and call it
# Your code here

Hint

Define a function that returns the rebalance strategy from the config dictionary, then call it.

4

Print the rebalance protocol

Write a print statement to display the text "Rebalance protocol used: " followed by the value of rebalance_protocol.

Kafka

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'test-group',
    'partition.assignment.strategy': 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor'
}

def simulate_consumer_subscribe(config):
    return config['partition.assignment.strategy']

rebalance_protocol = simulate_consumer_subscribe(consumer_config)
# Print the rebalance protocol used
# Your code here

Hint

Use an f-string to include the variable in the printed message.

Practice

1. What is the main advantage of cooperative rebalancing in Kafka consumer groups?

easy

A. It reduces consumer pause time by reassigning partitions gradually.

B. It reassigns all partitions at once for faster processing.

C. It eliminates the need for consumer group coordination.

D. It increases the number of partitions automatically.

Solution

1. Step 1: Understand cooperative rebalancing behavior

   Cooperative rebalancing allows consumers to leave partitions gradually, which reduces the pause time during rebalancing.

2. Step 2: Compare with eager rebalancing

   Eager rebalancing reassigns all partitions at once, causing longer pauses, unlike cooperative rebalancing.

3. Final Answer:

   It reduces consumer pause time by reassigning partitions gradually. -> Option A

4. Quick Check:

   Cooperative rebalancing = gradual partition reassignment [OK]

Hint: Cooperative means gradual, eager means all at once [OK]

Common Mistakes:

• Confusing eager with cooperative rebalancing
• Thinking cooperative rebalancing eliminates coordination
• Assuming rebalancing increases partitions

2. Which configuration property enables cooperative rebalancing in Kafka consumers?

easy

A. group.rebalance.strategy = 'eager'

B. rebalance.policy = 'cooperative'

C. enable.cooperative.rebalance = true

D. partition.assignment.strategy = 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor'

Solution

1. Step 1: Identify the correct property for assignment strategy

   Kafka uses partition.assignment.strategy to set the rebalance strategy class.

2. Step 2: Recognize the cooperative assignor class

   The cooperative rebalancing strategy is implemented by org.apache.kafka.clients.consumer.CooperativeStickyAssignor.

3. Final Answer:

   partition.assignment.strategy = 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor' -> Option D

4. Quick Check:

   Cooperative assignor class = partition.assignment.strategy = 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor' [OK]

Hint: Look for 'partition.assignment.strategy' with cooperative class [OK]

Common Mistakes:

• Using incorrect property names
• Confusing eager and cooperative strategy names
• Assuming a boolean flag enables cooperative rebalancing

3. Given two consumers in a group using eager rebalancing, what happens when one consumer leaves?

medium

A. All partitions in the group are reassigned immediately, causing a pause.

B. Only the partitions of the leaving consumer are reassigned gradually.

C. The remaining consumer continues without any rebalancing.

D. Partitions are reassigned only after a timeout period.

Solution

1. Step 1: Understand eager rebalancing behavior on consumer leave

   In eager rebalancing, when any consumer leaves, all partitions are reassigned immediately to maintain balance.

2. Step 2: Recognize the impact on consumer pause

   This immediate reassignment causes a pause for all consumers during the rebalance.

3. Final Answer:

   All partitions in the group are reassigned immediately, causing a pause. -> Option A

4. Quick Check:

   Eager rebalancing = full immediate reassignment [OK]

Hint: Eager means all partitions reassign at once [OK]

Common Mistakes:

• Thinking only leaving consumer's partitions reassign
• Assuming no pause occurs
• Believing reassignment waits for timeout

4. You have a Kafka consumer group using cooperative rebalancing but notice long pauses during rebalances. What is a likely cause?

medium

A. Partitions are being reassigned gradually as expected.

B. The consumer group has only one consumer.

C. Consumers are using the eager assignor instead of cooperative assignor.

D. The rebalance timeout is set too low.

Solution

1. Step 1: Check the assignor configuration

   If long pauses occur despite expecting cooperative rebalancing, the consumers might be using the eager assignor by mistake.

2. Step 2: Understand the impact of eager assignor

   Eager assignor causes all partitions to reassign at once, leading to longer pauses.

3. Final Answer:

   Consumers are using the eager assignor instead of cooperative assignor. -> Option C

4. Quick Check:

   Long pauses + cooperative expected = eager assignor used [OK]

Hint: Long pauses mean eager assignor, not cooperative [OK]

Common Mistakes:

• Assuming gradual reassignment causes long pauses
• Ignoring assignor configuration
• Blaming rebalance timeout without checking assignor

5. You want to minimize downtime in a Kafka consumer group with frequent consumer joins and leaves. Which strategy and configuration should you choose?

hard

A. Use eager rebalancing with default assignor for simplicity.

B. Use cooperative rebalancing with CooperativeStickyAssignor to reduce pause time.

C. Disable rebalancing to avoid any pauses.

D. Use eager rebalancing but increase session timeout to reduce rebalances.

Solution

1. Step 1: Identify the goal

   The goal is to minimize downtime during frequent consumer group changes.

2. Step 2: Choose the appropriate rebalance strategy

   Cooperative rebalancing reduces pause time by reassigning partitions gradually, which suits frequent joins/leaves.

3. Step 3: Select the correct configuration

   Using CooperativeStickyAssignor enables cooperative rebalancing in Kafka consumers.

4. Final Answer:

   Use cooperative rebalancing with CooperativeStickyAssignor to reduce pause time. -> Option B

5. Quick Check:

   Minimize downtime = cooperative rebalancing + correct assignor [OK]

Hint: CooperativeStickyAssignor minimizes downtime with frequent changes [OK]

Common Mistakes:

• Choosing eager rebalancing for low downtime
• Disabling rebalancing causing stale assignments
• Misconfiguring session timeout instead of strategy