Group coordinator in Kafka - Mini Project: Build & Apply

Group Coordinator in Kafka

📖 Scenario: You are working with Apache Kafka, a system that helps many computers talk to each other by sending messages. In Kafka, a group coordinator is like a team leader who manages a group of computers (called consumers) that read messages together.Understanding how the group coordinator works helps keep the message reading organized and balanced.

🎯 Goal: Build a simple Kafka consumer group setup where you create a group coordinator to manage consumers reading messages from a topic.You will create the initial consumer group data, configure the group coordinator ID, implement the logic to assign partitions to consumers, and finally print the assignment result.

📋 What You'll Learn

Create a dictionary called consumers with these exact entries: 'consumer1': [], 'consumer2': [], 'consumer3': []

Create a variable called group_coordinator_id and set it to 1

Use a for loop to assign partitions from partitions_list to consumers in a round-robin way

Print the consumers dictionary to show the final partition assignments

💡 Why This Matters

🌍 Real World

Kafka uses group coordinators to manage consumer groups reading messages from topics, ensuring balanced load and fault tolerance.

💼 Career

Understanding group coordinators is important for roles like Kafka administrator, backend developer, or data engineer working with real-time data pipelines.

1

DATA SETUP: Create the consumers dictionary

Create a dictionary called consumers with these exact entries: 'consumer1': [], 'consumer2': [], 'consumer3': []

Kafka

# Create the consumers dictionary with empty lists
# Your code here

Need a hint?

Use curly braces {} to create a dictionary. Each consumer name is a key with an empty list as its value.

2

CONFIGURATION: Set the group coordinator ID

Create a variable called group_coordinator_id and set it to 1

Kafka

consumers = {'consumer1': [], 'consumer2': [], 'consumer3': []}
# Set the group coordinator ID
# Your code here

Need a hint?

Just assign the number 1 to the variable group_coordinator_id.

3

CORE LOGIC: Assign partitions to consumers

Create a list called partitions_list with these exact values: 0, 1, 2, 3, 4, 5. Then use a for loop to assign partitions from partitions_list to consumers in a round-robin way so each consumer gets partitions evenly.

Kafka

consumers = {'consumer1': [], 'consumer2': [], 'consumer3': []}
group_coordinator_id = 1
# Create partitions_list and assign partitions to consumers in round-robin
# Your code here

Need a hint?

Use enumerate() to get index and partition. Use modulo % to cycle through consumers.

4

OUTPUT: Print the consumers with their assigned partitions

Write print(consumers) to display the dictionary showing each consumer and their assigned partitions.

Kafka

consumers = {'consumer1': [], 'consumer2': [], 'consumer3': []}
group_coordinator_id = 1
partitions_list = [0, 1, 2, 3, 4, 5]
for i, partition in enumerate(partitions_list):
    consumer = list(consumers.keys())[i % len(consumers)]
    consumers[consumer].append(partition)
# Print the consumers dictionary
# Your code here

Need a hint?

Use print(consumers) to show the final assignments.