Partition assignment in Kafka - Mini Project: Build & Apply

Partition assignment

📖 Scenario: You are managing a Kafka topic with multiple partitions. You want to assign partitions to consumers in a group so that each consumer handles specific partitions.

🎯 Goal: Build a simple Kafka consumer group partition assignment simulation using Python dictionaries. You will create a data structure for partitions, configure consumers, assign partitions to consumers, and display the assignments.

📋 What You'll Learn

Create a dictionary with partitions and their IDs

Create a list of consumer IDs

Assign partitions to consumers evenly

Print the final partition assignment

💡 Why This Matters

🌍 Real World

Kafka partition assignment is important to distribute workload evenly among consumers in a group, ensuring efficient data processing.

💼 Career

Understanding partition assignment helps in roles like data engineering, backend development, and system architecture where Kafka is used for streaming data.

1

Create partitions dictionary

Create a dictionary called partitions with these exact entries: 0: 'partition_0', 1: 'partition_1', 2: 'partition_2', 3: 'partition_3', 4: 'partition_4'

Kafka

# Your code here

Need a hint?

Use curly braces to create a dictionary with keys 0 to 4 and values as 'partition_0' to 'partition_4'.

2

Create consumers list

Create a list called consumers with these exact string values: 'consumer_1', 'consumer_2', 'consumer_3'

Kafka

partitions = {0: 'partition_0', 1: 'partition_1', 2: 'partition_2', 3: 'partition_3', 4: 'partition_4'}
# Your code here

Need a hint?

Use square brackets to create a list with the three consumer strings.

3

Assign partitions to consumers

Create a dictionary called assignment that assigns partitions to consumers evenly. Use a for loop with variables partition_id and partition_name to iterate over partitions.items(). Assign each partition to a consumer by cycling through consumers using the modulo operator. Store the result as a list of partition names for each consumer.

Kafka

partitions = {0: 'partition_0', 1: 'partition_1', 2: 'partition_2', 3: 'partition_3', 4: 'partition_4'}
consumers = ['consumer_1', 'consumer_2', 'consumer_3']
# Your code here

Need a hint?

Initialize assignment as a dictionary with empty lists for each consumer. Use modulo to cycle through consumers.

4

Print partition assignment

Write a print statement to display the assignment dictionary.

Kafka

partitions = {0: 'partition_0', 1: 'partition_1', 2: 'partition_2', 3: 'partition_3', 4: 'partition_4'}
consumers = ['consumer_1', 'consumer_2', 'consumer_3']
assignment = {consumer: [] for consumer in consumers}
for partition_id, partition_name in partitions.items():
    consumer = consumers[partition_id % len(consumers)]
    assignment[consumer].append(partition_name)
# Your code here

Need a hint?

Use print(assignment) to show the final dictionary.