Partitioner behavior in Kafka - Mini Project: Build & Apply

Partitioner Behavior in Kafka

📖 Scenario: You are working with Apache Kafka, a system that helps send messages between different parts of an application. Messages are sent to topics, which are divided into partitions. How messages are assigned to partitions depends on the partitioner.Understanding partitioner behavior helps ensure messages are distributed correctly for processing.

🎯 Goal: Build a simple Kafka producer that sends messages to a topic using a custom partitioner. You will create the data to send, configure the partitioner, implement the partitioning logic, and then print which partition each message goes to.

📋 What You'll Learn

Create a list of messages with exact values

Create a configuration variable for the number of partitions

Implement a custom partitioner function that assigns partitions based on message key length

Print the message and its assigned partition

💡 Why This Matters

🌍 Real World

Kafka is used in many applications to handle streams of data. Understanding how partitioners work helps distribute data evenly and maintain order where needed.

💼 Career

Kafka developers and data engineers often customize partitioners to optimize data flow and processing in real-time systems.

1

Create the list of messages

Create a list called messages with these exact string values: 'apple', 'banana', 'cherry', 'date', 'elderberry'.

Kafka

# Create the list of messages
# Your code here

Need a hint?

Use square brackets [] to create a list and separate items with commas.

2

Set the number of partitions

Create a variable called num_partitions and set it to 3.

Kafka

messages = ['apple', 'banana', 'cherry', 'date', 'elderberry']
# Set the number of partitions
# Your code here

Need a hint?

Just assign the number 3 to the variable num_partitions.

3

Implement the custom partitioner function

Define a function called custom_partitioner that takes a message string and returns the partition number by calculating len(message) % num_partitions.

Kafka

messages = ['apple', 'banana', 'cherry', 'date', 'elderberry']
num_partitions = 3
# Define the custom_partitioner function
# Your code here

Need a hint?

Use the len() function to get the length of the message string.

4

Print each message with its assigned partition

Use a for loop with variable msg to iterate over messages. Inside the loop, print the message and its partition using custom_partitioner(msg) in this exact format: Message: {msg}, Partition: {partition}.

Kafka

messages = ['apple', 'banana', 'cherry', 'date', 'elderberry']
num_partitions = 3

def custom_partitioner(message):
    return len(message) % num_partitions

# Print each message with its partition
# Your code here

Need a hint?

Use an f-string to format the print output exactly as shown.