Partition key and routing in Kafka - Mini Project: Build & Apply

Partition Key and Routing in Kafka

📖 Scenario: You are working on a messaging system using Kafka. You want to send messages to a topic where messages with the same key always go to the same partition. This helps keep related messages together for processing.

🎯 Goal: Build a simple Kafka producer script that sends messages with a partition key to a topic. You will create the message data, set a partition key, send the message, and print the result showing which partition the message went to.

📋 What You'll Learn

Create a message dictionary with specific key-value pairs

Set a partition key variable with a specific string

Send the message to a Kafka topic using the partition key

Print the partition number where the message was sent

💡 Why This Matters

🌍 Real World

In real applications, using partition keys helps keep related messages together, making processing and ordering easier.

💼 Career

Understanding partition keys and routing is essential for roles working with Kafka or distributed messaging systems.

1

Create the message data

Create a dictionary called message with these exact entries: 'user_id': 123, 'action': 'login', 'timestamp': '2024-06-01T12:00:00Z'.

Kafka

# Create the message dictionary with user_id, action, and timestamp
# Your code here

Need a hint?

Use curly braces to create a dictionary with the exact keys and values.

2

Set the partition key

Create a variable called partition_key and set it to the string 'user-123'.

Kafka

message = {'user_id': 123, 'action': 'login', 'timestamp': '2024-06-01T12:00:00Z'}
# Set the partition_key variable to 'user-123'
# Your code here

Need a hint?

Assign the string 'user-123' to the variable named partition_key.

3

Send the message with the partition key

Use the Kafka producer's send method to send the message to the topic 'user-actions' with the key set to partition_key.encode(). Store the result in a variable called future.

Kafka

message = {'user_id': 123, 'action': 'login', 'timestamp': '2024-06-01T12:00:00Z'}
partition_key = 'user-123'
from kafka import KafkaProducer
import json

producer = KafkaProducer(bootstrap_servers='localhost:9092', value_serializer=lambda v: json.dumps(v).encode('utf-8'))
# Send the message to 'user-actions' topic with key=partition_key.encode()
# Your code here

Need a hint?

Use producer.send with the topic name, message as value, and encoded partition_key as key.

4

Print the partition number

Call future.get() to get the send result. Then print the string "Message sent to partition: {partition}" where {partition} is replaced by the partition attribute of the result.

Kafka

message = {'user_id': 123, 'action': 'login', 'timestamp': '2024-06-01T12:00:00Z'}
partition_key = 'user-123'
from kafka import KafkaProducer
import json

producer = KafkaProducer(bootstrap_servers='localhost:9092', value_serializer=lambda v: json.dumps(v).encode('utf-8'))

future = producer.send('user-actions', value=message, key=partition_key.encode())
# Get the send result and print the partition number
# Your code here

Need a hint?

Use future.get() to wait for the send to complete, then print the partition number from the result.