Partition ordering guarantees in Kafka - Mini Project: Build & Apply

Partition Ordering Guarantees in Kafka

📖 Scenario: You are building a simple Kafka producer and consumer to understand how Kafka guarantees message order within partitions.

🎯 Goal: Create a Kafka producer that sends messages with keys to ensure ordering within partitions, then create a consumer that reads and prints messages in the order they were produced per partition.

📋 What You'll Learn

Create a Kafka producer that sends messages with keys

Configure the producer to send messages to specific partitions based on keys

Create a Kafka consumer that subscribes to the topic

Consume and print messages maintaining partition order

💡 Why This Matters

🌍 Real World

Kafka is widely used in real-time data pipelines and event-driven systems where message order within a partition is critical for data consistency.

💼 Career

Understanding partition ordering guarantees is essential for roles involving Kafka administration, backend development, and data engineering to build reliable streaming applications.

1

Create Kafka topic and produce messages with keys

Create a Kafka producer in Python using confluent_kafka.Producer that sends these exact messages with keys to the topic test-topic: (key='user1', value='message1'), (key='user2', value='message2'), (key='user1', value='message3').

Kafka

# Your code here to create a producer and send messages with keys

Need a hint?

Use Producer from confluent_kafka. Send messages with keys encoded as bytes.

2

Configure producer for partition ordering by key

Add a configuration to the producer to ensure messages with the same key go to the same partition by setting partitioner to consistent in the producer configuration.

Kafka

from confluent_kafka import Producer

# Create producer with partitioner configuration
producer = Producer({'bootstrap.servers': 'localhost:9092', 'partitioner': 'consistent'})

messages = [('user1', 'message1'), ('user2', 'message2'), ('user1', 'message3')]

for key, value in messages:
    producer.produce('test-topic', key=key.encode('utf-8'), value=value.encode('utf-8'))

producer.flush()

Need a hint?

Set 'partitioner': 'consistent' in the producer config dictionary.

3

Create Kafka consumer to read messages from topic

Create a Kafka consumer using confluent_kafka.Consumer that subscribes to test-topic and polls messages in a loop, decoding keys and values as UTF-8 strings.

Kafka

from confluent_kafka import Producer, Consumer

producer = Producer({'bootstrap.servers': 'localhost:9092', 'partitioner': 'consistent'})

messages = [('user1', 'message1'), ('user2', 'message2'), ('user1', 'message3')]

for key, value in messages:
    producer.produce('test-topic', key=key.encode('utf-8'), value=value.encode('utf-8'))

producer.flush()

# Create consumer and subscribe to 'test-topic'
consumer = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'group1', 'auto.offset.reset': 'earliest'})
consumer.subscribe(['test-topic'])

# Poll messages
for _ in range(3):
    msg = consumer.poll(1.0)
    if msg is not None and not msg.error():
        key = msg.key().decode('utf-8') if msg.key() else None
        value = msg.value().decode('utf-8')
        print(f"Key: {key}, Value: {value}")

consumer.close()

Need a hint?

Use Consumer with group.id and auto.offset.reset. Poll messages and decode keys and values.

4

Print messages maintaining partition order

Run the consumer code to print the messages. The output should show messages with the same key in the order they were produced, demonstrating partition ordering guarantees.

Kafka

# Run the code above and observe the printed output
# Your code here

Need a hint?

The printed messages should show that messages with the same key appear in the order sent.