Partition ordering guarantees in Kafka - Mini Project: Build & Apply

Partition Ordering Guarantees in Kafka

📖 Scenario: You are building a simple Kafka producer and consumer to understand how Kafka guarantees message order within partitions.

🎯 Goal: Create a Kafka producer that sends messages with keys to ensure ordering within partitions, then create a consumer that reads and prints messages in the order they were produced per partition.

📋 What You'll Learn

Create a Kafka producer that sends messages with keys

Configure the producer to send messages to specific partitions based on keys

Create a Kafka consumer that subscribes to the topic

Consume and print messages maintaining partition order

💡 Why This Matters

🌍 Real World

Kafka is widely used in real-time data pipelines and event-driven systems where message order within a partition is critical for data consistency.

💼 Career

Understanding partition ordering guarantees is essential for roles involving Kafka administration, backend development, and data engineering to build reliable streaming applications.

1

Create Kafka topic and produce messages with keys

Create a Kafka producer in Python using confluent_kafka.Producer that sends these exact messages with keys to the topic test-topic: (key='user1', value='message1'), (key='user2', value='message2'), (key='user1', value='message3').

Kafka

# Your code here to create a producer and send messages with keys

Hint

Use Producer from confluent_kafka. Send messages with keys encoded as bytes.

2

Configure producer for partition ordering by key

Add a configuration to the producer to ensure messages with the same key go to the same partition by setting partitioner to consistent in the producer configuration.

Kafka

from confluent_kafka import Producer

# Create producer with partitioner configuration
producer = Producer({'bootstrap.servers': 'localhost:9092', 'partitioner': 'consistent'})

messages = [('user1', 'message1'), ('user2', 'message2'), ('user1', 'message3')]

for key, value in messages:
    producer.produce('test-topic', key=key.encode('utf-8'), value=value.encode('utf-8'))

producer.flush()

Hint

Set 'partitioner': 'consistent' in the producer config dictionary.

3

Create Kafka consumer to read messages from topic

Create a Kafka consumer using confluent_kafka.Consumer that subscribes to test-topic and polls messages in a loop, decoding keys and values as UTF-8 strings.

Kafka

from confluent_kafka import Producer, Consumer

producer = Producer({'bootstrap.servers': 'localhost:9092', 'partitioner': 'consistent'})

messages = [('user1', 'message1'), ('user2', 'message2'), ('user1', 'message3')]

for key, value in messages:
    producer.produce('test-topic', key=key.encode('utf-8'), value=value.encode('utf-8'))

producer.flush()

# Create consumer and subscribe to 'test-topic'
consumer = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'group1', 'auto.offset.reset': 'earliest'})
consumer.subscribe(['test-topic'])

# Poll messages
for _ in range(3):
    msg = consumer.poll(1.0)
    if msg is not None and not msg.error():
        key = msg.key().decode('utf-8') if msg.key() else None
        value = msg.value().decode('utf-8')
        print(f"Key: {key}, Value: {value}")

consumer.close()

Hint

Use Consumer with group.id and auto.offset.reset. Poll messages and decode keys and values.

4

Print messages maintaining partition order

Run the consumer code to print the messages. The output should show messages with the same key in the order they were produced, demonstrating partition ordering guarantees.

Kafka

# Run the code above and observe the printed output
# Your code here

Hint

The printed messages should show that messages with the same key appear in the order sent.

Practice

1. What does Kafka guarantee about message order when using partitions?

easy

A. Messages are ordered only within the same partition.

B. Messages are ordered across all partitions.

C. Messages are never ordered in Kafka.

D. Messages are ordered only if sent without keys.

Solution

1. Step 1: Understand Kafka partition concept

   Kafka divides topics into partitions, and each partition stores messages in order.

2. Step 2: Identify ordering guarantee scope

   Ordering is guaranteed only inside each partition, not across partitions.

3. Final Answer:

   Messages are ordered only within the same partition. -> Option A

4. Quick Check:

   Partition ordering = within partition only [OK]

Hint: Remember: order is per partition, not across partitions [OK]

Common Mistakes:

• Thinking Kafka orders messages across all partitions
• Assuming messages without keys are ordered
• Believing Kafka never guarantees order

2. Which Kafka producer code snippet correctly ensures messages with the same key go to the same partition?

easy

A. producer.send(new ProducerRecord<>(topic, key, value));

B. producer.send(new ProducerRecord<>(topic, null, value));

C. producer.send(new ProducerRecord<>(topic, value));

D. producer.send(new ProducerRecord<>(key, topic, value));

Solution

1. Step 1: Check ProducerRecord constructor usage

   The correct constructor to send with key is ProducerRecord(topic, key, value).

2. Step 2: Identify correct parameter order

   producer.send(new ProducerRecord<>(topic, key, value)); uses (topic, key, value) correctly; others misuse parameters or omit key.

3. Final Answer:

   producer.send(new ProducerRecord<>(topic, key, value)); -> Option A

4. Quick Check:

   Key in ProducerRecord = (topic, key, value) [OK]

Hint: Use ProducerRecord(topic, key, value) to keep keys [OK]

Common Mistakes:

• Omitting the key parameter
• Swapping topic and key positions
• Using null as key to preserve order

3. Given a Kafka topic with 3 partitions, what happens if you send messages with different keys in this code?

producer.send(new ProducerRecord<>("topic", "key1", "msg1"));
producer.send(new ProducerRecord<>("topic", "key2", "msg2"));
producer.send(new ProducerRecord<>("topic", "key1", "msg3"));

What is true about message order?

medium

A. All messages are ordered in the same partition.

B. "msg1" and "msg3" are ordered; "msg2" may be in a different partition.

C. "msg2" and "msg3" are ordered; "msg1" is unordered.

D. Messages are randomly ordered regardless of keys.

Solution

1. Step 1: Identify partition assignment by key

   Messages with the same key ("key1") go to the same partition; "key2" goes to another partition.

2. Step 2: Understand ordering within partitions

   "msg1" and "msg3" share "key1" so they keep order; "msg2" with "key2" is separate.

3. Final Answer:

   "msg1" and "msg3" are ordered; "msg2" may be in a different partition. -> Option B

4. Quick Check:

   Same key = same partition order [OK]

Hint: Same key means same partition and order [OK]

Common Mistakes:

• Assuming all messages go to one partition
• Thinking different keys keep order together
• Believing order is global across partitions

4. You have this Kafka producer code snippet:

producer.send(new ProducerRecord<>("topic", null, "msg1"));
producer.send(new ProducerRecord<>("topic", null, "msg2"));

Why might message order not be preserved?

medium

A. Because the topic has only one partition.

B. Because Kafka always reorders messages regardless of keys.

C. Because messages with null keys can go to different partitions, losing order.

D. Because the producer is synchronous.

Solution

1. Step 1: Analyze key usage in ProducerRecord

   Null keys mean Kafka uses round-robin or default partitioner, sending messages to different partitions.

2. Step 2: Understand ordering impact

   Messages in different partitions have no ordering guarantee, so order may be lost.

3. Final Answer:

   Because messages with null keys can go to different partitions, losing order. -> Option C

4. Quick Check:

   Null key = no partition order guarantee [OK]

Hint: Null keys cause messages to spread partitions, breaking order [OK]

Common Mistakes:

• Thinking Kafka always orders messages
• Assuming one partition means no order loss
• Confusing synchronous with ordering guarantee

5. You want to ensure strict message order for user events in Kafka. You have multiple users sending events simultaneously. Which approach best guarantees order per user?

hard

A. Send all messages without keys to a single partition topic.

B. Use multiple topics, one per user, to keep order.

C. Use random keys to balance load and rely on consumer ordering.

D. Use the user ID as the message key so all events for a user go to the same partition.

Solution

1. Step 1: Identify how Kafka partitions messages

   Kafka uses the key to assign messages to partitions; same key means same partition.

2. Step 2: Apply to user event ordering

   Using user ID as key ensures all events for that user go to one partition, preserving order.

3. Step 3: Evaluate other options

   Sending without keys or random keys breaks order; multiple topics is complex and inefficient.

4. Final Answer:

   Use the user ID as the message key so all events for a user go to the same partition. -> Option D

5. Quick Check:

   User ID key = per-user order guaranteed [OK]

Hint: Key by user ID to keep their events ordered [OK]

Common Mistakes:

• Sending messages without keys expecting order
• Using random keys and expecting order
• Creating many topics instead of using keys