Partition ordering guarantees in Kafka - Step-by-Step Execution

Process Flow - Partition ordering guarantees

Producer sends message

↓

Message assigned to partition

↓

Partition appends message in order

↓

Consumer reads messages from partition

↓

Messages delivered in order per partition

Messages sent by a producer go to a specific partition where they are stored in order. Consumers read from partitions in that same order, ensuring ordering per partition.

Execution Sample

Kafka

producer.send(topic, key='user1', value='msg1')
producer.send(topic, key='user1', value='msg2')
consumer.poll()

Producer sends two messages with the same key to the topic; consumer reads them in the order they were sent within the same partition.

Process Table

Step	Action	Partition Selected	Message Offset	Ordering Guarantee
1	Send msg1 with key 'user1'	Partition 2	0	First message in partition 2
2	Send msg2 with key 'user1'	Partition 2	1	Second message in partition 2, after msg1
3	Consumer polls messages	Partition 2	0	Reads msg1 first
4	Consumer polls messages	Partition 2	1	Reads msg2 second
5	Consumer polls messages	-	-	No more messages, end of partition

💡 All messages read in order from partition 2; ordering guaranteed per partition

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Partition 2 Offset	-	0 (msg1 stored)	1 (msg2 stored)	0 (msg1 read)	1 (msg2 read)	1 (all read)

Key Moments - 3 Insights

Why do messages with the same key go to the same partition?

Kafka uses the message key to determine the partition, ensuring all messages with the same key are stored in the same partition, preserving order as shown in steps 1 and 2.

Does Kafka guarantee ordering across different partitions?

No, ordering is only guaranteed within a single partition. Messages in different partitions can be processed in parallel and may arrive out of order.

What happens if a consumer reads from multiple partitions?

The consumer reads messages in order per partition, but messages from different partitions may be interleaved without global ordering.

Visual Quiz - 3 Questions

Look at the execution table, what is the offset of the second message sent with key 'user1'?

A 2

B 0

C 1

D Not assigned

💡 Hint

Check Step 2 in the execution table where msg2 is stored with offset 1

At which step does the consumer read the first message from partition 2?

A Step 1

B Step 3

C Step 4

D Step 5

💡 Hint

Look at Step 3 in the execution table where consumer polls and reads offset 0

If messages had different keys, how would the partition assignment change?

A Messages would be assigned to different partitions

B Messages would go to the same partition

C Messages would be lost

D Messages would be reordered within the same partition

💡 Hint

Kafka uses keys to assign partitions; different keys usually mean different partitions

Concept Snapshot

Kafka guarantees message order only within a single partition.
Messages with the same key go to the same partition.
Producers send messages that are appended in order.
Consumers read messages in the order stored per partition.
No ordering guarantee across different partitions.

Full Transcript

In Kafka, when a producer sends messages with the same key, those messages are assigned to the same partition. Each partition stores messages in the order they arrive, assigning them sequential offsets. Consumers read messages from partitions in offset order, preserving the order of messages per partition. This means ordering is guaranteed only within a partition, not across partitions. The execution table shows two messages sent with the same key going to partition 2 with offsets 0 and 1. The consumer reads them in that order. If messages have different keys, they may be assigned to different partitions, and ordering is not guaranteed across those partitions.

Practice

1. What does Kafka guarantee about message order when using partitions?

easy

A. Messages are ordered only within the same partition.

B. Messages are ordered across all partitions.

C. Messages are never ordered in Kafka.

D. Messages are ordered only if sent without keys.

Solution

1. Step 1: Understand Kafka partition concept

   Kafka divides topics into partitions, and each partition stores messages in order.

2. Step 2: Identify ordering guarantee scope

   Ordering is guaranteed only inside each partition, not across partitions.

3. Final Answer:

   Messages are ordered only within the same partition. -> Option A

4. Quick Check:

   Partition ordering = within partition only [OK]

Hint: Remember: order is per partition, not across partitions [OK]

Common Mistakes:

• Thinking Kafka orders messages across all partitions
• Assuming messages without keys are ordered
• Believing Kafka never guarantees order

2. Which Kafka producer code snippet correctly ensures messages with the same key go to the same partition?

easy

A. producer.send(new ProducerRecord<>(topic, key, value));

B. producer.send(new ProducerRecord<>(topic, null, value));

C. producer.send(new ProducerRecord<>(topic, value));

D. producer.send(new ProducerRecord<>(key, topic, value));

Solution

1. Step 1: Check ProducerRecord constructor usage

   The correct constructor to send with key is ProducerRecord(topic, key, value).

2. Step 2: Identify correct parameter order

   producer.send(new ProducerRecord<>(topic, key, value)); uses (topic, key, value) correctly; others misuse parameters or omit key.

3. Final Answer:

   producer.send(new ProducerRecord<>(topic, key, value)); -> Option A

4. Quick Check:

   Key in ProducerRecord = (topic, key, value) [OK]

Hint: Use ProducerRecord(topic, key, value) to keep keys [OK]

Common Mistakes:

• Omitting the key parameter
• Swapping topic and key positions
• Using null as key to preserve order

3. Given a Kafka topic with 3 partitions, what happens if you send messages with different keys in this code?

producer.send(new ProducerRecord<>("topic", "key1", "msg1"));
producer.send(new ProducerRecord<>("topic", "key2", "msg2"));
producer.send(new ProducerRecord<>("topic", "key1", "msg3"));

What is true about message order?

medium

A. All messages are ordered in the same partition.

B. "msg1" and "msg3" are ordered; "msg2" may be in a different partition.

C. "msg2" and "msg3" are ordered; "msg1" is unordered.

D. Messages are randomly ordered regardless of keys.

Solution

1. Step 1: Identify partition assignment by key

   Messages with the same key ("key1") go to the same partition; "key2" goes to another partition.

2. Step 2: Understand ordering within partitions

   "msg1" and "msg3" share "key1" so they keep order; "msg2" with "key2" is separate.

3. Final Answer:

   "msg1" and "msg3" are ordered; "msg2" may be in a different partition. -> Option B

4. Quick Check:

   Same key = same partition order [OK]

Hint: Same key means same partition and order [OK]

Common Mistakes:

• Assuming all messages go to one partition
• Thinking different keys keep order together
• Believing order is global across partitions

4. You have this Kafka producer code snippet:

producer.send(new ProducerRecord<>("topic", null, "msg1"));
producer.send(new ProducerRecord<>("topic", null, "msg2"));

Why might message order not be preserved?

medium

A. Because the topic has only one partition.

B. Because Kafka always reorders messages regardless of keys.

C. Because messages with null keys can go to different partitions, losing order.

D. Because the producer is synchronous.

Solution

1. Step 1: Analyze key usage in ProducerRecord

   Null keys mean Kafka uses round-robin or default partitioner, sending messages to different partitions.

2. Step 2: Understand ordering impact

   Messages in different partitions have no ordering guarantee, so order may be lost.

3. Final Answer:

   Because messages with null keys can go to different partitions, losing order. -> Option C

4. Quick Check:

   Null key = no partition order guarantee [OK]

Hint: Null keys cause messages to spread partitions, breaking order [OK]

Common Mistakes:

• Thinking Kafka always orders messages
• Assuming one partition means no order loss
• Confusing synchronous with ordering guarantee

5. You want to ensure strict message order for user events in Kafka. You have multiple users sending events simultaneously. Which approach best guarantees order per user?

hard

A. Send all messages without keys to a single partition topic.

B. Use multiple topics, one per user, to keep order.

C. Use random keys to balance load and rely on consumer ordering.

D. Use the user ID as the message key so all events for a user go to the same partition.

Solution

1. Step 1: Identify how Kafka partitions messages

   Kafka uses the key to assign messages to partitions; same key means same partition.

2. Step 2: Apply to user event ordering

   Using user ID as key ensures all events for that user go to one partition, preserving order.

3. Step 3: Evaluate other options

   Sending without keys or random keys breaks order; multiple topics is complex and inefficient.

4. Final Answer:

   Use the user ID as the message key so all events for a user go to the same partition. -> Option D

5. Quick Check:

   User ID key = per-user order guaranteed [OK]

Hint: Key by user ID to keep their events ordered [OK]

Common Mistakes:

• Sending messages without keys expecting order
• Using random keys and expecting order
• Creating many topics instead of using keys