Partition ordering guarantees in Kafka - Commands & Configuration

Introduction

When sending messages in Kafka, it is important to know how the order of messages is kept. Partition ordering guarantees explain how Kafka keeps messages in the same order within a partition, so consumers read them in the same sequence they were sent.

When you want to process events in the exact order they happened, like user actions or transactions.

When you have multiple producers sending messages but need to keep order for each key or category.

When building systems that rely on ordered logs or event streams for accurate results.

When troubleshooting message order issues in your Kafka consumers.

When designing topics and partitions to ensure correct message flow.

Commands

Create a Kafka topic named 'example-topic' with 3 partitions to distribute messages and allow ordering within each partition.

Terminal

kafka-topics.sh --create --topic example-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic example-topic.

→

--partitions - Sets number of partitions for the topic

→

--replication-factor - Sets how many copies of data are kept for fault tolerance

Start a producer that sends messages with keys separated by ':' to ensure messages with the same key go to the same partition, preserving order per key.

Terminal

kafka-console-producer.sh --topic example-topic --bootstrap-server localhost:9092 --property parse.key=true --property key.separator=:

Expected Output

No output (command runs silently)

→

parse.key=true - Enables key parsing for messages

→

key.separator=: - Defines the separator between key and message

Start a consumer that reads all messages from the beginning, showing keys to verify the order of messages per partition.

Terminal

kafka-console-consumer.sh --topic example-topic --bootstrap-server localhost:9092 --from-beginning --property print.key=true --property key.separator=:

Expected Output

key1:message1

key1:message2

key2:message3

key1:message4

→

--from-beginning - Reads messages from the start of the topic

→

print.key=true - Shows the key of each message in output

Key Concept

Kafka guarantees that messages with the same key are stored and read in order within the same partition.

Common Mistakes

Sending messages without keys expecting global ordering.

Without keys, messages are distributed across partitions randomly, so order is only guaranteed within each partition, not across the whole topic.

Always send messages with keys when order matters, so Kafka routes them to the same partition.

Using too few partitions and expecting high throughput.

Fewer partitions limit parallelism and throughput, but more partitions can split keys across partitions if keys are not used properly.

Balance the number of partitions and use keys to control ordering and throughput.

Summary

Create topics with multiple partitions to allow parallel processing and ordering within partitions.

Send messages with keys so Kafka routes them to the same partition, preserving order per key.

Consume messages from the beginning with keys shown to verify ordering guarantees.

Practice

1. What does Kafka guarantee about message order when using partitions?

easy

A. Messages are ordered only within the same partition.

B. Messages are ordered across all partitions.

C. Messages are never ordered in Kafka.

D. Messages are ordered only if sent without keys.

Solution

1. Step 1: Understand Kafka partition concept

   Kafka divides topics into partitions, and each partition stores messages in order.

2. Step 2: Identify ordering guarantee scope

   Ordering is guaranteed only inside each partition, not across partitions.

3. Final Answer:

   Messages are ordered only within the same partition. -> Option A

4. Quick Check:

   Partition ordering = within partition only [OK]

Hint: Remember: order is per partition, not across partitions [OK]

Common Mistakes:

• Thinking Kafka orders messages across all partitions
• Assuming messages without keys are ordered
• Believing Kafka never guarantees order

2. Which Kafka producer code snippet correctly ensures messages with the same key go to the same partition?

easy

A. producer.send(new ProducerRecord<>(topic, key, value));

B. producer.send(new ProducerRecord<>(topic, null, value));

C. producer.send(new ProducerRecord<>(topic, value));

D. producer.send(new ProducerRecord<>(key, topic, value));

Solution

1. Step 1: Check ProducerRecord constructor usage

   The correct constructor to send with key is ProducerRecord(topic, key, value).

2. Step 2: Identify correct parameter order

   producer.send(new ProducerRecord<>(topic, key, value)); uses (topic, key, value) correctly; others misuse parameters or omit key.

3. Final Answer:

   producer.send(new ProducerRecord<>(topic, key, value)); -> Option A

4. Quick Check:

   Key in ProducerRecord = (topic, key, value) [OK]

Hint: Use ProducerRecord(topic, key, value) to keep keys [OK]

Common Mistakes:

• Omitting the key parameter
• Swapping topic and key positions
• Using null as key to preserve order

3. Given a Kafka topic with 3 partitions, what happens if you send messages with different keys in this code?

producer.send(new ProducerRecord<>("topic", "key1", "msg1"));
producer.send(new ProducerRecord<>("topic", "key2", "msg2"));
producer.send(new ProducerRecord<>("topic", "key1", "msg3"));

What is true about message order?

medium

A. All messages are ordered in the same partition.

B. "msg1" and "msg3" are ordered; "msg2" may be in a different partition.

C. "msg2" and "msg3" are ordered; "msg1" is unordered.

D. Messages are randomly ordered regardless of keys.

Solution

1. Step 1: Identify partition assignment by key

   Messages with the same key ("key1") go to the same partition; "key2" goes to another partition.

2. Step 2: Understand ordering within partitions

   "msg1" and "msg3" share "key1" so they keep order; "msg2" with "key2" is separate.

3. Final Answer:

   "msg1" and "msg3" are ordered; "msg2" may be in a different partition. -> Option B

4. Quick Check:

   Same key = same partition order [OK]

Hint: Same key means same partition and order [OK]

Common Mistakes:

• Assuming all messages go to one partition
• Thinking different keys keep order together
• Believing order is global across partitions

4. You have this Kafka producer code snippet:

producer.send(new ProducerRecord<>("topic", null, "msg1"));
producer.send(new ProducerRecord<>("topic", null, "msg2"));

Why might message order not be preserved?

medium

A. Because the topic has only one partition.

B. Because Kafka always reorders messages regardless of keys.

C. Because messages with null keys can go to different partitions, losing order.

D. Because the producer is synchronous.

Solution

1. Step 1: Analyze key usage in ProducerRecord

   Null keys mean Kafka uses round-robin or default partitioner, sending messages to different partitions.

2. Step 2: Understand ordering impact

   Messages in different partitions have no ordering guarantee, so order may be lost.

3. Final Answer:

   Because messages with null keys can go to different partitions, losing order. -> Option C

4. Quick Check:

   Null key = no partition order guarantee [OK]

Hint: Null keys cause messages to spread partitions, breaking order [OK]

Common Mistakes:

• Thinking Kafka always orders messages
• Assuming one partition means no order loss
• Confusing synchronous with ordering guarantee

5. You want to ensure strict message order for user events in Kafka. You have multiple users sending events simultaneously. Which approach best guarantees order per user?

hard

A. Send all messages without keys to a single partition topic.

B. Use multiple topics, one per user, to keep order.

C. Use random keys to balance load and rely on consumer ordering.

D. Use the user ID as the message key so all events for a user go to the same partition.

Solution

1. Step 1: Identify how Kafka partitions messages

   Kafka uses the key to assign messages to partitions; same key means same partition.

2. Step 2: Apply to user event ordering

   Using user ID as key ensures all events for that user go to one partition, preserving order.

3. Step 3: Evaluate other options

   Sending without keys or random keys breaks order; multiple topics is complex and inefficient.

4. Final Answer:

   Use the user ID as the message key so all events for a user go to the same partition. -> Option D

5. Quick Check:

   User ID key = per-user order guaranteed [OK]

Hint: Key by user ID to keep their events ordered [OK]

Common Mistakes:

• Sending messages without keys expecting order
• Using random keys and expecting order
• Creating many topics instead of using keys