Message key and value in Kafka - Step-by-Step Execution

Process Flow - Message key and value

Create Message

↓

Assign Key and Value

↓

Send to Kafka Topic

↓

Kafka stores message

↓

Consumers read Key and Value

This flow shows how a Kafka message is created with a key and value, sent to a topic, stored, and then read by consumers.

Execution Sample

Kafka

producer.send(new ProducerRecord<>("my-topic", "user1", "Hello Kafka"));

Send a message with key 'user1' and value 'Hello Kafka' to the topic 'my-topic'.

Process Table

Step	Action	Key	Value	Topic	Result
1	Create ProducerRecord	user1	Hello Kafka	my-topic	Record ready to send
2	Send message	user1	Hello Kafka	my-topic	Message sent to Kafka broker
3	Kafka stores message	user1	Hello Kafka	my-topic	Message stored with key and value
4	Consumer reads message	user1	Hello Kafka	my-topic	Consumer receives key and value

💡 Message key and value successfully sent, stored, and consumed.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4
key	null	user1	user1	user1	user1
value	null	Hello Kafka	Hello Kafka	Hello Kafka	Hello Kafka
topic	null	my-topic	my-topic	my-topic	my-topic

Key Moments - 2 Insights

Why do we need a key in a Kafka message?

The key helps Kafka decide which partition the message goes to and allows consumers to process messages with the same key in order, as shown in step 3 of the execution_table.

Can a Kafka message have a null key?

Yes, a message can have a null key, but then Kafka distributes messages round-robin across partitions. This is different from the example where the key is 'user1' (step 1).

Visual Quiz - 3 Questions

Look at the execution_table, what is the key value at Step 2?

A null

B "Hello Kafka"

C "user1"

D "my-topic"

💡 Hint

Check the 'Key' column in row for Step 2 in execution_table.

At which step does Kafka store the message?

A Step 1

B Step 3

C Step 2

D Step 4

💡 Hint

Look at the 'Action' column in execution_table for the storing action.

If the key was null, how would the partitioning change?

A Messages are distributed round-robin across partitions

B Messages are sent to a single partition only

C Messages go to partitions based on key hash

D Messages are rejected by Kafka

💡 Hint

Refer to key_moments explanation about null keys and partitioning.

Concept Snapshot

Kafka messages have a key and a value.
The key determines the partition and ordering.
The value is the actual data sent.
Messages are sent to a topic.
Consumers read both key and value.
Null keys cause round-robin partitioning.

Full Transcript

This visual execution shows how a Kafka message is created with a key and value, sent to a topic, stored by Kafka, and then read by consumers. The key helps Kafka decide the partition and maintain order for messages with the same key. The value is the main content of the message. The execution table traces each step from creating the message record, sending it, storing it, and consuming it. Variables key, value, and topic are tracked through each step. Key moments clarify why keys are important and what happens if the key is null. The quiz tests understanding of key values at each step, when Kafka stores the message, and the effect of null keys on partitioning.

Practice

1. What is the primary role of the message key in Kafka?

easy

A. To decide which partition the message goes to

B. To store the actual data of the message

C. To encrypt the message content

D. To delete old messages automatically

Solution

1. Step 1: Understand message key purpose

   The message key in Kafka is used to determine the partition where the message will be stored.

2. Step 2: Differentiate key and value roles

   The message value holds the actual data, while the key helps with partitioning and ordering.

3. Final Answer:

   To decide which partition the message goes to -> Option A

4. Quick Check:

   Message key = Partition decision [OK]

Hint: Key decides partition, value holds data [OK]

Common Mistakes:

• Confusing key with message value
• Thinking key encrypts data
• Assuming key deletes messages

2. Which of the following is the correct way to send a Kafka message with a key and value in Java?

easy

A. producer.send(new ProducerRecord<>("topic", value));

B. producer.send(new ProducerRecord<>(key, value));

C. producer.send("topic", value);

D. producer.send(new ProducerRecord<>("topic", key, value));

Solution

1. Step 1: Review ProducerRecord constructor

   The correct constructor for sending a message with key and value includes topic, key, and value in that order.

2. Step 2: Check each option

   producer.send(new ProducerRecord<>("topic", key, value)); uses new ProducerRecord<>("topic", key, value), which is correct syntax. Others miss topic or key.

3. Final Answer:

   producer.send(new ProducerRecord<>("topic", key, value)); -> Option D

4. Quick Check:

   ProducerRecord(topic, key, value) = correct [OK]

Hint: ProducerRecord needs topic, key, then value [OK]

Common Mistakes:

• Omitting the topic parameter
• Swapping key and value order
• Using wrong constructor overload

3. Given the following Kafka producer code snippet, what will be the key of the message sent?

ProducerRecord record = new ProducerRecord<>("orders", "12345", "order details");
producer.send(record);

medium

A. "orders"

B. null

C. "12345"

D. "order details"

Solution

1. Step 1: Identify parameters in ProducerRecord

   The constructor is ProducerRecord(topic, key, value). Here, "orders" is topic, "12345" is key, and "order details" is value.

2. Step 2: Determine the key value

   The key is the second argument, which is "12345".

3. Final Answer:

   "12345" -> Option C

4. Quick Check:

   Second argument = key = "12345" [OK]

Hint: Second argument in ProducerRecord is key [OK]

Common Mistakes:

• Confusing topic with key
• Choosing message value as key
• Assuming key is null if not obvious

4. Identify the error in this Kafka producer code snippet:

ProducerRecord record = new ProducerRecord<>("topic", null, "data");
producer.send(record);

medium

A. Message key cannot be null

B. This code is valid; no error

C. ProducerRecord constructor parameters are in wrong order

D. Message value cannot be null

Solution

1. Step 1: Check if null key is allowed

   Kafka allows message keys to be null; this means messages can be sent without a key.

2. Step 2: Verify constructor usage

   The constructor ProducerRecord<>("topic", null, "data") is valid and sends a message with no key.

3. Final Answer:

   This code is valid; no error -> Option B

4. Quick Check:

   Null key allowed in Kafka messages [OK]

Hint: Null keys are allowed in Kafka messages [OK]

Common Mistakes:

• Assuming key must never be null
• Confusing parameter order
• Thinking value cannot be null without checking

5. You want to ensure all messages with the same user ID go to the same partition for ordering. Which Kafka message feature should you use and how?

hard

A. Use the user ID as the message key to ensure partitioning by key

B. Include the user ID only in the message value and leave key null

C. Set the user ID as the topic name

D. Send messages without keys and rely on default partitioning

Solution

1. Step 1: Understand partitioning by key

   Kafka uses the message key to decide the partition. Messages with the same key go to the same partition.

2. Step 2: Apply to user ID scenario

   Using the user ID as the message key ensures all messages for that user go to the same partition, preserving order.

3. Final Answer:

   Use the user ID as the message key to ensure partitioning by key -> Option A

4. Quick Check:

   Key = user ID for partitioning [OK]

Hint: Use key for grouping messages by user ID [OK]

Common Mistakes:

• Putting user ID only in value loses partition control
• Changing topic per user is inefficient
• Relying on default partitioning breaks ordering