Message key and value in Kafka - Commands & Configuration

Introduction

Kafka sends messages as pairs of keys and values. The key helps decide where the message goes, and the value is the actual data you want to send.

When you want to keep related messages together in the same order.

When you need to update or delete messages by key later.

When you want to balance load evenly across servers.

When you want to group messages by customer ID or session ID.

When you want to ensure messages with the same key go to the same place.

Commands

Start a Kafka producer that reads messages from your keyboard. It uses ':' to split the key and value in each message.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --property parse.key=true --property key.separator=:

Expected Output

>

→

--property parse.key=true - Enable parsing of message keys from input

→

--property key.separator=: - Set ':' as the separator between key and value

Send a message with key 'user1' and value 'Hello Kafka'. This keeps messages with key 'user1' together.

Terminal

user1:Hello Kafka

Expected Output

No output (command runs silently)

Start a consumer to read messages from the beginning, showing both keys and values.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --property print.key=true --from-beginning

Expected Output

user1	Hello Kafka

→

--property print.key=true - Show message keys in output

→

--from-beginning - Read all messages from the start

Key Concept

The message key groups related messages and controls where they go, while the value holds the actual data.

Common Mistakes

Not setting --property parse.key=true when producing messages with keys

Kafka treats the whole input as the value, ignoring the key part.

Always use --property parse.key=true and set a key separator when sending keyed messages.

Not using --property print.key=true when consuming messages with keys

You only see the value, missing the key information.

Add --property print.key=true to see both keys and values.

Summary

Use message keys to group related messages and control partitioning.

Set --property parse.key=true and key.separator when producing keyed messages.

Use --property print.key=true when consuming to see keys with values.