Message key and value in Kafka - Mini Project: Build & Apply

Message key and value

📖 Scenario: You are working with Apache Kafka to send messages to a topic. Each message has a key and a value. The key helps Kafka decide which partition the message goes to, and the value is the actual data you want to send.Imagine you are sending user login events where the key is the user ID and the value is the login timestamp.

🎯 Goal: Build a simple Kafka producer that sends messages with a key and a value to a topic called user-logins.

📋 What You'll Learn

Create a Kafka producer configuration

Send messages with a key and a value

Use the topic name user-logins

Print confirmation after sending each message

💡 Why This Matters

🌍 Real World

Kafka is widely used for real-time data streaming. Sending messages with keys helps organize data by partitions, which improves performance and scalability.

💼 Career

Understanding how to send messages with keys and values is essential for roles involving data engineering, backend development, and real-time analytics.

1

Set up Kafka producer properties

Create a dictionary called producer_props with 'bootstrap_servers': 'localhost:9092'.

Kafka

# Create the producer properties dictionary
# Your code here

Hint

Use a Python dictionary with the exact keys and values for Kafka producer configuration.

2

Create the Kafka producer instance

Create a variable called producer and assign it a new Kafka producer instance using producer_props.

Kafka

producer_props = {
    'bootstrap_servers': 'localhost:9092'
}
# Create the Kafka producer instance
# Your code here

Hint

Use KafkaProducer with the unpacked producer_props dictionary.

3

Send messages with key and value

Use producer.send to send a message to the topic 'user-logins' with the key b'user123' and the value b'2024-06-01T10:00:00Z'.

Kafka

producer_props = {
    'bootstrap_servers': 'localhost:9092'
}
producer = KafkaProducer(**producer_props)
# Send a message with key and value
# Your code here

Hint

Use producer.send with the exact topic, key, and value bytes.

4

Print confirmation after sending

Write a print statement that outputs exactly "Sent message with key user123 to topic user-logins".

Kafka

producer_props = {
    'bootstrap_servers': 'localhost:9092'
}
producer = KafkaProducer(**producer_props)
producer.send('user-logins', key=b'user123', value=b'2024-06-01T10:00:00Z')
# Print confirmation message
# Your code here

Hint

Use print with the exact message text.

Practice

1. What is the primary role of the message key in Kafka?

easy

A. To decide which partition the message goes to

B. To store the actual data of the message

C. To encrypt the message content

D. To delete old messages automatically

Solution

1. Step 1: Understand message key purpose

   The message key in Kafka is used to determine the partition where the message will be stored.

2. Step 2: Differentiate key and value roles

   The message value holds the actual data, while the key helps with partitioning and ordering.

3. Final Answer:

   To decide which partition the message goes to -> Option A

4. Quick Check:

   Message key = Partition decision [OK]

Hint: Key decides partition, value holds data [OK]

Common Mistakes:

• Confusing key with message value
• Thinking key encrypts data
• Assuming key deletes messages

2. Which of the following is the correct way to send a Kafka message with a key and value in Java?

easy

A. producer.send(new ProducerRecord<>("topic", value));

B. producer.send(new ProducerRecord<>(key, value));

C. producer.send("topic", value);

D. producer.send(new ProducerRecord<>("topic", key, value));

Solution

1. Step 1: Review ProducerRecord constructor

   The correct constructor for sending a message with key and value includes topic, key, and value in that order.

2. Step 2: Check each option

   producer.send(new ProducerRecord<>("topic", key, value)); uses new ProducerRecord<>("topic", key, value), which is correct syntax. Others miss topic or key.

3. Final Answer:

   producer.send(new ProducerRecord<>("topic", key, value)); -> Option D

4. Quick Check:

   ProducerRecord(topic, key, value) = correct [OK]

Hint: ProducerRecord needs topic, key, then value [OK]

Common Mistakes:

• Omitting the topic parameter
• Swapping key and value order
• Using wrong constructor overload

3. Given the following Kafka producer code snippet, what will be the key of the message sent?

ProducerRecord record = new ProducerRecord<>("orders", "12345", "order details");
producer.send(record);

medium

A. "orders"

B. null

C. "12345"

D. "order details"

Solution

1. Step 1: Identify parameters in ProducerRecord

   The constructor is ProducerRecord(topic, key, value). Here, "orders" is topic, "12345" is key, and "order details" is value.

2. Step 2: Determine the key value

   The key is the second argument, which is "12345".

3. Final Answer:

   "12345" -> Option C

4. Quick Check:

   Second argument = key = "12345" [OK]

Hint: Second argument in ProducerRecord is key [OK]

Common Mistakes:

• Confusing topic with key
• Choosing message value as key
• Assuming key is null if not obvious

4. Identify the error in this Kafka producer code snippet:

ProducerRecord record = new ProducerRecord<>("topic", null, "data");
producer.send(record);

medium

A. Message key cannot be null

B. This code is valid; no error

C. ProducerRecord constructor parameters are in wrong order

D. Message value cannot be null

Solution

1. Step 1: Check if null key is allowed

   Kafka allows message keys to be null; this means messages can be sent without a key.

2. Step 2: Verify constructor usage

   The constructor ProducerRecord<>("topic", null, "data") is valid and sends a message with no key.

3. Final Answer:

   This code is valid; no error -> Option B

4. Quick Check:

   Null key allowed in Kafka messages [OK]

Hint: Null keys are allowed in Kafka messages [OK]

Common Mistakes:

• Assuming key must never be null
• Confusing parameter order
• Thinking value cannot be null without checking

5. You want to ensure all messages with the same user ID go to the same partition for ordering. Which Kafka message feature should you use and how?

hard

A. Use the user ID as the message key to ensure partitioning by key

B. Include the user ID only in the message value and leave key null

C. Set the user ID as the topic name

D. Send messages without keys and rely on default partitioning

Solution

1. Step 1: Understand partitioning by key

   Kafka uses the message key to decide the partition. Messages with the same key go to the same partition.

2. Step 2: Apply to user ID scenario

   Using the user ID as the message key ensures all messages for that user go to the same partition, preserving order.

3. Final Answer:

   Use the user ID as the message key to ensure partitioning by key -> Option A

4. Quick Check:

   Key = user ID for partitioning [OK]

Hint: Use key for grouping messages by user ID [OK]

Common Mistakes:

• Putting user ID only in value loses partition control
• Changing topic per user is inefficient
• Relying on default partitioning breaks ordering