Why producers publish data in Kafka - See It in Action

Why Producers Publish Data in Kafka

📖 Scenario: Imagine you run a small online store. You want to keep track of every order your customers make. To do this, you use Kafka, a system that helps send and store messages quickly and safely.

🎯 Goal: Learn why producers publish data to Kafka and how to create a simple producer that sends order messages.

📋 What You'll Learn

Create a Kafka topic named orders

Write a producer that sends order data to the orders topic

Understand the role of the producer in sending data

Print confirmation after sending each order

💡 Why This Matters

🌍 Real World

Online stores, banks, and many apps use Kafka producers to send data like orders, transactions, or logs quickly and reliably.

💼 Career

Understanding Kafka producers is important for roles in data engineering, backend development, and real-time data processing.

1

Create a list of orders

Create a list called orders with these exact dictionaries: {'order_id': 101, 'item': 'Book'}, {'order_id': 102, 'item': 'Pen'}, {'order_id': 103, 'item': 'Notebook'}

Kafka

# Create the list of orders
# Your code here

Hint

Use a list with dictionaries inside. Each dictionary has keys 'order_id' and 'item'.

2

Set the Kafka topic name

Create a variable called topic_name and set it to the string 'orders'

Kafka

orders = [{'order_id': 101, 'item': 'Book'}, {'order_id': 102, 'item': 'Pen'}, {'order_id': 103, 'item': 'Notebook'}]
# Set the Kafka topic name
# Your code here

Hint

Just assign the string 'orders' to the variable topic_name.

3

Send orders to Kafka topic

Use a for loop with variable order to go through orders and inside the loop, call producer.send(topic_name, value=order)

Kafka

orders = [{'order_id': 101, 'item': 'Book'}, {'order_id': 102, 'item': 'Pen'}, {'order_id': 103, 'item': 'Notebook'}]
topic_name = 'orders'

from kafka import KafkaProducer
import json

producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))

# Send each order to the Kafka topic
# Your code here

Hint

Use a for loop to send each order dictionary to Kafka using producer.send().

4

Print confirmation after sending

Inside the for loop, after sending, write print(f"Sent order {order['order_id']} to topic {topic_name}")

Kafka

orders = [{'order_id': 101, 'item': 'Book'}, {'order_id': 102, 'item': 'Pen'}, {'order_id': 103, 'item': 'Notebook'}]
topic_name = 'orders'

from kafka import KafkaProducer
import json

producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))

for order in orders:
    producer.send(topic_name, value=order)
    # Print confirmation
    # Your code here

Hint

Use a print statement inside the loop to show which order was sent and to which topic.

Practice

1. Why do Kafka producers publish data to topics?

easy

A. To delete old messages from Kafka topics

B. To store data permanently on the producer machine

C. To send messages so other applications can receive and process them

D. To create new Kafka topics automatically

Solution

1. Step 1: Understand the role of producers in Kafka

   Producers send data to Kafka topics to share information with other applications.

2. Step 2: Identify the purpose of publishing data

   Publishing allows other consumers to receive and process the messages in real time.

3. Final Answer:

   To send messages so other applications can receive and process them -> Option C

4. Quick Check:

   Producers publish data to share it [OK]

Hint: Producers send data for others to use it [OK]

Common Mistakes:

• Thinking producers store data locally
• Confusing producers with consumers
• Believing producers delete messages

2. Which of the following is the correct way for a Kafka producer to send a message to a topic named orders?

easy

A. producer.send('orders', message)

B. producer.publish('orders', message)

C. producer.write('orders', message)

D. producer.push('orders', message)

Solution

1. Step 1: Recall Kafka producer API method

   The Kafka producer uses the send method to publish messages to a topic.

2. Step 2: Match method to correct syntax

   Only producer.send('orders', message) matches the correct Kafka producer syntax.

3. Final Answer:

   producer.send('orders', message) -> Option A

4. Quick Check:

   Use send() to publish messages [OK]

Hint: Use send() method to publish messages [OK]

Common Mistakes:

• Using publish(), write(), or push() which are not Kafka methods
• Confusing consumer methods with producer methods
• Incorrect method names cause runtime errors

3. What will be the output if a Kafka producer sends a message to a topic but the topic does not exist and auto-creation is disabled?

medium

A. The message is sent successfully and stored

B. The producer automatically creates the topic and sends the message

C. The message is silently dropped without error

D. An error is thrown indicating the topic does not exist

Solution

1. Step 1: Understand Kafka topic auto-creation setting

   If auto-creation is disabled, Kafka will not create topics automatically when a producer sends data.

2. Step 2: Identify behavior when topic is missing

   Sending to a non-existent topic causes Kafka to throw an error to the producer.

3. Final Answer:

   An error is thrown indicating the topic does not exist -> Option D

4. Quick Check:

   Missing topic with no auto-create = error [OK]

Hint: No topic + no auto-create = error thrown [OK]

Common Mistakes:

• Assuming topic is created automatically
• Thinking message is silently dropped
• Believing message is sent without topic

4. A Kafka producer code snippet is failing to send messages. Which fix resolves this error?

producer.send(topic, message)

Assuming message is a string and topic is a string.

medium

A. Ensure message is encoded to bytes before sending

B. Remove .encode() because Kafka expects strings, not bytes

C. Change topic to a bytes object

D. Call producer.flush() before sending

Solution

1. Step 1: Check Kafka producer message format

   Kafka producers expect message data as bytes, so encoding strings is necessary.

2. Step 2: Confirm correct usage of encode()

   Using message.encode() converts the string to bytes, which is correct.

3. Final Answer:

   Ensure message is encoded to bytes before sending -> Option A

4. Quick Check:

   Messages must be bytes, so encode strings [OK]

Hint: Encode strings to bytes before sending [OK]

Common Mistakes:

• Not encoding the message to bytes before sending
• Changing topic to bytes (topic must be string)
• Calling flush() before send() instead of after

5. You want to publish user activity events to Kafka so multiple services can react in real time. Which approach best fits this use case?

hard

A. Write user events to a local file and have services read the file

B. Use a Kafka producer to send serialized event data to a shared topic named user-activity

C. Send user events directly to each service via HTTP requests

D. Store user events in a database and poll it periodically

Solution

1. Step 1: Identify real-time data sharing method

   Kafka producers send data to topics that multiple consumers can read in real time.

2. Step 2: Match use case to Kafka producer pattern

   Sending serialized events to a shared topic allows all services to react immediately.

3. Final Answer:

   Use a Kafka producer to send serialized event data to a shared topic named user-activity -> Option B

4. Quick Check:

   Producers publish events to topics for real-time sharing [OK]

Hint: Use Kafka topic for real-time multi-service events [OK]

Common Mistakes:

• Using files or DB polling which are not real-time
• Sending HTTP requests to each service manually
• Not using Kafka topics for event distribution