Python producer (confluent-kafka) - Mini Project: Build & Apply

Python Producer with Confluent-Kafka

📖 Scenario: You are building a simple Python program that sends messages to a Kafka topic. Kafka is like a post office where your program can send letters (messages) to be delivered to others.We will use the confluent-kafka library to create a producer that sends messages to a Kafka topic named test_topic.

🎯 Goal: Create a Python Kafka producer that sends three messages to the test_topic topic and prints confirmation for each message sent.

📋 What You'll Learn

Create a Kafka producer with the confluent-kafka library

Set the Kafka broker address to localhost:9092

Send three specific messages to the topic test_topic

Print confirmation after each message is sent

💡 Why This Matters

🌍 Real World

Kafka producers are used in real applications to send data streams like logs, user activity, or sensor data to Kafka topics for processing.

💼 Career

Understanding how to produce messages to Kafka is important for roles in data engineering, backend development, and real-time data processing.

1

Set up Kafka producer configuration

Create a dictionary called conf with the key 'bootstrap.servers' set to 'localhost:9092'.

Kafka

# Your code here

Need a hint?

This dictionary tells the producer where the Kafka server is located.

2

Create the Kafka producer

Import Producer from confluent_kafka and create a producer object called producer using the conf dictionary.

Kafka

conf = {'bootstrap.servers': 'localhost:9092'}
# Your code here

Need a hint?

Use Producer(conf) to create the producer object.

3

Send messages to the Kafka topic

Use producer.produce() to send these messages to the topic 'test_topic': 'Hello Kafka', 'This is message 2', and 'Final message'. Use a for loop with variables msg and messages to iterate over the list of messages.

Kafka

from confluent_kafka import Producer

conf = {'bootstrap.servers': 'localhost:9092'}
producer = Producer(conf)

# Your code here

Need a hint?

Use producer.produce('test_topic', msg.encode('utf-8')) inside the loop to send each message.

4

Flush and print confirmation

Call producer.flush() to ensure all messages are sent. Then, use a for loop with variables msg and messages to print "Sent: {msg}" for each message.

Kafka

from confluent_kafka import Producer

conf = {'bootstrap.servers': 'localhost:9092'}
producer = Producer(conf)

messages = ['Hello Kafka', 'This is message 2', 'Final message']
for msg in messages:
    producer.produce('test_topic', msg.encode('utf-8'))

# Your code here

Need a hint?

Use producer.flush() to wait for all messages to be sent before printing.