Python producer (confluent-kafka) - Commands & Configuration

Introduction

Sending messages to Kafka topics can be tricky without the right tools. The Python producer from confluent-kafka helps you send data easily and reliably to Kafka servers using simple Python code.

When you want to send user activity logs from a Python app to Kafka for processing.

When you need to stream sensor data from a Python script to a Kafka topic for real-time analysis.

When you want to build a Python service that pushes notifications into Kafka for other services to consume.

When you want to test Kafka message flow by producing sample messages from Python.

When you want to integrate Python applications with Kafka without writing complex Java code.

Commands

This command installs the confluent-kafka Python library, which provides the Kafka producer functionality.

Terminal

pip install confluent-kafka

Expected Output

Collecting confluent-kafka

  Downloading confluent_kafka-2.1.1-cp39-cp39-manylinux2014_x86_64.whl (1.3 MB)

Installing collected packages: confluent-kafka

Successfully installed confluent-kafka-2.1.1

Runs the Python script that sends a message to the Kafka topic using the confluent-kafka producer.

Terminal

python produce_message.py

Expected Output

Message delivered to topic 'example-topic' partition [0] at offset 15

Key Concept

If you remember nothing else from this pattern, remember: the confluent-kafka Python producer sends messages asynchronously and requires a delivery callback to confirm success.

Code Example

Kafka

from confluent_kafka import Producer

conf = {'bootstrap.servers': 'localhost:9092'}
producer = Producer(conf)

def delivery_report(err, msg):
    if err is not None:
        print(f'Message delivery failed: {err}')
    else:
        print(f"Message delivered to topic '{msg.topic()}' partition [{msg.partition()}] at offset {msg.offset()}")

producer.produce('example-topic', key='key1', value='Hello Kafka!', callback=delivery_report)
producer.flush()

Common Mistakes

Not installing the confluent-kafka library before running the script.

The Python script will fail with a ModuleNotFoundError because the library is missing.

Always run 'pip install confluent-kafka' before executing the producer script.

Not flushing the producer after sending messages.

Messages may remain in the buffer and not be sent to Kafka, causing data loss.

Call producer.flush() after producing messages to ensure delivery.

Using incorrect Kafka broker address or topic name.

The producer cannot connect or send messages, resulting in errors or silent failures.

Verify the broker address and topic name are correct and accessible.

Summary

Install the confluent-kafka Python library to enable Kafka message production.

Write a Python script that creates a Producer, sends a message with a delivery callback, and flushes the producer.

Run the script to send messages and confirm delivery with printed output.