Python producer (confluent-kafka) - Step-by-Step Execution

Process Flow - Python producer (confluent-kafka)

Create Producer Config

↓

Initialize Producer

↓

Prepare Message

↓

Call produce()

↓

Message Queued

↓

Call flush() to send

↓

Message Sent to Kafka

↓

Exit

This flow shows how a Python Kafka producer is configured, sends a message, and ensures delivery.

Execution Sample

Kafka

from confluent_kafka import Producer

p = Producer({'bootstrap.servers': 'localhost:9092'})
p.produce('my_topic', key='key1', value='hello')
p.flush()

This code creates a Kafka producer, sends one message with key and value, then waits for delivery.

Process Table

Step	Action	Internal State	Result/Output
1	Create Producer with config	Producer object created, connected to localhost:9092	Ready to send messages
2	Call produce() with topic='my_topic', key='key1', value='hello'	Message queued in producer buffer	No immediate output
3	Call flush()	Producer sends all queued messages to Kafka broker	Message delivered to 'my_topic'
4	Exit	No pending messages	Program ends

💡 All messages flushed and sent, producer cleanly closed

Status Tracker

Variable	Start	After produce()	After flush()	Final
p	None	Producer object with 1 message queued	Producer object with empty queue	Producer object closed
message_queue	Empty	1 message queued	Empty	Empty

Key Moments - 3 Insights

Why do we need to call flush() after produce()?

produce() only queues the message; flush() actually sends all queued messages to Kafka. See execution_table step 3.

What happens if we call produce() without flush()?

Messages may stay in the buffer and not be sent, so flush() ensures delivery before program ends (execution_table step 4).

Why do we pass 'bootstrap.servers' in the config?

It tells the producer where the Kafka broker is, so it can connect and send messages (execution_table step 1).

Visual Quiz - 3 Questions

Look at the execution_table, what is the state of the producer after calling produce()?

A Producer closed

B Message sent to Kafka broker

C Message queued in producer buffer

D No messages queued

💡 Hint

Check execution_table row 2 under 'Internal State'

At which step does the message actually get sent to Kafka?

A Step 3

B Step 1

C Step 2

D Step 4

💡 Hint

Look at execution_table row 3 'Result/Output'

If flush() is not called, what is the likely outcome?

A Message is sent immediately

B Message stays queued and may not be sent

C Producer throws an error

D Message is lost before queuing

💡 Hint

Refer to key_moments about flush() importance and execution_table step 4

Concept Snapshot

Python Kafka Producer (confluent-kafka):
- Create Producer with config {'bootstrap.servers': 'host:port'}
- Use produce(topic, key, value) to queue messages
- Call flush() to send all queued messages
- flush() ensures delivery before program ends
- Without flush(), messages may not be sent

Full Transcript

This visual trace shows how to use the Python confluent-kafka Producer. First, you create a Producer object with the Kafka broker address. Then, you call produce() to queue a message with a topic, key, and value. The message is not sent immediately but stored in a buffer. To actually send the message to Kafka, you call flush(), which sends all queued messages and waits for confirmation. Finally, the program ends after all messages are delivered. Remember, without flush(), messages may remain unsent. This step-by-step helps beginners see how the producer works internally and why flush() is important.