Python consumer in Kafka - Step-by-Step Execution

Process Flow - Python consumer

Start Consumer

↓

Connect to Kafka Broker

↓

Subscribe to Topic

↓

Poll for Messages

↓

Message Received?

No→Wait and Poll Again

Yes↓

Process Message

↓

Commit Offset

↓

Repeat Polling Loop

↓

Stop Consumer on Signal

The Python consumer connects to Kafka, subscribes to a topic, polls messages in a loop, processes each message, commits offsets, and repeats until stopped.

Execution Sample

Kafka

from kafka import KafkaConsumer
consumer = KafkaConsumer('my_topic', bootstrap_servers='localhost:9092')
for message in consumer:
    print(f"Received: {message.value.decode()}")

This code connects to Kafka, subscribes to 'my_topic', and prints each message received.

Process Table

Step	Action	Evaluation	Result
1	Create KafkaConsumer instance	Connect to localhost:9092	Consumer ready, subscribed to 'my_topic'
2	Start polling loop	Wait for message	No message yet, keep polling
3	Message arrives	message.value = b'Hello'	Message received
4	Decode message	message.value.decode() = 'Hello'	Decoded message 'Hello'
5	Print message	print output	Output: Received: Hello
6	Commit offset	Offset committed	Ready for next message
7	Poll next message	Wait for message	No message yet, keep polling
8	Stop consumer	Signal received	Consumer stops gracefully

💡 Consumer stops when external signal triggers shutdown

Status Tracker

Variable	Start	After Step 3	After Step 4	After Step 6	Final
consumer	None	KafkaConsumer instance connected	Same instance	Same instance	Closed
message	None	b'Hello'	b'Hello'	b'Hello'	None
decoded_message	None	None	'Hello'	'Hello'	None

Key Moments - 3 Insights

Why does the consumer keep polling even when no messages arrive?

The consumer polls in a loop waiting for messages; if none arrive, it waits and polls again as shown in step 2 and 7 of the execution_table.

What does committing the offset mean and why is it important?

Committing the offset (step 6) tells Kafka the message was processed, so the consumer won't re-read it after restart, ensuring no duplicates.

Why do we decode message.value before printing?

Kafka messages are bytes; decoding converts bytes to readable text, as shown in step 4 where b'Hello' becomes 'Hello'.

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of 'message' after step 3?

A b'Hello'

B 'Hello'

C None

D KafkaConsumer instance

💡 Hint

Check the 'message' variable in variable_tracker after step 3

At which step does the consumer commit the offset?

A Step 4

B Step 6

C Step 2

D Step 8

💡 Hint

Look for 'Commit offset' action in execution_table

If no messages arrive, what does the consumer do according to the execution_table?

A Stops immediately

B Prints 'Received: None'

C Keeps polling and waiting

D Throws an error

💡 Hint

See steps 2 and 7 where it waits and polls again

Concept Snapshot

Python Kafka Consumer:
- Create KafkaConsumer with topic and broker
- Poll messages in a loop
- Decode message bytes to string
- Process and commit offset
- Repeat until stopped
- Commit ensures no duplicate processing

Full Transcript

A Python Kafka consumer connects to the Kafka broker and subscribes to a topic. It enters a loop where it polls for messages. If no message is available, it waits and polls again. When a message arrives, it is received as bytes, decoded to a string, and processed (e.g., printed). After processing, the consumer commits the offset to mark the message as read. This loop continues until an external signal stops the consumer gracefully.