Consumer API basics in Kafka - Step-by-Step Execution

Process Flow - Consumer API basics

Start Consumer

↓

Subscribe to Topic

↓

Poll for Messages

↓

Process Messages

↓

Commit Offsets

↓

Repeat Polling

↩Back to Poll for Messages

The consumer starts, subscribes to a topic, polls messages, processes them, commits offsets, and repeats until stopped.

Execution Sample

Kafka

consumer = KafkaConsumer('my_topic')
consumer.subscribe(['my_topic'])
for message in consumer:
    print(message.value)
    consumer.commit()

This code creates a consumer, subscribes to 'my_topic', reads messages one by one, prints their values, and commits offsets.

Process Table

Step	Action	Message Polled	Offset	Output	Commit Offset
1	Start Consumer and Subscribe	None	None	None	None
2	Poll for message	Message1	0	Print Message1	Commit offset 1
3	Poll for message	Message2	1	Print Message2	Commit offset 2
4	Poll for message	Message3	2	Print Message3	Commit offset 3
5	Poll for message	No new message	2	No output	No commit
6	Stop Consumer	None	2	None	None

💡 No new messages available, consumer stops polling.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
message	None	Message1	Message2	Message3	None	None
offset	None	0	1	2	2	2
committed_offset	None	1	2	3	3	3

Key Moments - 3 Insights

Why do we commit offsets after processing each message?

Committing offsets tells Kafka which messages have been processed, so if the consumer restarts, it continues from the last committed offset (see steps 2-4 in execution_table).

What happens if no new messages are available during polling?

The poll returns no messages, so the consumer waits or stops polling (see step 5 in execution_table where no new message is polled).

Why do we subscribe to a topic before polling?

Subscription tells the consumer which topic(s) to listen to, so polling knows where to get messages from (see step 1 in execution_table).

Visual Quiz - 3 Questions

Look at the execution_table, what is the committed_offset after step 3?

A 2

B 1

C 0

D None

💡 Hint

Check the committed_offset value in variable_tracker after step 3.

At which step does the consumer find no new messages?

A Step 2

B Step 5

C Step 3

D Step 6

💡 Hint

Look at the 'Message Polled' column in execution_table.

If we skip committing offsets, what would happen on restart?

A Consumer starts from next offset after last commit

B Consumer starts from last committed offset

C Consumer starts from beginning of topic

D Consumer cannot start

💡 Hint

Refer to key_moments about why committing offsets is important.

Concept Snapshot

Kafka Consumer API basics:
- Create consumer and subscribe to topic
- Poll messages in a loop
- Process each message
- Commit offsets after processing
- Repeat polling until stopped
- Committing offsets ensures no message loss or duplication

Full Transcript

This visual execution shows how a Kafka consumer works. First, the consumer starts and subscribes to a topic. Then it polls messages one by one. Each message is processed by printing its value. After processing, the consumer commits the offset to mark the message as done. This repeats until no new messages are available, then the consumer stops. Committing offsets is important to avoid reprocessing messages after restart. If no messages are found during polling, the consumer waits or stops polling. Subscribing to the topic tells the consumer where to get messages from.