Event streaming concept in Kafka - Step-by-Step Execution

Process Flow - Event streaming concept

Event Produced

↓

Event Sent to Kafka Topic

↓

Kafka Stores Event in Partition

↓

Consumer Reads Event from Partition

↓

Consumer Processes Event

↓

Acknowledgment Sent

↓

Next Event Processed

Events are created and sent to Kafka topics, stored in partitions, then consumers read and process them in order.

Execution Sample

Kafka

producer.send('orders', {'order_id': 1, 'item': 'book'})
consumer.poll()
process(event)
consumer.commit()

A producer sends an order event to the 'orders' topic; the consumer polls, processes, and commits the event.

Process Table

Step	Action	Event Data	Kafka State	Consumer State	Output
1	Producer sends event	{order_id:1, item:'book'}	Event stored in 'orders' partition 0	No events read yet	Event queued
2	Consumer polls topic	N/A	Event still stored	Reads event from partition 0 offset 0	Event received
3	Consumer processes event	{order_id:1, item:'book'}	Event stored	Processing event	Order processed
4	Consumer commits offset	N/A	Event stored	Offset 1 committed	Ready for next event
5	No new events	N/A	No new events	No events to read	Waiting for events

💡 No new events to process, consumer waits for more.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Kafka Topic 'orders'	empty	1 event stored	1 event stored	1 event stored	1 event stored	1 event stored
Consumer Offset	0	0	0	0	1	1
Event in Consumer	none	event(order_id=1)	event(order_id=1)	event(order_id=1)	none	none

Key Moments - 3 Insights

Why does the consumer commit the offset after processing?

Committing the offset (see step 4) tells Kafka the event was processed successfully, so the consumer won't re-read it.

What happens if the consumer polls but no new events exist?

As shown in step 5, the consumer waits without processing until new events arrive.

Is the event removed from Kafka after the consumer reads it?

No, Kafka keeps events for a set time or size limit; consumers track offsets to know what to read next.

Visual Quiz - 3 Questions

Look at the execution table, what is the consumer offset after step 3?

A 1

B 0

C Not set

D Undefined

💡 Hint

Check the 'Consumer Offset' row in variable_tracker after step 3.

At which step does the consumer acknowledge it processed the event?

A Step 4

B Step 2

C Step 3

D Step 5

💡 Hint

Look for 'Offset committed' in the execution_table step 4.

If a new event arrives after step 5, what changes in the consumer state?

A Consumer offset resets to 0

B Kafka deletes old events

C Consumer reads new event from next offset

D Consumer stops polling

💡 Hint

Refer to how consumer reads events from partitions in execution_table steps 2 and 5.

Concept Snapshot

Event streaming means sending data as events to a Kafka topic.
Kafka stores events in partitions.
Consumers read events in order using offsets.
Consumers commit offsets after processing.
Events remain in Kafka until retention expires.

Full Transcript

Event streaming with Kafka works by producers sending events to topics. Kafka stores these events in partitions. Consumers poll these partitions to read events in order. After processing, consumers commit their offsets to mark progress. Kafka keeps events for a retention period, allowing multiple consumers to read independently. This flow ensures reliable, ordered event processing.