Why topics organize messages in Kafka - Visual Breakdown

Process Flow - Why topics organize messages

Producer sends message

↓

Message goes to Topic

↓

Topic stores messages in order

↓

Consumers read messages from Topic

↓

Consumers process messages in order

Messages are grouped in topics to keep them organized and in order, so producers send messages to topics, and consumers read them from there.

Execution Sample

Kafka

producer.send('orders', 'order1')
producer.send('orders', 'order2')
consumer.read('orders')

This code sends two messages to the 'orders' topic and then a consumer reads messages from it in order.

Process Table

Step	Action	Topic State	Message Sent/Read	Output
1	Producer sends 'order1' to 'orders'	orders: ['order1']	Sent 'order1'	None
2	Producer sends 'order2' to 'orders'	orders: ['order1', 'order2']	Sent 'order2'	None
3	Consumer reads from 'orders'	orders: ['order1', 'order2']	Read 'order1'	'order1' processed
4	Consumer reads next message	orders: ['order1', 'order2']	Read 'order2'	'order2' processed
5	No more messages	orders: ['order1', 'order2']	None	Consumer waits

💡 Consumer stops reading when no new messages are available in the topic.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Topic 'orders'	[]	['order1']	['order1', 'order2']	['order1', 'order2']	['order1', 'order2']	['order1', 'order2']
Consumer position	0	0	0	1	2	2

Key Moments - 2 Insights

Why do messages go into a topic instead of directly to consumers?

Messages go into a topic to keep them organized and stored in order, so multiple consumers can read them independently and reliably, as shown in steps 1 and 3 of the execution_table.

Does the topic remove messages after consumers read them?

No, the topic keeps messages until they expire or are deleted by retention policy. Consumers track their own position, as seen in the variable_tracker where the topic state stays the same after reads.

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what is the state of the 'orders' topic?

A ['order1']

B ['order2']

C ['order1', 'order2']

D []

💡 Hint

Check the 'Topic State' column at step 2 in the execution_table.

At which step does the consumer read the first message?

A Step 1

B Step 3

C Step 4

D Step 5

💡 Hint

Look at the 'Message Sent/Read' column in the execution_table.

If the producer sends a third message, how would the topic state change after step 2?

A ['order1', 'order2', 'order3']

B ['order3']

C ['order1', 'order3']

D No change

💡 Hint

Refer to how the topic state changes when messages are sent in the execution_table steps 1 and 2.

Concept Snapshot

Kafka topics organize messages by grouping them in named streams.
Producers send messages to topics.
Topics store messages in order.
Consumers read messages from topics independently.
This keeps messages organized and allows multiple consumers.

Full Transcript

In Kafka, messages are organized using topics. Producers send messages to a topic, which stores them in order. Consumers then read messages from the topic in the same order. This organization helps keep messages grouped and allows multiple consumers to read independently without losing messages. The topic keeps messages until they expire or are deleted, and consumers track their reading position. This way, Kafka ensures reliable and ordered message delivery.