Message broker architecture in Kafka - Step-by-Step Execution

Process Flow - Message broker architecture

Producer sends message

↓

Message broker (Kafka)

↓

Message stored in topic partition

↓

Consumer reads message

↓

Consumer processes message

This flow shows how a producer sends messages to Kafka, which stores them in topic partitions, and consumers read and process these messages.

Execution Sample

Kafka

producer.send('topic1', 'Hello Kafka')
# Kafka stores message
consumer.poll()
# Consumer reads message

A producer sends a message to a Kafka topic, Kafka stores it, and a consumer reads the message.

Process Table

Step	Action	Component	Message State	Output
1	Producer sends message 'Hello Kafka' to 'topic1'	Producer	Message created	Message sent to broker
2	Broker receives message	Kafka Broker	Message stored in topic1 partition	Acknowledgement sent to producer
3	Consumer polls 'topic1'	Consumer	Reads message from topic1 partition	Message 'Hello Kafka' received
4	Consumer processes message	Consumer	Message processed	Processing complete
5	No more messages	Consumer	No new messages	Polling waits or ends

💡 No more messages in topic1, consumer polling ends or waits

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
message	None	'Hello Kafka' created	'Hello Kafka' stored in broker	'Hello Kafka' read by consumer	'Hello Kafka' processed	Processed

Key Moments - 2 Insights

Why does the consumer need to poll the broker to get messages?

Because Kafka stores messages in topics and consumers must request (poll) to receive new messages, as shown in step 3 of the execution_table.

What happens if the producer sends a message but the broker is down?

The message won't be stored and the producer won't get an acknowledgement, so the message is not confirmed as sent, unlike step 2 where the broker stores the message.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the message get stored in Kafka?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Check the 'Message State' column in step 2 where the message is stored in the topic partition.

According to variable_tracker, what is the state of 'message' after step 3?

A 'Hello Kafka' read by consumer

B None

C 'Hello Kafka' processed

D 'Hello Kafka' created

💡 Hint

Look at the 'After Step 3' column for 'message' in variable_tracker.

If the consumer never polls, what happens to the messages in Kafka?

A Messages are automatically deleted

B Messages are lost immediately

C Messages remain stored in the topic partition

D Messages are sent back to producer

💡 Hint

Kafka stores messages until consumers poll them, as shown in step 2 and 3.

Concept Snapshot

Message broker architecture in Kafka:
- Producers send messages to topics
- Kafka stores messages in partitions
- Consumers poll topics to read messages
- Messages remain until consumed or expired
- Broker acknowledges message receipt to producers

Full Transcript

This visual execution shows how a message flows in Kafka's message broker architecture. First, a producer creates and sends a message to a topic. The Kafka broker receives and stores the message in a topic partition, then acknowledges the producer. Consumers poll the broker to read messages from the topic. After reading, consumers process the messages. If no messages remain, polling waits or ends. Variables like 'message' change state from creation to processing. Key points include the need for consumers to poll to receive messages and the broker's role in storing messages reliably.