Why topics organize messages in Kafka - Visual Breakdown

Process Flow - Why topics organize messages

Producer sends message

↓

Message goes to Topic

↓

Topic stores messages in order

↓

Consumers read messages from Topic

↓

Consumers process messages in order

Messages are grouped in topics to keep them organized and in order, so producers send messages to topics, and consumers read them from there.

Execution Sample

Kafka

producer.send('orders', 'order1')
producer.send('orders', 'order2')
consumer.read('orders')

This code sends two messages to the 'orders' topic and then a consumer reads messages from it in order.

Process Table

Step	Action	Topic State	Message Sent/Read	Output
1	Producer sends 'order1' to 'orders'	orders: ['order1']	Sent 'order1'	None
2	Producer sends 'order2' to 'orders'	orders: ['order1', 'order2']	Sent 'order2'	None
3	Consumer reads from 'orders'	orders: ['order1', 'order2']	Read 'order1'	'order1' processed
4	Consumer reads next message	orders: ['order1', 'order2']	Read 'order2'	'order2' processed
5	No more messages	orders: ['order1', 'order2']	None	Consumer waits

💡 Consumer stops reading when no new messages are available in the topic.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Topic 'orders'	[]	['order1']	['order1', 'order2']	['order1', 'order2']	['order1', 'order2']	['order1', 'order2']
Consumer position	0	0	0	1	2	2

Key Moments - 2 Insights

Why do messages go into a topic instead of directly to consumers?

Messages go into a topic to keep them organized and stored in order, so multiple consumers can read them independently and reliably, as shown in steps 1 and 3 of the execution_table.

Does the topic remove messages after consumers read them?

No, the topic keeps messages until they expire or are deleted by retention policy. Consumers track their own position, as seen in the variable_tracker where the topic state stays the same after reads.

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what is the state of the 'orders' topic?

A ['order1']

B ['order2']

C ['order1', 'order2']

D []

💡 Hint

Check the 'Topic State' column at step 2 in the execution_table.

At which step does the consumer read the first message?

A Step 1

B Step 3

C Step 4

D Step 5

💡 Hint

Look at the 'Message Sent/Read' column in the execution_table.

If the producer sends a third message, how would the topic state change after step 2?

A ['order1', 'order2', 'order3']

B ['order3']

C ['order1', 'order3']

D No change

💡 Hint

Refer to how the topic state changes when messages are sent in the execution_table steps 1 and 2.

Concept Snapshot

Kafka topics organize messages by grouping them in named streams.
Producers send messages to topics.
Topics store messages in order.
Consumers read messages from topics independently.
This keeps messages organized and allows multiple consumers.

Full Transcript

In Kafka, messages are organized using topics. Producers send messages to a topic, which stores them in order. Consumers then read messages from the topic in the same order. This organization helps keep messages grouped and allows multiple consumers to read independently without losing messages. The topic keeps messages until they expire or are deleted, and consumers track their reading position. This way, Kafka ensures reliable and ordered message delivery.

Practice

1. Why does Kafka use topics to organize messages?

easy

A. To group related messages together for easier management

B. To store messages permanently on the producer side

C. To encrypt messages automatically

D. To limit the number of messages sent

Solution

1. Step 1: Understand the purpose of topics in Kafka

   Topics are used to group messages that belong to the same category or stream.

2. Step 2: Identify the role of topics in message management

   Grouping messages helps producers and consumers handle data efficiently and logically.

3. Final Answer:

   To group related messages together for easier management -> Option A

4. Quick Check:

   Topics group messages = A [OK]

Hint: Topics group messages logically for easy handling [OK]

Common Mistakes:

• Thinking topics store messages on producer side
• Confusing topics with encryption features
• Believing topics limit message quantity

2. Which of the following is the correct way to create a topic in Kafka using the command line?

easy

A. kafka topic create myTopic --partitions 3

B. kafka-topics --create --topic myTopic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

C. create kafka topic myTopic partitions=3

D. kafka-create-topic --name myTopic --partitions 3

Solution

1. Step 1: Recall the Kafka CLI syntax for topic creation

   The correct command uses 'kafka-topics' with '--create' and required flags.

2. Step 2: Match the command with the correct options

   kafka-topics --create --topic myTopic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092 matches the official syntax including bootstrap-server, partitions, and replication-factor.

3. Final Answer:

   kafka-topics --create --topic myTopic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092 -> Option B

4. Quick Check:

   Correct Kafka topic creation syntax = D [OK]

Hint: Use kafka-topics with --create and required flags [OK]

Common Mistakes:

• Using incorrect command names like kafka-create-topic
• Omitting bootstrap-server flag
• Wrong order or missing required flags

3. Given a Kafka topic with 3 partitions, what happens when a producer sends 6 messages without specifying a key?

medium

A. Messages are sent randomly to any partition

B. All messages go to the first partition

C. Messages are distributed evenly across the 3 partitions

D. Messages are rejected because no key is provided

Solution

1. Step 1: Understand default partitioning behavior without a key

   When no key is provided, Kafka uses round-robin to distribute messages evenly across partitions.

2. Step 2: Apply this to 6 messages and 3 partitions

   Messages will be sent 2 per partition, balancing the load.

3. Final Answer:

   Messages are distributed evenly across the 3 partitions -> Option C

4. Quick Check:

   No key means round-robin distribution = A [OK]

Hint: No key means even round-robin distribution [OK]

Common Mistakes:

• Assuming all messages go to one partition
• Thinking messages are rejected without a key
• Believing distribution is random without pattern

4. You have a Kafka topic with 2 partitions but your consumer only reads from partition 0. What is the likely problem?

medium

A. Kafka does not support multiple partitions

B. The topic only has one partition

C. The producer is sending messages only to partition 1

D. The consumer is not configured to subscribe to all partitions

Solution

1. Step 1: Analyze consumer subscription behavior

   If a consumer reads only from partition 0, it likely subscribed only to that partition explicitly.

2. Step 2: Check topic partition count and producer behavior

   The topic has 2 partitions, so consumer should subscribe to both to read all messages.

3. Final Answer:

   The consumer is not configured to subscribe to all partitions -> Option D

4. Quick Check:

   Consumer subscription controls partitions read = C [OK]

Hint: Consumer must subscribe to all partitions to read all messages [OK]

Common Mistakes:

• Assuming topic has only one partition
• Believing Kafka does not support multiple partitions
• Thinking producer controls consumer reading partitions

5. How can partitioning in Kafka topics improve performance when handling large message streams?

hard

A. By allowing parallel processing of messages across multiple consumers

B. By encrypting messages faster

C. By reducing the total number of messages sent

D. By storing messages only on the producer side

Solution

1. Step 1: Understand the role of partitions in Kafka topics

   Partitions split a topic into multiple parts that can be processed independently.

2. Step 2: Connect partitions to performance benefits

   Multiple consumers can read from different partitions in parallel, increasing throughput and speed.

3. Final Answer:

   By allowing parallel processing of messages across multiple consumers -> Option A

4. Quick Check:

   Partitions enable parallelism = B [OK]

Hint: Partitions enable parallel consumer processing [OK]

Common Mistakes:

• Confusing partitioning with encryption
• Thinking partitioning reduces message count
• Believing messages are stored only on producer