Why topics organize messages in Kafka - Why It Works

Introduction

When you send many messages in a system, you need a way to keep them organized. Kafka uses topics to group related messages together so you can find and use them easily.

When you want to separate messages by type, like orders and payments, so they don't mix up.

When you need to let different parts of your app listen only to messages they care about.

When you want to keep messages in order for a specific category, like all messages about one user.

When you want to scale your system by splitting messages into smaller groups.

When you want to store messages for a while and let many apps read them independently.

Commands

This command creates a new topic named 'orders' with 3 partitions. Partitions help organize messages inside the topic and allow parallel processing.

Terminal

kafka-topics --create --topic orders --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic orders.

→

--topic - Name of the topic to create

→

--partitions - Number of partitions to split the topic into

→

--replication-factor - Number of copies of the data for fault tolerance

This command lists all topics in the Kafka server so you can see the topics available to organize messages.

Terminal

kafka-topics --list --bootstrap-server localhost:9092

Expected Output

orders

→

--list - List all topics on the server

This command starts a producer that sends messages to the 'orders' topic. Messages sent here are organized under 'orders'.

Terminal

kafka-console-producer --topic orders --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Specify the topic to send messages to

This command reads messages from the 'orders' topic starting from the oldest message. It shows how messages are grouped and retrieved by topic.

Terminal

kafka-console-consumer --topic orders --bootstrap-server localhost:9092 --from-beginning --max-messages 3

Expected Output

order1

order2

order3

→

--from-beginning - Read messages from the start of the topic

→

--max-messages - Limit the number of messages to read

Key Concept

Topics in Kafka group related messages so you can organize, find, and process them easily.

Common Mistakes

Trying to send messages without creating a topic first

Kafka requires topics to exist before sending messages; otherwise, the send fails.

Always create the topic with kafka-topics before producing messages.

Using too few partitions for a high volume topic

Too few partitions limit parallel processing and can slow down message handling.

Create topics with enough partitions to handle your expected load.

Confusing topics with partitions

Partitions are parts inside a topic, not separate topics; mixing them up causes wrong assumptions about message grouping.

Remember that topics group messages broadly, partitions split them inside the topic for scaling.

Summary

Create topics to group related messages for better organization.

Use partitions inside topics to allow parallel processing and scaling.

Produce and consume messages by specifying the topic name to keep data organized.