Why producers publish data in Kafka - Why It Works

Introduction

Producers send data to Kafka topics so that other applications can use it. This helps share information quickly and reliably between different parts of a system.

When you want to send user activity logs from a website to a central system for analysis.

When an application needs to send sensor data from devices to a monitoring service.

When you want to stream order details from an online store to inventory and shipping systems.

When you need to collect real-time data from multiple sources for processing.

When you want to decouple parts of your system so they can work independently but still share data.

Commands

This command starts a producer that sends messages to the Kafka topic named 'example-topic'. You type messages and press enter to send each one.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka server address to connect to.

→

--topic - Specifies the topic where messages will be sent.

This command reads messages from the 'example-topic' starting from the oldest message. It shows the first 3 messages sent by the producer.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 3

Expected Output

Hello Kafka

This is a test

Data streaming

→

--bootstrap-server - Specifies the Kafka server address to connect to.

→

--topic - Specifies the topic to read messages from.

→

--from-beginning - Reads messages from the start of the topic.

→

--max-messages - Limits the number of messages to read.

Key Concept

Producers publish data to Kafka topics to share information with other parts of a system in real time.

Common Mistakes

Trying to send messages to a topic that does not exist.

Kafka will reject messages if the topic is missing or misnamed, so data is lost.

Create the topic first or verify the topic name before producing messages.

Not specifying the correct broker address when running the producer.

The producer cannot connect to Kafka and fails silently or shows connection errors.

Use the correct --broker-list or --bootstrap-server flag with the right Kafka server address.

Expecting messages to appear immediately without consuming them.

Messages are stored in Kafka but need a consumer to read and display them.

Run a consumer command to verify messages were published successfully.

Summary

Use kafka-console-producer to send messages to a Kafka topic.

Use kafka-console-consumer to read messages from the topic and verify data flow.

Always check topic existence and broker connection before producing data.