First message (produce and consume) in Kafka - Commands & Configuration

Introduction

Kafka helps you send messages between different parts of your system quickly and reliably. Producing means sending a message to Kafka, and consuming means reading that message from Kafka.

When you want to send a notification from one app to another without losing it.

When you need to log events from a website and process them later.

When you want to connect different services that work independently but share data.

When you want to build a chat app where messages are passed through a central system.

When you want to collect sensor data from devices and analyze it in real time.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

This is the Kafka broker configuration file.

broker.id identifies the Kafka server.

listeners tells Kafka where to listen for connections.

log.dirs is where Kafka stores messages.

zookeeper.connect points to the Zookeeper service Kafka uses to manage itself.

Commands

This command creates a new topic named 'my-first-topic' where messages will be sent and received.

Terminal

kafka-topics --create --topic my-first-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic my-first-topic.

→

--topic - Name of the topic to create

→

--partitions - Number of partitions for the topic

→

--replication-factor - Number of copies of the data for fault tolerance

This starts a producer that lets you type messages to send to the topic 'my-first-topic'. Type your message and press Enter to send.

Terminal

kafka-console-producer --topic my-first-topic --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka server address

This command reads the first message from the topic 'my-first-topic' starting from the beginning of the message log.

Terminal

kafka-console-consumer --topic my-first-topic --bootstrap-server localhost:9092 --from-beginning --max-messages 1

Expected Output

Hello Kafka

→

--from-beginning - Read messages from the start of the topic

→

--max-messages - Stop after reading this many messages

Key Concept

If you remember nothing else from this pattern, remember: producing sends messages to Kafka topics, and consuming reads them back in order.

Common Mistakes

Trying to consume messages before creating the topic

Kafka will show an error because the topic does not exist yet.

Always create the topic first using kafka-topics before producing or consuming messages.

Not specifying --from-beginning when consuming messages for the first time

The consumer will only read new messages from the time it starts, missing earlier messages.

Use --from-beginning flag to read all messages from the start of the topic.

Not running Kafka broker before producing or consuming

Commands will fail because Kafka server is not running to accept connections.

Start Kafka broker with the proper configuration before running producer or consumer commands.

Summary

Create a Kafka topic to hold messages using kafka-topics command.

Send messages to the topic using kafka-console-producer.

Read messages from the topic using kafka-console-consumer with --from-beginning to see all messages.