Producer API basics in Kafka - Commands & Configuration

Introduction

Sending messages to a Kafka topic is done using the Producer API. It helps you put data into Kafka so other applications can read it later.

When you want to send user activity logs from a web app to Kafka for processing.

When you need to push sensor data from IoT devices into Kafka for real-time analysis.

When your application generates events that other services should consume asynchronously.

When you want to batch and send messages efficiently to Kafka topics.

When you need to ensure messages are sent reliably with acknowledgments.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
acks=all
retries=3
batch.size=16384
linger.ms=1
buffer.memory=33554432

bootstrap.servers: Kafka server addresses to connect to.

key.serializer and value.serializer: Convert keys and values to bytes.

acks: Wait for all replicas to confirm message receipt for reliability.

retries: Number of times to retry sending on failure.

batch.size and linger.ms: Control batching for efficiency.

buffer.memory: Memory size for buffering messages before sending.

Commands

Starts a simple command-line Kafka producer to send messages to 'example-topic'. You type messages and press Enter to send each one.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka server addresses to connect to.

→

--topic - Specifies the Kafka topic to send messages to.

Sends a single message 'Hello Kafka' to 'example-topic' using the console producer.

Terminal

echo "Hello Kafka" | kafka-console-producer --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka server addresses to connect to.

→

--topic - Specifies the Kafka topic to send messages to.

Reads one message from 'example-topic' starting from the beginning to verify the message was sent.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 1

Expected Output

Hello Kafka

→

--bootstrap-server - Specifies Kafka server addresses to connect to.

→

--topic - Specifies the Kafka topic to read messages from.

→

--from-beginning - Reads messages from the start of the topic.

→

--max-messages - Limits the number of messages to read.

Key Concept

If you remember nothing else from this pattern, remember: the Producer API sends messages to Kafka topics so other apps can consume them later.

Common Mistakes

Not setting the correct serializers for key and value.

Kafka needs to convert data to bytes; wrong serializers cause errors or unreadable messages.

Always set key.serializer and value.serializer to match your data types, like StringSerializer for text.

Using the wrong broker address or port.

The producer cannot connect to Kafka, so messages fail to send.

Verify Kafka is running and use the correct host and port in bootstrap.servers or --broker-list.

Not verifying messages by consuming them after sending.

You might think messages were sent but they were not, causing silent failures.

Use kafka-console-consumer to read messages and confirm delivery.

Summary

Use kafka-console-producer to send messages to a Kafka topic interactively or via echo.

Configure producer properties like serializers and broker addresses in a properties file.

Verify messages are sent by consuming them with kafka-console-consumer.