Idempotent producer in Kafka - Commands & Configuration

Introduction

When sending messages to Kafka, sometimes duplicates happen if the network or system retries. An idempotent producer makes sure that even if it sends the same message multiple times, Kafka stores it only once. This helps keep data clean and consistent.

When you want to avoid duplicate messages in Kafka caused by retries.

When your application needs exactly-once message delivery guarantees.

When network issues cause message resending and you want to prevent duplicates.

When processing financial transactions or orders where duplicates cause errors.

When you want Kafka to handle message uniqueness automatically without extra code.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
enable.idempotence=true
acks=all
retries=5
max.in.flight.requests.per.connection=5

This configuration file sets up a Kafka producer with idempotence enabled.

• bootstrap.servers: Kafka server address.
• key.serializer and value.serializer: How to convert keys and values to bytes.
• enable.idempotence=true: Turns on idempotent producer to avoid duplicates.
• acks=all: Waits for all replicas to confirm message receipt for safety.
• retries=5: Number of times to retry sending on failure.
• max.in.flight.requests.per.connection=5: Controls how many messages can be sent without waiting for acknowledgment, balancing speed and ordering.

Commands

Starts a Kafka producer using the configuration file with idempotence enabled to send messages to 'example-topic'.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer.config producer.properties

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka server addresses.

→

--topic - Specifies the topic to send messages to.

→

--producer.config - Loads producer settings from the given file.

Sends a single message 'Hello Kafka' to the topic using the idempotent producer configuration.

Terminal

echo "Hello Kafka" | kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer.config producer.properties

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka server addresses.

→

--topic - Specifies the topic to send messages to.

→

--producer.config - Loads producer settings from the given file.

Reads the first message from 'example-topic' to verify that the message was sent once without duplicates.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 1

Expected Output

Hello Kafka

→

--bootstrap-server - Specifies Kafka server addresses.

→

--topic - Specifies the topic to read messages from.

→

--from-beginning - Reads messages from the start of the topic.

→

--max-messages - Limits the number of messages to read.

Key Concept

If you remember nothing else from this pattern, remember: enabling idempotence in the Kafka producer prevents duplicate messages even if retries happen.

Common Mistakes

Not setting enable.idempotence=true in the producer configuration.

Without this setting, Kafka does not guarantee no duplicates on retries.

Always include enable.idempotence=true in your producer properties to activate idempotent behavior.

Setting acks to 0 or 1 instead of all.

Lower acks values reduce reliability and can cause duplicates or lost messages.

Set acks=all to ensure all replicas confirm the message for safe delivery.

Using max.in.flight.requests.per.connection higher than 5.

Higher values can cause message reordering which breaks idempotence guarantees.

Keep max.in.flight.requests.per.connection at 5 or less to maintain ordering.

Summary

Create a producer configuration file with enable.idempotence=true to avoid duplicate messages.

Use kafka-console-producer with the config file to send messages safely.

Verify message delivery with kafka-console-consumer to confirm no duplicates.