Producer retries and idempotency in Kafka - Commands & Configuration

Introduction

When a Kafka producer sends messages, sometimes network issues or server errors cause failures. Producer retries help resend messages automatically. Idempotency ensures that even if a message is sent multiple times, it is stored only once, avoiding duplicates.

When your application must not lose messages even if temporary network problems occur

When you want to avoid duplicate messages in Kafka caused by retries

When you have a critical data pipeline where message duplication can cause errors downstream

When you want to improve reliability of message delivery without manual intervention

When you want Kafka to handle retry logic safely without risking data corruption

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
acks=all
retries=5
enable.idempotence=true
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

bootstrap.servers: Kafka server address.
acks=all: Wait for all replicas to confirm message.
retries=5: Retry sending message up to 5 times on failure.
enable.idempotence=true: Enable idempotent producer to avoid duplicates.
key.serializer and value.serializer: Convert data to bytes for Kafka.

Commands

Start a Kafka producer using the configuration file with retries and idempotency enabled. This ensures messages are retried safely and duplicates are avoided.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer.config producer.properties

Expected Output

> This is a test message

> Another message

→

--broker-list - Specifies Kafka server addresses to connect

→

--topic - Specifies the Kafka topic to send messages to

→

--producer.config - Loads producer settings from a configuration file

Consume the first two messages from the topic to verify that messages were sent once without duplicates.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 2

Expected Output

This is a test message

Another message

→

--bootstrap-server - Specifies Kafka server address

→

--topic - Specifies the topic to read messages from

→

--from-beginning - Reads messages from the start of the topic

→

--max-messages - Limits the number of messages to consume

Key Concept

If you remember nothing else from this pattern, remember: enabling idempotency with retries guarantees each message is stored exactly once even if sent multiple times.

Common Mistakes

Not setting enable.idempotence=true when using retries

This can cause duplicate messages because retries resend messages without deduplication.

Always enable idempotency in the producer configuration when using retries.

Setting retries to a very high number without idempotency

High retries increase duplicate risk and can overload Kafka brokers.

Use moderate retries with idempotency enabled to balance reliability and safety.

Using acks=1 or 0 with retries enabled

This can cause message loss or duplicates because the producer does not wait for full confirmation.

Set acks=all to ensure full commit before considering a message sent.

Summary

Configure the Kafka producer with retries and enable idempotency to avoid duplicate messages.

Use acks=all to ensure messages are fully committed before confirming success.

Verify message delivery by consuming from the topic and checking for duplicates.