Producer retries and idempotency in Kafka - Commands & Configuration

Introduction

When a Kafka producer sends messages, sometimes network issues or server errors cause failures. Producer retries help resend messages automatically. Idempotency ensures that even if a message is sent multiple times, it is stored only once, avoiding duplicates.

When your application must not lose messages even if temporary network problems occur

When you want to avoid duplicate messages in Kafka caused by retries

When you have a critical data pipeline where message duplication can cause errors downstream

When you want to improve reliability of message delivery without manual intervention

When you want Kafka to handle retry logic safely without risking data corruption

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
acks=all
retries=5
enable.idempotence=true
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

bootstrap.servers: Kafka server address.
acks=all: Wait for all replicas to confirm message.
retries=5: Retry sending message up to 5 times on failure.
enable.idempotence=true: Enable idempotent producer to avoid duplicates.
key.serializer and value.serializer: Convert data to bytes for Kafka.

Commands

Start a Kafka producer using the configuration file with retries and idempotency enabled. This ensures messages are retried safely and duplicates are avoided.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer.config producer.properties

Expected Output

> This is a test message

> Another message

→

--broker-list - Specifies Kafka server addresses to connect

→

--topic - Specifies the Kafka topic to send messages to

→

--producer.config - Loads producer settings from a configuration file

Consume the first two messages from the topic to verify that messages were sent once without duplicates.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 2

Expected Output

This is a test message

Another message

→

--bootstrap-server - Specifies Kafka server address

→

--topic - Specifies the topic to read messages from

→

--from-beginning - Reads messages from the start of the topic

→

--max-messages - Limits the number of messages to consume

Key Concept

If you remember nothing else from this pattern, remember: enabling idempotency with retries guarantees each message is stored exactly once even if sent multiple times.

Common Mistakes

Not setting enable.idempotence=true when using retries

This can cause duplicate messages because retries resend messages without deduplication.

Always enable idempotency in the producer configuration when using retries.

Setting retries to a very high number without idempotency

High retries increase duplicate risk and can overload Kafka brokers.

Use moderate retries with idempotency enabled to balance reliability and safety.

Using acks=1 or 0 with retries enabled

This can cause message loss or duplicates because the producer does not wait for full confirmation.

Set acks=all to ensure full commit before considering a message sent.

Summary

Configure the Kafka producer with retries and enable idempotency to avoid duplicate messages.

Use acks=all to ensure messages are fully committed before confirming success.

Verify message delivery by consuming from the topic and checking for duplicates.

Practice

1. What is the main purpose of enabling retries in a Kafka producer?

easy

A. To resend messages automatically if sending fails

B. To speed up message delivery

C. To prevent duplicate messages

D. To encrypt messages before sending

Solution

1. Step 1: Understand retries in Kafka producer

   Retries allow the producer to resend messages if the first attempt fails due to temporary issues.

2. Step 2: Identify the main purpose

   The main goal is to improve message delivery success by retrying failed sends.

3. Final Answer:

   To resend messages automatically if sending fails -> Option A

4. Quick Check:

   Retries = Resend on failure [OK]

Hint: Retries mean automatic resend on failure [OK]

Common Mistakes:

• Confusing retries with idempotency
• Thinking retries speed up delivery
• Assuming retries encrypt messages

2. Which configuration must be set to true to enable idempotency in Kafka producer?

easy

A. retries = 0

B. enable.idempotency = false

C. enable.idempotency = true

D. acks = 0

Solution

1. Step 1: Identify idempotency configuration

   The Kafka producer uses enable.idempotency to control idempotency behavior.

2. Step 2: Determine correct setting

   Setting enable.idempotency to true activates idempotency to avoid duplicates.

3. Final Answer:

   enable.idempotency = true -> Option C

4. Quick Check:

   Idempotency enabled = true [OK]

Hint: Idempotency config is enable.idempotency = true [OK]

Common Mistakes:

• Setting enable.idempotency to false
• Confusing retries with idempotency config
• Using acks = 0 disables guarantees

3. Given the following Kafka producer settings:

enable.idempotency = true
retries = 3
acks = 'all'

What is the expected behavior if a message send fails temporarily?

medium

A. The producer will stop sending messages

B. The message will be sent only once, no retries

C. Duplicates may occur due to retries

D. The message will be retried up to 3 times without duplicates

Solution

1. Step 1: Analyze retries and idempotency settings

   Retries = 3 means the producer will retry sending up to 3 times on failure. Idempotency = true prevents duplicates.

2. Step 2: Understand combined effect

   Retries with idempotency ensure retries do not cause duplicate messages.

3. Final Answer:

   The message will be retried up to 3 times without duplicates -> Option D

4. Quick Check:

   Retries + idempotency = safe retries [OK]

Hint: Retries with idempotency avoid duplicates [OK]

Common Mistakes:

• Assuming retries cause duplicates always
• Ignoring idempotency effect
• Thinking producer stops after failure

4. You have a Kafka producer with retries=5 but enable.idempotency=false. What problem might occur?

medium

A. Messages will be encrypted automatically

B. Duplicate messages may be produced on retries

C. Producer will crash on first failure

D. Messages will never be sent

Solution

1. Step 1: Understand retries without idempotency

   Retries resend messages on failure, but without idempotency duplicates can happen.

2. Step 2: Identify the problem

   Without idempotency, each retry may produce the same message again, causing duplicates.

3. Final Answer:

   Duplicate messages may be produced on retries -> Option B

4. Quick Check:

   No idempotency + retries = duplicates risk [OK]

Hint: Retries without idempotency cause duplicates [OK]

Common Mistakes:

• Thinking producer crashes on failure
• Assuming messages never send
• Confusing encryption with idempotency

5. You want to ensure exactly-once message delivery in Kafka with retries enabled. Which combination of settings should you use?

hard

A. enable.idempotency=true, retries>0, acks='all'

B. enable.idempotency=false, retries=0, acks=1

C. enable.idempotency=true, retries=0, acks=0

D. enable.idempotency=false, retries>0, acks=0

Solution

1. Step 1: Identify settings for exactly-once delivery

   Exactly-once requires idempotency enabled, retries to resend on failure, and acks='all' for full commit guarantee.

2. Step 2: Evaluate each option

   enable.idempotency=true, retries>0, acks='all' has all required settings. Others disable idempotency or retries or use weak acks.

3. Final Answer:

   enable.idempotency=true, retries>0, acks='all' -> Option A

4. Quick Check:

   Idempotency + retries + acks='all' = exactly-once [OK]

Hint: Enable idempotency, retries, and acks='all' for exactly-once [OK]

Common Mistakes:

• Disabling idempotency with retries
• Using acks=0 or 1 for strong guarantees
• Setting retries=0 disables resend