Idempotent producer in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output when sending messages with idempotent producer enabled?

Consider the following Kafka producer configuration and code snippet. What will be the output in the Kafka topic after running this code?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

for (int i = 0; i < 3; i++) {
    producer.send(new ProducerRecord<>("my-topic", "key1", "message"));
}
producer.flush();
producer.close();

A The topic will contain 3 identical messages with key 'key1' and value 'message'.

B The topic will contain 1 message with key 'key1' and value 'message' because idempotence prevents duplicates.

C The topic will contain no messages because the producer is idempotent and suppresses all sends.

D The code will throw a runtime exception because sending multiple messages with the same key is not allowed.

💡 Hint

Idempotent producer prevents duplicates caused by retries, not multiple sends in the same session.

Idempotent producer ensures that retries of the same message do not result in duplicates. However, sending the same message multiple times intentionally will produce multiple messages in the topic.

❓ Predict Output

intermediate

What error occurs if 'enable.idempotence' is set to false but 'acks' is set to 'all'?

Given the following Kafka producer configuration, what will happen when sending messages?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "false");
props.put("acks", "all");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("my-topic", "key", "value"));
producer.flush();
producer.close();

A The producer sends messages but with a warning logged about inconsistent configuration.

B The producer throws a ConfigException because 'acks=all' requires 'enable.idempotence=true'.

C The producer silently enables idempotence despite the configuration to ensure data safety.

D The producer sends messages successfully with full acknowledgment but without idempotence guarantees.

💡 Hint

Idempotence and acks are related but not strictly dependent in Kafka configuration.

'acks=all' means the leader waits for all replicas to acknowledge. Idempotence is optional and disabled here, so messages send successfully but without idempotent guarantees.

🔧 Debug

advanced

Why does this idempotent producer code cause duplicate messages after a network failure?

Examine the code below. Despite enabling idempotence, duplicate messages appear in the topic after a network failure. What is the cause?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

for (int i = 0; i < 5; i++) {
    producer.send(new ProducerRecord<>("my-topic", "key", "message"));
    if (i == 2) {
        // Simulate network failure by closing producer abruptly
        producer.close();
        producer = new KafkaProducer<>(props);
    }
}
producer.flush();
producer.close();

A Idempotence only works within a single producer instance; closing and reopening resets sequence numbers causing duplicates.

B The producer configuration is missing 'acks=all', so duplicates occur despite idempotence.

C The topic is configured with multiple partitions, so duplicates are expected with idempotence enabled.

D The key serializer is incorrect, causing messages to be treated as different keys and duplicated.

💡 Hint

Idempotence tracks sequence numbers per producer instance to avoid duplicates.

When the producer is closed and reopened, it loses its sequence number state. This causes Kafka to treat messages as new and duplicates can occur after retries.

📝 Syntax

advanced

Which Kafka producer configuration snippet correctly enables idempotence with safe retries?

Select the configuration snippet that correctly enables idempotence and configures retries safely.

A props.put("enable.idempotence", true); props.put("retries", -1); props.put("acks", "1");

B props.put("enable.idempotence", "true"); props.put("retries", Integer.MAX_VALUE); props.put("acks", "all");

C props.put("enable.idempotence", "false"); props.put("retries", 5); props.put("acks", "all");

D props.put("enable.idempotence", "true"); props.put("retries", 0); props.put("acks", "all");

💡 Hint

Idempotence requires 'acks=all' and retries enabled to be effective.

Option B correctly sets 'enable.idempotence' to string "true", retries to max integer, and 'acks' to 'all'. Option B uses invalid retries (-1) and 'acks' to '1'. Option B disables idempotence. Option B disables retries.

🚀 Application

expert

How to ensure exactly-once semantics in Kafka producer with idempotence enabled?

You want to guarantee that messages are delivered exactly once to a Kafka topic using an idempotent producer. Which approach below correctly achieves this?

A Enable idempotence and set 'acks' to '1' to reduce latency and rely on retries for duplicates.

B Disable idempotence but use manual offset commits in the consumer to avoid duplicates.

C Enable idempotence, set 'acks' to 'all', configure infinite retries, and use transactions to commit messages atomically.

D Enable idempotence and set 'retries' to 0 to avoid resending messages and duplicates.

💡 Hint

Exactly-once semantics require atomic writes and idempotent producer with proper acknowledgments.

Exactly-once delivery requires idempotence, 'acks=all', infinite retries to handle failures, and transactions to atomically commit messages. Other options either reduce reliability or do not guarantee exactly-once.

Practice

1.

What is the main purpose of enabling enable.idempotence=true in a Kafka producer?

easy

A. To enable message encryption

B. To prevent duplicate messages from being sent to Kafka topics

C. To compress messages before sending

D. To increase the speed of message delivery

Solution

1. Step 1: Understand idempotence in Kafka producer

   Idempotence means sending the same message multiple times results in only one message stored.

2. Step 2: Identify the effect of enabling idempotence

   Setting enable.idempotence=true ensures no duplicate messages are produced even if retries happen.

3. Final Answer:

   To prevent duplicate messages from being sent to Kafka topics -> Option B

4. Quick Check:

   Idempotence prevents duplicates = D [OK]

Hint: Idempotence means no duplicates sent [OK]

Common Mistakes:

• Confusing idempotence with message speed
• Thinking it compresses messages
• Assuming it encrypts data

2.

Which of the following is the correct way to enable idempotence in Kafka producer configuration?

producerProps = new Properties();
// Which line correctly enables idempotence?

easy

A. producerProps.put("enableIdempotence", true);

B. producerProps.put("idempotent.enable", "true");

C. producerProps.put("enable.idempotence", "true");

D. producerProps.put("enable.idempotence", true);

Solution

1. Step 1: Check Kafka producer config key for idempotence

   The correct key is exactly "enable.idempotence" as a string.

2. Step 2: Check value type and syntax

   Values in Properties must be strings, so "true" as a string is correct.

3. Final Answer:

   producerProps.put("enable.idempotence", "true"); -> Option C

4. Quick Check:

   Correct key and string value = C [OK]

Hint: Use exact key "enable.idempotence" with string "true" [OK]

Common Mistakes:

• Using wrong key name like "idempotent.enable"
• Passing boolean true instead of string "true"
• Using camelCase key instead of dot notation

3.

Given the following Kafka producer configuration snippet, what will be the effect when sending messages?

props.put("enable.idempotence", "true");
props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);

Choose the best description of the behavior.

medium

A. Messages are guaranteed to be delivered exactly once without duplicates

B. Messages may be duplicated if retries happen

C. Messages will be lost if broker is down

D. Producer will not retry sending messages

Solution

1. Step 1: Analyze idempotence and acks settings

   Enabling idempotence with acks=all ensures leader and replicas confirm message receipt.

2. Step 2: Consider retries setting

   Unlimited retries mean producer will keep trying until success, avoiding message loss or duplicates.

3. Final Answer:

   Messages are guaranteed to be delivered exactly once without duplicates -> Option A

4. Quick Check:

   Idempotence + acks=all + retries = exactly once [OK]

Hint: Idempotence + acks=all + retries = no duplicates [OK]

Common Mistakes:

• Thinking duplicates still happen with idempotence
• Assuming messages can be lost with retries set
• Ignoring the importance of acks=all

4.

Identify the error in this Kafka producer configuration snippet that aims to enable idempotence:

Properties props = new Properties();
props.put("enable.idempotence", true);
props.put("acks", "all");
props.put("retries", 5);

medium

A. The value for "enable.idempotence" should be a string "true", not boolean true

B. "acks" should be set to "1" instead of "all"

C. Retries cannot be set to 5 when idempotence is enabled

D. Properties object cannot be used for Kafka configuration

Solution

1. Step 1: Check the type of value for "enable.idempotence" key

   Kafka Properties expect string values, but here boolean true is used.

2. Step 2: Understand impact of wrong value type

   Using boolean instead of string may cause config to be ignored or error.

3. Final Answer:

   The value for "enable.idempotence" should be a string "true", not boolean true -> Option A

4. Quick Check:

   Config values must be strings = A [OK]

Hint: Config values in Properties must be strings [OK]

Common Mistakes:

• Using boolean true instead of string "true"
• Changing acks to 1 which weakens guarantees
• Thinking retries=5 is invalid with idempotence

5.

You want to ensure your Kafka producer sends messages exactly once even if the network is unstable. Which combination of settings should you use?

// Choose the best configuration combination

hard

A. enable.idempotence=false, acks=1, retries=0

B. enable.idempotence=false, acks=all, retries=Integer.MAX_VALUE

C. enable.idempotence=true, acks=0, retries=3

D. enable.idempotence=true, acks=all, retries=Integer.MAX_VALUE

Solution

1. Step 1: Identify settings for exactly-once delivery

   Idempotence must be enabled to avoid duplicates on retries.

2. Step 2: Check acknowledgement and retry settings

   Setting acks=all ensures full commit, and unlimited retries prevent message loss.

3. Final Answer:

   enable.idempotence=true, acks=all, retries=Integer.MAX_VALUE -> Option D

4. Quick Check:

   Idempotence + acks=all + max retries = exactly once [OK]

Hint: Enable idempotence + acks=all + max retries for exactly once [OK]

Common Mistakes:

• Disabling idempotence but expecting no duplicates
• Using acks=0 or 1 which weakens delivery guarantees
• Setting retries to 0 causing message loss