Producer retries and idempotency in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output when a Kafka producer with idempotency enabled retries sending a message?

Consider a Kafka producer configured with enable.idempotence=true and retries=3. If the producer sends a message and a transient network error occurs causing a retry, what will be the effect on the topic?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");
props.put("retries", 3);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "value1");
producer.send(record);
producer.close();

A The message will be lost if a retry happens because idempotency disables retries.

B The message will be delivered exactly once to the topic, even if retries occur.

C The message may be duplicated in the topic because retries resend the message multiple times.

D The producer will throw an exception and stop sending messages on the first retry.

💡 Hint

Idempotency ensures no duplicate messages even if retries happen.

When enable.idempotence=true, Kafka producer assigns a unique sequence number to each message. If a retry occurs, the broker recognizes duplicates and prevents them from being written multiple times. This guarantees exactly-once delivery despite retries.

🧠 Conceptual

intermediate

Why is idempotency important when enabling retries in Kafka producers?

Select the best explanation for why enabling idempotency is critical when a Kafka producer is configured to retry sending messages.

A Idempotency prevents duplicate messages caused by retries, ensuring message uniqueness.

B Idempotency increases throughput by batching messages during retries.

C Idempotency disables retries to avoid message duplication.

D Idempotency encrypts messages to secure them during retries.

💡 Hint

Think about what happens if the same message is sent multiple times.

Retries can cause the same message to be sent multiple times. Idempotency ensures that even if the producer retries, the broker will write the message only once, preventing duplicates.

🔧 Debug

advanced

Identify the error in this Kafka producer configuration for retries and idempotency

Given the following Kafka producer configuration, what error will occur when sending messages with retries and idempotency enabled?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");
props.put("retries", -1);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "value1");
producer.send(record);
producer.close();

A The producer will disable idempotency automatically due to invalid retries.

B The producer will ignore the retries setting and send messages only once.

C The producer will send infinite retries without error.

D The producer will throw an IllegalArgumentException because retries cannot be negative.

💡 Hint

Check the allowed values for the retries configuration.

The retries configuration must be zero or a positive integer. Setting it to -1 is invalid and causes an exception when creating the producer.

❓ Predict Output

advanced

What happens if a Kafka producer with idempotency enabled sends messages with different keys but same value during retries?

Consider a Kafka producer with enable.idempotence=true sending two messages with the same value but different keys. If a retry occurs for one message, what will be the result in the topic?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");
props.put("retries", 3);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

producer.send(new ProducerRecord<>("my-topic", "key1", "value"));
producer.send(new ProducerRecord<>("my-topic", "key2", "value"));
producer.close();

A Both messages will be delivered exactly once, even if retries happen, because keys differ.

B The message with the same value will be deduplicated and only one will appear in the topic.

C The producer will fail to send the second message due to duplicate value.

D Retries will cause duplicate messages for both keys because idempotency only works per value.

💡 Hint

Idempotency works per producer sequence, not by message content.

Idempotency prevents duplicates by tracking sequence numbers per partition and producer. Messages with different keys are treated as separate records. Even if retries occur, each message is delivered once.

🧠 Conceptual

expert

How does Kafka ensure exactly-once delivery semantics with producer retries and idempotency enabled?

Which mechanism best describes how Kafka achieves exactly-once delivery when a producer has retries and idempotency enabled?

A Kafka stores all messages in a temporary buffer and only commits them after all retries succeed.

B Kafka disables retries and uses transactional commits to guarantee exactly-once delivery.

C Kafka assigns a unique producer ID and sequence number to each message, allowing the broker to detect and discard duplicates during retries.

D Kafka encrypts messages with unique keys to prevent duplicates during retries.

💡 Hint

Think about how the broker can recognize duplicate messages from the same producer.

Kafka uses a combination of a unique producer ID and a monotonically increasing sequence number per partition. When retries happen, the broker compares sequence numbers to detect duplicates and only writes the first occurrence, ensuring exactly-once delivery.

Practice

1. What is the main purpose of enabling retries in a Kafka producer?

easy

A. To resend messages automatically if sending fails

B. To speed up message delivery

C. To prevent duplicate messages

D. To encrypt messages before sending

Solution

1. Step 1: Understand retries in Kafka producer

   Retries allow the producer to resend messages if the first attempt fails due to temporary issues.

2. Step 2: Identify the main purpose

   The main goal is to improve message delivery success by retrying failed sends.

3. Final Answer:

   To resend messages automatically if sending fails -> Option A

4. Quick Check:

   Retries = Resend on failure [OK]

Hint: Retries mean automatic resend on failure [OK]

Common Mistakes:

• Confusing retries with idempotency
• Thinking retries speed up delivery
• Assuming retries encrypt messages

2. Which configuration must be set to true to enable idempotency in Kafka producer?

easy

A. retries = 0

B. enable.idempotency = false

C. enable.idempotency = true

D. acks = 0

Solution

1. Step 1: Identify idempotency configuration

   The Kafka producer uses enable.idempotency to control idempotency behavior.

2. Step 2: Determine correct setting

   Setting enable.idempotency to true activates idempotency to avoid duplicates.

3. Final Answer:

   enable.idempotency = true -> Option C

4. Quick Check:

   Idempotency enabled = true [OK]

Hint: Idempotency config is enable.idempotency = true [OK]

Common Mistakes:

• Setting enable.idempotency to false
• Confusing retries with idempotency config
• Using acks = 0 disables guarantees

3. Given the following Kafka producer settings:

enable.idempotency = true
retries = 3
acks = 'all'

What is the expected behavior if a message send fails temporarily?

medium

A. The producer will stop sending messages

B. The message will be sent only once, no retries

C. Duplicates may occur due to retries

D. The message will be retried up to 3 times without duplicates

Solution

1. Step 1: Analyze retries and idempotency settings

   Retries = 3 means the producer will retry sending up to 3 times on failure. Idempotency = true prevents duplicates.

2. Step 2: Understand combined effect

   Retries with idempotency ensure retries do not cause duplicate messages.

3. Final Answer:

   The message will be retried up to 3 times without duplicates -> Option D

4. Quick Check:

   Retries + idempotency = safe retries [OK]

Hint: Retries with idempotency avoid duplicates [OK]

Common Mistakes:

• Assuming retries cause duplicates always
• Ignoring idempotency effect
• Thinking producer stops after failure

4. You have a Kafka producer with retries=5 but enable.idempotency=false. What problem might occur?

medium

A. Messages will be encrypted automatically

B. Duplicate messages may be produced on retries

C. Producer will crash on first failure

D. Messages will never be sent

Solution

1. Step 1: Understand retries without idempotency

   Retries resend messages on failure, but without idempotency duplicates can happen.

2. Step 2: Identify the problem

   Without idempotency, each retry may produce the same message again, causing duplicates.

3. Final Answer:

   Duplicate messages may be produced on retries -> Option B

4. Quick Check:

   No idempotency + retries = duplicates risk [OK]

Hint: Retries without idempotency cause duplicates [OK]

Common Mistakes:

• Thinking producer crashes on failure
• Assuming messages never send
• Confusing encryption with idempotency

5. You want to ensure exactly-once message delivery in Kafka with retries enabled. Which combination of settings should you use?

hard

A. enable.idempotency=true, retries>0, acks='all'

B. enable.idempotency=false, retries=0, acks=1

C. enable.idempotency=true, retries=0, acks=0

D. enable.idempotency=false, retries>0, acks=0

Solution

1. Step 1: Identify settings for exactly-once delivery

   Exactly-once requires idempotency enabled, retries to resend on failure, and acks='all' for full commit guarantee.

2. Step 2: Evaluate each option

   enable.idempotency=true, retries>0, acks='all' has all required settings. Others disable idempotency or retries or use weak acks.

3. Final Answer:

   enable.idempotency=true, retries>0, acks='all' -> Option A

4. Quick Check:

   Idempotency + retries + acks='all' = exactly-once [OK]

Hint: Enable idempotency, retries, and acks='all' for exactly-once [OK]

Common Mistakes:

• Disabling idempotency with retries
• Using acks=0 or 1 for strong guarantees
• Setting retries=0 disables resend