Producer retries and idempotency in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output when a Kafka producer with idempotency enabled retries sending a message?

Consider a Kafka producer configured with enable.idempotence=true and retries=3. If the producer sends a message and a transient network error occurs causing a retry, what will be the effect on the topic?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");
props.put("retries", 3);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "value1");
producer.send(record);
producer.close();

A The message will be lost if a retry happens because idempotency disables retries.

B The message will be delivered exactly once to the topic, even if retries occur.

C The message may be duplicated in the topic because retries resend the message multiple times.

D The producer will throw an exception and stop sending messages on the first retry.

💡 Hint

Idempotency ensures no duplicate messages even if retries happen.

When enable.idempotence=true, Kafka producer assigns a unique sequence number to each message. If a retry occurs, the broker recognizes duplicates and prevents them from being written multiple times. This guarantees exactly-once delivery despite retries.

🧠 Conceptual

intermediate

Why is idempotency important when enabling retries in Kafka producers?

Select the best explanation for why enabling idempotency is critical when a Kafka producer is configured to retry sending messages.

A Idempotency prevents duplicate messages caused by retries, ensuring message uniqueness.

B Idempotency increases throughput by batching messages during retries.

C Idempotency disables retries to avoid message duplication.

D Idempotency encrypts messages to secure them during retries.

💡 Hint

Think about what happens if the same message is sent multiple times.

Retries can cause the same message to be sent multiple times. Idempotency ensures that even if the producer retries, the broker will write the message only once, preventing duplicates.

🔧 Debug

advanced

Identify the error in this Kafka producer configuration for retries and idempotency

Given the following Kafka producer configuration, what error will occur when sending messages with retries and idempotency enabled?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");
props.put("retries", -1);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "value1");
producer.send(record);
producer.close();

A The producer will disable idempotency automatically due to invalid retries.

B The producer will ignore the retries setting and send messages only once.

C The producer will send infinite retries without error.

D The producer will throw an IllegalArgumentException because retries cannot be negative.

💡 Hint

Check the allowed values for the retries configuration.

The retries configuration must be zero or a positive integer. Setting it to -1 is invalid and causes an exception when creating the producer.

❓ Predict Output

advanced

What happens if a Kafka producer with idempotency enabled sends messages with different keys but same value during retries?

Consider a Kafka producer with enable.idempotence=true sending two messages with the same value but different keys. If a retry occurs for one message, what will be the result in the topic?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", "true");
props.put("retries", 3);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

producer.send(new ProducerRecord<>("my-topic", "key1", "value"));
producer.send(new ProducerRecord<>("my-topic", "key2", "value"));
producer.close();

A Both messages will be delivered exactly once, even if retries happen, because keys differ.

B The message with the same value will be deduplicated and only one will appear in the topic.

C The producer will fail to send the second message due to duplicate value.

D Retries will cause duplicate messages for both keys because idempotency only works per value.

💡 Hint

Idempotency works per producer sequence, not by message content.

Idempotency prevents duplicates by tracking sequence numbers per partition and producer. Messages with different keys are treated as separate records. Even if retries occur, each message is delivered once.

🧠 Conceptual

expert

How does Kafka ensure exactly-once delivery semantics with producer retries and idempotency enabled?

Which mechanism best describes how Kafka achieves exactly-once delivery when a producer has retries and idempotency enabled?

A Kafka stores all messages in a temporary buffer and only commits them after all retries succeed.

B Kafka disables retries and uses transactional commits to guarantee exactly-once delivery.

C Kafka assigns a unique producer ID and sequence number to each message, allowing the broker to detect and discard duplicates during retries.

D Kafka encrypts messages with unique keys to prevent duplicates during retries.

💡 Hint

Think about how the broker can recognize duplicate messages from the same producer.

Kafka uses a combination of a unique producer ID and a monotonically increasing sequence number per partition. When retries happen, the broker compares sequence numbers to detect duplicates and only writes the first occurrence, ensuring exactly-once delivery.