Why delivery guarantees affect correctness in Kafka - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding At-Least-Once Delivery Impact

In Kafka, what is a key correctness concern when using at-least-once delivery guarantee?

A Messages are delivered in random order without any guarantee.

B Messages can be delivered more than once, causing duplicates.

C Messages are always delivered exactly once with no duplicates.

D Messages might be lost and never delivered to consumers.

💡 Hint

Think about what happens if a message is retried after a failure.

At-least-once delivery means Kafka ensures messages are not lost but may deliver duplicates if retries occur. This can affect correctness if the consumer is not idempotent.

❓ Predict Output

intermediate

Output with Exactly-Once Semantics Enabled

What will be the output count of messages processed if Kafka producer uses exactly-once semantics and the consumer commits offsets only after processing?

Assume 3 messages are sent and no failures occur.

Kafka

producer.send('topic', 'msg1')
producer.send('topic', 'msg2')
producer.send('topic', 'msg3')
consumer.process_and_commit_offsets()

A 3 messages processed but some duplicates may occur.

B Less than 3 messages processed due to message loss.

C More than 3 messages processed due to retries.

D 3 messages processed exactly once.

💡 Hint

Exactly-once semantics prevent duplicates and loss when configured properly.

With exactly-once semantics and proper offset commits after processing, each message is processed once and only once, so output count matches input count exactly.

❓ Predict Output

advanced

Effect of At-Most-Once Delivery on Message Loss

Given the following Kafka consumer code snippet with at-most-once delivery, what is the possible output?

consumer.poll()
consumer.commit_offsets()
process_messages()

Assume a failure happens after commit but before processing.

A Some messages may be lost and never processed.

B All messages are processed with no loss.

C Messages are processed multiple times causing duplicates.

D Consumer crashes without committing offsets.

💡 Hint

Consider what happens if offsets are committed before processing.

At-most-once delivery commits offsets before processing, so if failure occurs after commit but before processing, messages are lost because consumer thinks they are done.

🧠 Conceptual

advanced

Why Exactly-Once Semantics Are Hard to Achieve

Which of the following best explains why achieving exactly-once delivery in Kafka is challenging?

A Because Kafka does not support message ordering at all.

B Because Kafka automatically deletes messages after one delivery.

C Because network failures and retries can cause duplicates or losses without careful coordination.

D Because consumers cannot commit offsets manually.

💡 Hint

Think about what happens when failures happen during message send or processing.

Exactly-once requires coordination between producer, broker, and consumer to handle retries and failures without duplicates or losses, which is complex due to network and system failures.

🚀 Application

expert

Designing a Consumer to Handle At-Least-Once Delivery Correctly

You have a Kafka consumer configured with at-least-once delivery. Which approach below best ensures your application maintains correctness despite possible duplicate messages?

A Use idempotent processing logic or deduplication mechanisms to handle repeated messages safely.

B Commit offsets before processing messages to avoid duplicates.

C Process messages normally without any duplicate checks, relying on Kafka to avoid duplicates.

D Ignore offset commits and process messages multiple times.

💡 Hint

Think about how to handle duplicates when they happen.

At-least-once delivery can cause duplicates, so the consumer must use idempotent processing or deduplication to maintain correctness.