Why delivery guarantees affect correctness in Kafka - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding At-Least-Once Delivery Impact

In Kafka, what is a key correctness concern when using at-least-once delivery guarantee?

A Messages are delivered in random order without any guarantee.

B Messages can be delivered more than once, causing duplicates.

C Messages are always delivered exactly once with no duplicates.

D Messages might be lost and never delivered to consumers.

💡 Hint

Think about what happens if a message is retried after a failure.

At-least-once delivery means Kafka ensures messages are not lost but may deliver duplicates if retries occur. This can affect correctness if the consumer is not idempotent.

❓ Predict Output

intermediate

Output with Exactly-Once Semantics Enabled

What will be the output count of messages processed if Kafka producer uses exactly-once semantics and the consumer commits offsets only after processing?

Assume 3 messages are sent and no failures occur.

Kafka

producer.send('topic', 'msg1')
producer.send('topic', 'msg2')
producer.send('topic', 'msg3')
consumer.process_and_commit_offsets()

A 3 messages processed but some duplicates may occur.

B Less than 3 messages processed due to message loss.

C More than 3 messages processed due to retries.

D 3 messages processed exactly once.

💡 Hint

Exactly-once semantics prevent duplicates and loss when configured properly.

With exactly-once semantics and proper offset commits after processing, each message is processed once and only once, so output count matches input count exactly.

❓ Predict Output

advanced

Effect of At-Most-Once Delivery on Message Loss

Given the following Kafka consumer code snippet with at-most-once delivery, what is the possible output?

consumer.poll()
consumer.commit_offsets()
process_messages()

Assume a failure happens after commit but before processing.

A Some messages may be lost and never processed.

B All messages are processed with no loss.

C Messages are processed multiple times causing duplicates.

D Consumer crashes without committing offsets.

💡 Hint

Consider what happens if offsets are committed before processing.

At-most-once delivery commits offsets before processing, so if failure occurs after commit but before processing, messages are lost because consumer thinks they are done.

🧠 Conceptual

advanced

Why Exactly-Once Semantics Are Hard to Achieve

Which of the following best explains why achieving exactly-once delivery in Kafka is challenging?

A Because Kafka does not support message ordering at all.

B Because Kafka automatically deletes messages after one delivery.

C Because network failures and retries can cause duplicates or losses without careful coordination.

D Because consumers cannot commit offsets manually.

💡 Hint

Think about what happens when failures happen during message send or processing.

Exactly-once requires coordination between producer, broker, and consumer to handle retries and failures without duplicates or losses, which is complex due to network and system failures.

🚀 Application

expert

Designing a Consumer to Handle At-Least-Once Delivery Correctly

You have a Kafka consumer configured with at-least-once delivery. Which approach below best ensures your application maintains correctness despite possible duplicate messages?

A Use idempotent processing logic or deduplication mechanisms to handle repeated messages safely.

B Commit offsets before processing messages to avoid duplicates.

C Process messages normally without any duplicate checks, relying on Kafka to avoid duplicates.

D Ignore offset commits and process messages multiple times.

💡 Hint

Think about how to handle duplicates when they happen.

At-least-once delivery can cause duplicates, so the consumer must use idempotent processing or deduplication to maintain correctness.

Practice

1. What does the at-least-once delivery guarantee in Kafka ensure?

easy

A. Messages might be lost but never duplicated.

B. Messages are delivered exactly once with no duplicates.

C. Messages are never lost but may be delivered more than once.

D. Messages are delivered only once, but some may be lost.

Solution

1. Step 1: Understand at-least-once guarantee meaning

   At-least-once means Kafka tries to deliver messages one or more times, so no message is lost but duplicates can occur.

2. Step 2: Compare options with guarantee definition

   Messages are never lost but may be delivered more than once. matches this exactly: no loss but possible duplicates. Others describe different guarantees.

3. Final Answer:

   Messages are never lost but may be delivered more than once. -> Option C

4. Quick Check:

   At-least-once = No loss, possible duplicates [OK]

Hint: At-least-once means no loss, duplicates allowed [OK]

Common Mistakes:

• Confusing at-least-once with exactly-once
• Thinking at-least-once means no duplicates
• Assuming messages can be lost with at-least-once

2. Which Kafka producer configuration enables exactly-once delivery semantics?

easy

A. enable.idempotence = true

B. acks = 0

C. retries = 0

D. enable.idempotence = false

Solution

1. Step 1: Identify config for exactly-once

   Kafka requires enable.idempotence = true to avoid duplicates and support exactly-once delivery.

2. Step 2: Check other options

   The other options disable idempotence, acknowledgments, or retries, all preventing exactly-once.

3. Final Answer:

   enable.idempotence = true -> Option A

4. Quick Check:

   Idempotence true = exactly-once enabled [OK]

Hint: Exactly-once needs idempotence enabled true [OK]

Common Mistakes:

• Setting acks to 0 disables delivery guarantees
• Disabling retries breaks message reliability
• Confusing idempotence false with true

3. Given this Kafka consumer code snippet with enable.auto.commit = false and manual commit after processing, what is the likely delivery guarantee?

medium

A. At-least-once

B. At-most-once

C. Exactly-once

D. No guarantee

Solution

1. Step 1: Analyze manual commit effect

   Manual commit after processing means messages are processed before offset commit, so duplicates can happen if failure occurs before commit.

2. Step 2: Match with delivery guarantees

   This matches at-least-once: no loss but possible duplicates. Exactly-once requires more setup; at-most-once commits before processing.

3. Final Answer:

   At-least-once -> Option A

4. Quick Check:

   Manual commit after processing = at-least-once [OK]

Hint: Manual commit after processing means at-least-once [OK]

Common Mistakes:

• Assuming manual commit means exactly-once
• Confusing commit timing with at-most-once
• Ignoring possibility of duplicates

4. You notice duplicate messages in your Kafka consumer. Which fix improves correctness by avoiding duplicates?

medium

A. Set acks = 0 on the producer

B. Set enable.idempotence = true on the producer

C. Disable retries on the producer

D. Use enable.auto.commit = true on the consumer

Solution

1. Step 1: Identify cause of duplicates

   Duplicates often happen when producer retries send messages multiple times without idempotence.

2. Step 2: Choose fix to avoid duplicates

   Enabling idempotence ensures retries don't create duplicates. Other options disable guarantees or auto commit which can worsen duplicates.

3. Final Answer:

   Set enable.idempotence = true on the producer -> Option B

4. Quick Check:

   Idempotence true prevents duplicates [OK]

Hint: Enable idempotence to avoid duplicates [OK]

Common Mistakes:

• Setting acks=0 disables delivery guarantees
• Disabling retries causes message loss
• Relying on auto commit to fix duplicates

5. You build a financial app needing no lost or duplicated transactions. Which Kafka setup best ensures correctness?

hard

A. Producer with acks=1 and consumer with auto commit disabled

B. Producer with acks=0 and consumer with auto commit enabled

C. Producer with enable.idempotence=false and consumer with manual commit before processing

D. Producer with enable.idempotence=true, acks=all, and consumer with manual commit after processing

Solution

1. Step 1: Identify needed guarantees for financial app

   Financial apps require exactly-once semantics: no lost or duplicated messages.

2. Step 2: Match Kafka configs to exactly-once

   Enable idempotence and acks=all on producer ensures no duplicates and full replication. Manual commit after processing on consumer avoids message loss.

3. Step 3: Evaluate other options

   Other options either disable guarantees or commit offsets too early, risking loss or duplicates.

4. Final Answer:

   Producer with enable.idempotence=true, acks=all, and consumer with manual commit after processing -> Option D

5. Quick Check:

   Idempotence + acks=all + manual commit = exactly-once [OK]

Hint: Exactly-once needs idempotence, acks=all, manual commit [OK]

Common Mistakes:

• Using acks=0 or 1 risks message loss
• Committing offsets before processing causes loss
• Disabling idempotence allows duplicates