Why delivery guarantees affect correctness in Kafka - Visual Breakdown

Process Flow - Why delivery guarantees affect correctness

Message Produced

↓

Delivery Guarantee Check

↓

At Most Once

↓

Possible Loss

↓

Incorrect

↓

Message Consumed

↓

Application Correctness Depends on Guarantee

The flow shows how different delivery guarantees lead to different message outcomes, affecting correctness.

Execution Sample

Kafka

produce(message)
if guarantee == 'at_most_once':
    send_once()
elif guarantee == 'at_least_once':
    retry_until_ack()
else:
    transaction_commit()

This code sends a message with different delivery guarantees affecting retries and transactions.

Process Table

Step	Delivery Guarantee	Action	Message Outcome	Correctness Impact
1	at_most_once	Send message once, no retry	Message may be lost	Possible incorrect processing due to loss
2	at_least_once	Retry sending until ack received	Message may be duplicated	Possible incorrect processing due to duplicates
3	exactly_once	Send message within transaction	Message delivered once exactly	Correct processing assured
4	N/A	Message consumed by application	Outcome depends on guarantee	Correctness depends on guarantee

💡 Execution stops after message consumption; correctness depends on delivery guarantee chosen.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
message_status	not_sent	sent_once_or_lost	sent_with_possible_duplicates	sent_once_transactional	final_state_based_on_guarantee
retry_attempts	0	0	>=1	0	depends_on_guarantee

Key Moments - 3 Insights

Why can 'at_most_once' cause message loss?

Because it sends the message only once without retry, if the message is lost in transit, it is never resent (see execution_table row 1).

Why does 'at_least_once' risk duplicates?

Because it retries sending until acknowledged, duplicates can occur if the first send succeeded but ack was lost, causing resend (see execution_table row 2).

How does 'exactly_once' guarantee correctness?

It uses transactions to ensure the message is delivered once and only once, preventing loss and duplicates (see execution_table row 3).

Visual Quiz - 3 Questions

Look at the execution_table, what is the message outcome at step 2 with 'at_least_once' guarantee?

A Message may be duplicated

B Message delivered once exactly

C Message may be lost

D Message not sent

💡 Hint

Check execution_table row 2 under 'Message Outcome'

According to variable_tracker, how many retry attempts happen after step 2?

A 0

B Exactly 1

C At least 1

D Depends on transaction

💡 Hint

Look at 'retry_attempts' variable after step 2 in variable_tracker

If we change from 'at_most_once' to 'exactly_once', what changes in message_status after step 3?

A From 'not_sent' to 'sent_with_possible_duplicates'

B From 'sent_once_or_lost' to 'sent_once_transactional'

C No change

D From 'sent_with_possible_duplicates' to 'sent_once_or_lost'

💡 Hint

Compare 'message_status' values in variable_tracker after steps 1 and 3

Concept Snapshot

Delivery guarantees in Kafka affect message correctness:
- at_most_once: send once, risk loss
- at_least_once: retry, risk duplicates
- exactly_once: transactional, no loss or duplicates
Choose guarantee based on correctness needs.

Full Transcript

This visual execution shows how Kafka delivery guarantees impact message correctness. The code example sends messages differently based on guarantee. The execution table traces steps for at_most_once, at_least_once, and exactly_once guarantees, showing message outcomes and correctness impact. Variable tracker follows message status and retry attempts. Key moments clarify why loss or duplicates happen. The quiz tests understanding of message outcomes and variable changes. The snapshot summarizes guarantees and their effects on correctness.

Practice

1. What does the at-least-once delivery guarantee in Kafka ensure?

easy

A. Messages might be lost but never duplicated.

B. Messages are delivered exactly once with no duplicates.

C. Messages are never lost but may be delivered more than once.

D. Messages are delivered only once, but some may be lost.

Solution

1. Step 1: Understand at-least-once guarantee meaning

   At-least-once means Kafka tries to deliver messages one or more times, so no message is lost but duplicates can occur.

2. Step 2: Compare options with guarantee definition

   Messages are never lost but may be delivered more than once. matches this exactly: no loss but possible duplicates. Others describe different guarantees.

3. Final Answer:

   Messages are never lost but may be delivered more than once. -> Option C

4. Quick Check:

   At-least-once = No loss, possible duplicates [OK]

Hint: At-least-once means no loss, duplicates allowed [OK]

Common Mistakes:

• Confusing at-least-once with exactly-once
• Thinking at-least-once means no duplicates
• Assuming messages can be lost with at-least-once

2. Which Kafka producer configuration enables exactly-once delivery semantics?

easy

A. enable.idempotence = true

B. acks = 0

C. retries = 0

D. enable.idempotence = false

Solution

1. Step 1: Identify config for exactly-once

   Kafka requires enable.idempotence = true to avoid duplicates and support exactly-once delivery.

2. Step 2: Check other options

   The other options disable idempotence, acknowledgments, or retries, all preventing exactly-once.

3. Final Answer:

   enable.idempotence = true -> Option A

4. Quick Check:

   Idempotence true = exactly-once enabled [OK]

Hint: Exactly-once needs idempotence enabled true [OK]

Common Mistakes:

• Setting acks to 0 disables delivery guarantees
• Disabling retries breaks message reliability
• Confusing idempotence false with true

3. Given this Kafka consumer code snippet with enable.auto.commit = false and manual commit after processing, what is the likely delivery guarantee?

medium

A. At-least-once

B. At-most-once

C. Exactly-once

D. No guarantee

Solution

1. Step 1: Analyze manual commit effect

   Manual commit after processing means messages are processed before offset commit, so duplicates can happen if failure occurs before commit.

2. Step 2: Match with delivery guarantees

   This matches at-least-once: no loss but possible duplicates. Exactly-once requires more setup; at-most-once commits before processing.

3. Final Answer:

   At-least-once -> Option A

4. Quick Check:

   Manual commit after processing = at-least-once [OK]

Hint: Manual commit after processing means at-least-once [OK]

Common Mistakes:

• Assuming manual commit means exactly-once
• Confusing commit timing with at-most-once
• Ignoring possibility of duplicates

4. You notice duplicate messages in your Kafka consumer. Which fix improves correctness by avoiding duplicates?

medium

A. Set acks = 0 on the producer

B. Set enable.idempotence = true on the producer

C. Disable retries on the producer

D. Use enable.auto.commit = true on the consumer

Solution

1. Step 1: Identify cause of duplicates

   Duplicates often happen when producer retries send messages multiple times without idempotence.

2. Step 2: Choose fix to avoid duplicates

   Enabling idempotence ensures retries don't create duplicates. Other options disable guarantees or auto commit which can worsen duplicates.

3. Final Answer:

   Set enable.idempotence = true on the producer -> Option B

4. Quick Check:

   Idempotence true prevents duplicates [OK]

Hint: Enable idempotence to avoid duplicates [OK]

Common Mistakes:

• Setting acks=0 disables delivery guarantees
• Disabling retries causes message loss
• Relying on auto commit to fix duplicates

5. You build a financial app needing no lost or duplicated transactions. Which Kafka setup best ensures correctness?

hard

A. Producer with acks=1 and consumer with auto commit disabled

B. Producer with acks=0 and consumer with auto commit enabled

C. Producer with enable.idempotence=false and consumer with manual commit before processing

D. Producer with enable.idempotence=true, acks=all, and consumer with manual commit after processing

Solution

1. Step 1: Identify needed guarantees for financial app

   Financial apps require exactly-once semantics: no lost or duplicated messages.

2. Step 2: Match Kafka configs to exactly-once

   Enable idempotence and acks=all on producer ensures no duplicates and full replication. Manual commit after processing on consumer avoids message loss.

3. Step 3: Evaluate other options

   Other options either disable guarantees or commit offsets too early, risking loss or duplicates.

4. Final Answer:

   Producer with enable.idempotence=true, acks=all, and consumer with manual commit after processing -> Option D

5. Quick Check:

   Idempotence + acks=all + manual commit = exactly-once [OK]

Hint: Exactly-once needs idempotence, acks=all, manual commit [OK]

Common Mistakes:

• Using acks=0 or 1 risks message loss
• Committing offsets before processing causes loss
• Disabling idempotence allows duplicates