Why delivery guarantees affect correctness in Kafka - Why It Works

Introduction

When sending messages between systems, how often and when messages arrive can change how your app works. Delivery guarantees in Kafka help control this, making sure messages are not lost or repeated, which keeps your data correct.

When you want to make sure no messages are lost in a payment processing system

When you need to avoid processing the same message twice in an order tracking app

When you want to balance speed and reliability in a real-time analytics pipeline

When you must guarantee all messages are processed even if a server crashes

When you want to handle occasional duplicates but keep the system fast

Commands

This command sends messages to Kafka without waiting for any confirmation, which is called 'at most once' delivery. It is fast but can lose messages if the broker fails.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer-property acks=0

Expected Output

No output (command runs silently)

→

--producer-property acks=0 - No acknowledgment from broker, fastest but least reliable

This sends messages and waits for the leader broker to confirm. This is 'at least once' delivery, which avoids losing messages but can cause duplicates if retries happen.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer-property acks=1

Expected Output

No output (command runs silently)

→

--producer-property acks=1 - Wait for leader acknowledgment, balances speed and reliability

This waits for all in-sync replicas to confirm the message, called 'exactly once' delivery in ideal cases. It is the safest but slowest option, ensuring no loss or duplicates.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer-property acks=all

Expected Output

No output (command runs silently)

→

--producer-property acks=all - Wait for all replicas to acknowledge, highest reliability

This reads the first 5 messages from the topic to check what was delivered and if duplicates or losses happened.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 5

Expected Output

message1

message2

message3

message4

message5

→

--from-beginning - Read messages from the start of the topic

→

--max-messages 5 - Stop after reading 5 messages

Key Concept

If you remember nothing else, remember: delivery guarantees control message loss and duplication, directly affecting your system's correctness.

Common Mistakes

Using acks=0 when message loss is unacceptable

Messages can be lost without any warning, causing data errors

Use acks=1 or acks=all to ensure messages are confirmed by brokers

Assuming acks=1 means no duplicates

Retries can cause the same message to be sent twice, leading to duplicates

Implement idempotent consumers or use acks=all with idempotent producers

Ignoring consumer offsets and reprocessing messages blindly

This can cause duplicate processing even if delivery is guaranteed

Manage offsets carefully and use exactly-once processing features if needed

Summary

Delivery guarantees in Kafka control how messages are confirmed and affect message loss or duplication.

Using acks=0 is fastest but can lose messages; acks=all is safest but slower.

Consumers must handle duplicates and offsets to maintain correctness.

Practice

1. What does the at-least-once delivery guarantee in Kafka ensure?

easy

A. Messages might be lost but never duplicated.

B. Messages are delivered exactly once with no duplicates.

C. Messages are never lost but may be delivered more than once.

D. Messages are delivered only once, but some may be lost.

Solution

1. Step 1: Understand at-least-once guarantee meaning

   At-least-once means Kafka tries to deliver messages one or more times, so no message is lost but duplicates can occur.

2. Step 2: Compare options with guarantee definition

   Messages are never lost but may be delivered more than once. matches this exactly: no loss but possible duplicates. Others describe different guarantees.

3. Final Answer:

   Messages are never lost but may be delivered more than once. -> Option C

4. Quick Check:

   At-least-once = No loss, possible duplicates [OK]

Hint: At-least-once means no loss, duplicates allowed [OK]

Common Mistakes:

• Confusing at-least-once with exactly-once
• Thinking at-least-once means no duplicates
• Assuming messages can be lost with at-least-once

2. Which Kafka producer configuration enables exactly-once delivery semantics?

easy

A. enable.idempotence = true

B. acks = 0

C. retries = 0

D. enable.idempotence = false

Solution

1. Step 1: Identify config for exactly-once

   Kafka requires enable.idempotence = true to avoid duplicates and support exactly-once delivery.

2. Step 2: Check other options

   The other options disable idempotence, acknowledgments, or retries, all preventing exactly-once.

3. Final Answer:

   enable.idempotence = true -> Option A

4. Quick Check:

   Idempotence true = exactly-once enabled [OK]

Hint: Exactly-once needs idempotence enabled true [OK]

Common Mistakes:

• Setting acks to 0 disables delivery guarantees
• Disabling retries breaks message reliability
• Confusing idempotence false with true

3. Given this Kafka consumer code snippet with enable.auto.commit = false and manual commit after processing, what is the likely delivery guarantee?

medium

A. At-least-once

B. At-most-once

C. Exactly-once

D. No guarantee

Solution

1. Step 1: Analyze manual commit effect

   Manual commit after processing means messages are processed before offset commit, so duplicates can happen if failure occurs before commit.

2. Step 2: Match with delivery guarantees

   This matches at-least-once: no loss but possible duplicates. Exactly-once requires more setup; at-most-once commits before processing.

3. Final Answer:

   At-least-once -> Option A

4. Quick Check:

   Manual commit after processing = at-least-once [OK]

Hint: Manual commit after processing means at-least-once [OK]

Common Mistakes:

• Assuming manual commit means exactly-once
• Confusing commit timing with at-most-once
• Ignoring possibility of duplicates

4. You notice duplicate messages in your Kafka consumer. Which fix improves correctness by avoiding duplicates?

medium

A. Set acks = 0 on the producer

B. Set enable.idempotence = true on the producer

C. Disable retries on the producer

D. Use enable.auto.commit = true on the consumer

Solution

1. Step 1: Identify cause of duplicates

   Duplicates often happen when producer retries send messages multiple times without idempotence.

2. Step 2: Choose fix to avoid duplicates

   Enabling idempotence ensures retries don't create duplicates. Other options disable guarantees or auto commit which can worsen duplicates.

3. Final Answer:

   Set enable.idempotence = true on the producer -> Option B

4. Quick Check:

   Idempotence true prevents duplicates [OK]

Hint: Enable idempotence to avoid duplicates [OK]

Common Mistakes:

• Setting acks=0 disables delivery guarantees
• Disabling retries causes message loss
• Relying on auto commit to fix duplicates

5. You build a financial app needing no lost or duplicated transactions. Which Kafka setup best ensures correctness?

hard

A. Producer with acks=1 and consumer with auto commit disabled

B. Producer with acks=0 and consumer with auto commit enabled

C. Producer with enable.idempotence=false and consumer with manual commit before processing

D. Producer with enable.idempotence=true, acks=all, and consumer with manual commit after processing

Solution

1. Step 1: Identify needed guarantees for financial app

   Financial apps require exactly-once semantics: no lost or duplicated messages.

2. Step 2: Match Kafka configs to exactly-once

   Enable idempotence and acks=all on producer ensures no duplicates and full replication. Manual commit after processing on consumer avoids message loss.

3. Step 3: Evaluate other options

   Other options either disable guarantees or commit offsets too early, risking loss or duplicates.

4. Final Answer:

   Producer with enable.idempotence=true, acks=all, and consumer with manual commit after processing -> Option D

5. Quick Check:

   Idempotence + acks=all + manual commit = exactly-once [OK]

Hint: Exactly-once needs idempotence, acks=all, manual commit [OK]

Common Mistakes:

• Using acks=0 or 1 risks message loss
• Committing offsets before processing causes loss
• Disabling idempotence allows duplicates