Why delivery guarantees affect correctness in Kafka - Why It Works

Introduction

When sending messages between systems, how often and when messages arrive can change how your app works. Delivery guarantees in Kafka help control this, making sure messages are not lost or repeated, which keeps your data correct.

When you want to make sure no messages are lost in a payment processing system

When you need to avoid processing the same message twice in an order tracking app

When you want to balance speed and reliability in a real-time analytics pipeline

When you must guarantee all messages are processed even if a server crashes

When you want to handle occasional duplicates but keep the system fast

Commands

This command sends messages to Kafka without waiting for any confirmation, which is called 'at most once' delivery. It is fast but can lose messages if the broker fails.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer-property acks=0

Expected Output

No output (command runs silently)

→

--producer-property acks=0 - No acknowledgment from broker, fastest but least reliable

This sends messages and waits for the leader broker to confirm. This is 'at least once' delivery, which avoids losing messages but can cause duplicates if retries happen.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer-property acks=1

Expected Output

No output (command runs silently)

→

--producer-property acks=1 - Wait for leader acknowledgment, balances speed and reliability

This waits for all in-sync replicas to confirm the message, called 'exactly once' delivery in ideal cases. It is the safest but slowest option, ensuring no loss or duplicates.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer-property acks=all

Expected Output

No output (command runs silently)

→

--producer-property acks=all - Wait for all replicas to acknowledge, highest reliability

This reads the first 5 messages from the topic to check what was delivered and if duplicates or losses happened.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --max-messages 5

Expected Output

message1

message2

message3

message4

message5

→

--from-beginning - Read messages from the start of the topic

→

--max-messages 5 - Stop after reading 5 messages

Key Concept

If you remember nothing else, remember: delivery guarantees control message loss and duplication, directly affecting your system's correctness.

Common Mistakes

Using acks=0 when message loss is unacceptable

Messages can be lost without any warning, causing data errors

Use acks=1 or acks=all to ensure messages are confirmed by brokers

Assuming acks=1 means no duplicates

Retries can cause the same message to be sent twice, leading to duplicates

Implement idempotent consumers or use acks=all with idempotent producers

Ignoring consumer offsets and reprocessing messages blindly

This can cause duplicate processing even if delivery is guaranteed

Manage offsets carefully and use exactly-once processing features if needed

Summary

Delivery guarantees in Kafka control how messages are confirmed and affect message loss or duplication.

Using acks=0 is fastest but can lose messages; acks=all is safest but slower.

Consumers must handle duplicates and offsets to maintain correctness.