At-least-once delivery in Kafka - Commands & Configuration

Introduction

At-least-once delivery ensures that every message sent to Kafka is processed at least one time. This means messages might be delivered more than once but never lost, which helps avoid missing important data.

When you want to make sure no messages are lost even if the consumer crashes.

When processing financial transactions where missing a message is worse than processing duplicates.

When collecting logs or metrics where completeness is more important than avoiding duplicates.

When your system can handle duplicate messages safely or has a way to detect duplicates.

When you want reliable message delivery but can accept occasional repeated processing.

Config File - producer.properties

producer.properties

acks=all
retries=3
enable.idempotence=false

This configuration file sets the Kafka producer to wait for all replicas to acknowledge a message (acks=all) to ensure durability.

Retries=3 allows the producer to resend messages if sending fails temporarily.

Idempotence is disabled here to allow at-least-once delivery, which may cause duplicates.

Commands

This command starts a Kafka producer that sends messages to 'example-topic' using the configuration file to ensure at-least-once delivery.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer.config producer.properties

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the topic to send messages to.

→

--producer.config - Loads producer settings from the given configuration file.

This command starts a Kafka consumer that reads all messages from the beginning of 'example-topic' to verify message delivery.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning

Expected Output

message1

message2

message3

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the topic to read messages from.

→

--from-beginning - Reads all messages from the start of the topic.

Key Concept

If you remember nothing else from this pattern, remember: at-least-once delivery guarantees no message loss but may cause duplicates.

Common Mistakes

Setting 'acks=0' in producer configuration.

This causes the producer to not wait for any acknowledgment, risking message loss.

Set 'acks=all' to ensure the message is fully committed before continuing.

Not enabling retries in the producer.

Without retries, temporary network issues can cause message loss.

Set 'retries' to a positive number like 3 to resend failed messages.

Expecting exactly-once delivery without enabling idempotence.

Without idempotence, duplicates can occur, so exactly-once is not guaranteed.

Accept duplicates or enable idempotence for exactly-once semantics.

Summary

Configure the Kafka producer with 'acks=all' and retries to ensure messages are not lost.

Use the console producer to send messages to a topic with this configuration.

Use the console consumer to read messages from the topic and verify delivery.

At-least-once delivery means messages may be duplicated but never lost.