At-most-once delivery in Kafka - Commands & Configuration

Introduction

At-most-once delivery means messages are delivered zero or one time to consumers. This avoids duplicates but may lose some messages if failures happen.

When you want to avoid processing the same message twice even if some messages get lost

When occasional message loss is acceptable but duplicates cause big problems

When your system can tolerate missing data but cannot tolerate repeated data

When you want the simplest message delivery with minimal overhead

When you do not need guaranteed message processing but want fast throughput

Commands

This command starts a producer to send messages to the topic named example-topic on the Kafka broker running locally.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker address to connect to

→

--topic - Specifies the topic to send messages to

This command starts a consumer that reads messages from example-topic from the beginning with auto-commit disabled, simulating at-most-once by disabling commits.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --enable-auto-commit false

Expected Output

First message

Second message

Third message

→

--bootstrap-server - Specifies the Kafka broker address to connect to

→

--topic - Specifies the topic to consume messages from

→

--enable-auto-commit - Disables automatic offset commits to avoid duplicate processing

This command shows the current offset positions for the consumer group example-group to verify message consumption state.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --group example-group --describe

Expected Output

TOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

example-topic  0          3               3               0    consumer-1   /127.0.0.1  consumer-1

→

--group - Specifies the consumer group to describe

Key Concept

If you remember nothing else from this pattern, remember: at-most-once delivery means messages may be lost but never duplicated.

Common Mistakes

Enabling auto-commit in the consumer when trying to achieve at-most-once delivery

Auto-commit can cause offsets to be committed before processing, leading to message loss or duplicates if the consumer crashes

Disable auto-commit and manage offsets carefully to ensure messages are processed zero or one time

Not understanding that at-most-once can lose messages

Assuming all messages will be delivered can cause data loss in critical systems

Use at-most-once only when occasional message loss is acceptable

Summary

Use kafka-console-producer to send messages to a topic.

Use kafka-console-consumer with auto-commit disabled to consume messages at-most-once.

Check consumer group offsets with kafka-consumer-groups to monitor consumption.