Exactly-once semantics (EOS) in Kafka - Commands & Configuration

Introduction

When sending messages in Kafka, sometimes duplicates or lost messages happen. Exactly-once semantics (EOS) ensures each message is processed once and only once, avoiding duplicates or missing data.

When you want to process financial transactions without any duplicates or losses.

When updating inventory counts in a warehouse system where accuracy is critical.

When aggregating sensor data where each reading must be counted exactly once.

When performing data replication between systems and you want to avoid duplicate records.

When running batch jobs that consume Kafka messages and update databases without errors.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
acks=all
enable.idempotence=true
transactional.id=my-transactional-producer
max.in.flight.requests.per.connection=5
retries=5

This configuration file sets up a Kafka producer for exactly-once semantics.

• bootstrap.servers: Kafka server address.
• acks=all: Waits for all replicas to acknowledge messages for durability.
• enable.idempotence=true: Ensures no duplicate messages are sent.
• transactional.id: Enables transactions for atomic writes.
• max.in.flight.requests.per.connection=5: Allows multiple requests but keeps ordering.
• retries=5: Retries sending messages on failure.

Commands

Create a Kafka topic named 'my-topic' with 3 partitions and replication factor 1 to hold messages.

Terminal

kafka-topics --create --topic my-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic my-topic.

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Start a Kafka producer using the configuration file with EOS enabled to send messages safely.

Terminal

kafka-console-producer --topic my-topic --bootstrap-server localhost:9092 --producer.config producer.properties

Expected Output

No output (command runs silently)

→

--producer.config - Use the producer properties file with EOS settings

Consume the first 5 messages from 'my-topic' to verify messages were sent exactly once.

Terminal

kafka-console-consumer --topic my-topic --bootstrap-server localhost:9092 --from-beginning --max-messages 5

Expected Output

message1

message2

message3

message4

message5

→

--from-beginning - Read messages from the start of the topic

→

--max-messages - Limit number of messages to consume

Key Concept

If you remember nothing else from exactly-once semantics, remember: enabling idempotence and transactions in the Kafka producer ensures each message is processed once and only once.

Common Mistakes

Not setting enable.idempotence=true in the producer config

This causes duplicate messages when retries happen, breaking exactly-once guarantees.

Always set enable.idempotence=true to prevent duplicates.

Omitting transactional.id in the producer config when using transactions

Without transactional.id, the producer cannot start transactions, so atomic writes fail.

Set a unique transactional.id to enable transactions for exactly-once writes.

Using acks=1 or acks=0 instead of acks=all

This risks message loss or duplicates because not all replicas confirm the write.

Use acks=all to ensure full replication before acknowledging.

Summary

Create a Kafka topic to hold messages.

Configure the producer with enable.idempotence and transactional.id for exactly-once semantics.

Send messages using the configured producer to avoid duplicates.

Consume messages to verify each message appears exactly once.