Exactly-once semantics (EOS) in Kafka - Commands & Configuration

Introduction

When sending messages in Kafka, sometimes duplicates or lost messages happen. Exactly-once semantics (EOS) ensures each message is processed once and only once, avoiding duplicates or missing data.

When you want to process financial transactions without any duplicates or losses.

When updating inventory counts in a warehouse system where accuracy is critical.

When aggregating sensor data where each reading must be counted exactly once.

When performing data replication between systems and you want to avoid duplicate records.

When running batch jobs that consume Kafka messages and update databases without errors.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
acks=all
enable.idempotence=true
transactional.id=my-transactional-producer
max.in.flight.requests.per.connection=5
retries=5

This configuration file sets up a Kafka producer for exactly-once semantics.

• bootstrap.servers: Kafka server address.
• acks=all: Waits for all replicas to acknowledge messages for durability.
• enable.idempotence=true: Ensures no duplicate messages are sent.
• transactional.id: Enables transactions for atomic writes.
• max.in.flight.requests.per.connection=5: Allows multiple requests but keeps ordering.
• retries=5: Retries sending messages on failure.

Commands

Create a Kafka topic named 'my-topic' with 3 partitions and replication factor 1 to hold messages.

Terminal

kafka-topics --create --topic my-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic my-topic.

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Start a Kafka producer using the configuration file with EOS enabled to send messages safely.

Terminal

kafka-console-producer --topic my-topic --bootstrap-server localhost:9092 --producer.config producer.properties

Expected Output

No output (command runs silently)

→

--producer.config - Use the producer properties file with EOS settings

Consume the first 5 messages from 'my-topic' to verify messages were sent exactly once.

Terminal

kafka-console-consumer --topic my-topic --bootstrap-server localhost:9092 --from-beginning --max-messages 5

Expected Output

message1

message2

message3

message4

message5

→

--from-beginning - Read messages from the start of the topic

→

--max-messages - Limit number of messages to consume

Key Concept

If you remember nothing else from exactly-once semantics, remember: enabling idempotence and transactions in the Kafka producer ensures each message is processed once and only once.

Common Mistakes

Not setting enable.idempotence=true in the producer config

This causes duplicate messages when retries happen, breaking exactly-once guarantees.

Always set enable.idempotence=true to prevent duplicates.

Omitting transactional.id in the producer config when using transactions

Without transactional.id, the producer cannot start transactions, so atomic writes fail.

Set a unique transactional.id to enable transactions for exactly-once writes.

Using acks=1 or acks=0 instead of acks=all

This risks message loss or duplicates because not all replicas confirm the write.

Use acks=all to ensure full replication before acknowledging.

Summary

Create a Kafka topic to hold messages.

Configure the producer with enable.idempotence and transactional.id for exactly-once semantics.

Send messages using the configured producer to avoid duplicates.

Consume messages to verify each message appears exactly once.

Practice

1. What is the main goal of Exactly-once semantics (EOS) in Kafka?

easy

A. To compress messages for storage efficiency

B. To prevent duplicate or lost messages during data processing

C. To allow multiple consumers to read the same message simultaneously

D. To increase the speed of message delivery

Solution

1. Step 1: Understand the purpose of EOS

   Exactly-once semantics ensures that each message is processed exactly one time, avoiding duplicates or losses.

2. Step 2: Compare options with EOS goal

   Options A, B, and C describe other Kafka features but not EOS's main goal.

3. Final Answer:

   To prevent duplicate or lost messages during data processing -> Option B

4. Quick Check:

   EOS = prevent duplicates/losses [OK]

Hint: EOS means no duplicates or lost messages [OK]

Common Mistakes:

• Confusing EOS with message speed optimization
• Thinking EOS allows multiple consumers to read simultaneously
• Mixing EOS with message compression features

2. Which configuration must be enabled in a Kafka producer to support Exactly-once semantics?

easy

A. max.poll.records = 500

B. acks = 0

C. compression.type = gzip

D. enable.idempotence = true

Solution

1. Step 1: Identify EOS required producer setting

   Enabling idempotence (enable.idempotence = true) is necessary for EOS to avoid duplicates.

2. Step 2: Review other options

   Options B, C, and D relate to acknowledgments, compression, and consumer polling, not EOS.

3. Final Answer:

   enable.idempotence = true -> Option D

4. Quick Check:

   Idempotence enabled = EOS support [OK]

Hint: Enable idempotence for EOS in producer config [OK]

Common Mistakes:

• Setting acks to 0 disables guarantees
• Confusing compression with EOS
• Using consumer settings instead of producer

3. Given the following Kafka producer code snippet with transactions enabled, what will happen if producer.commitTransaction() is not called after producer.send()?

producer.initTransactions();
producer.beginTransaction();
producer.send(record);
// Missing commitTransaction call
producer.close();

medium

A. The transaction will be aborted and messages will not be committed

B. Messages will be committed automatically on close

C. Messages will be duplicated in the topic

D. Producer will throw a syntax error

Solution

1. Step 1: Understand transaction lifecycle in Kafka

   After beginning a transaction and sending messages, commitTransaction() must be called to finalize the transaction.

2. Step 2: Effect of missing commitTransaction()

   If commitTransaction() is not called, the transaction is aborted on close, so messages are not committed.

3. Final Answer:

   The transaction will be aborted and messages will not be committed -> Option A

4. Quick Check:

   Missing commitTransaction = abort transaction [OK]

Hint: Always commit or abort transactions explicitly [OK]

Common Mistakes:

• Assuming close commits automatically
• Expecting duplicates without commit
• Thinking missing commit causes syntax error

4. You have enabled EOS in your Kafka producer but notice duplicate messages in the topic. Which of the following is the most likely cause?

medium

A. Producer did not call initTransactions() before starting transactions

B. Producer set enable.idempotence to true

C. Consumer group is reading messages multiple times

D. Topic has replication factor set to 3

Solution

1. Step 1: Check EOS setup requirements

   initTransactions() must be called before using transactions to ensure EOS works properly.

2. Step 2: Analyze other options

   Producer set enable.idempotence to true is correct config, A relates to consumers not producers, D is unrelated to duplicates.

3. Final Answer:

   Producer did not call initTransactions() before starting transactions -> Option A

4. Quick Check:

   Missing initTransactions() causes duplicates [OK]

Hint: Call initTransactions() before transactions for EOS [OK]

Common Mistakes:

• Confusing consumer behavior with producer duplicates
• Assuming replication factor affects duplicates
• Thinking idempotence alone fixes all issues

5. You want to implement a Kafka producer that writes messages exactly once to multiple partitions atomically. Which combination of settings and steps is required to achieve this?

hard

A. Disable idempotence, send messages asynchronously, then flush producer

B. Set acks=0, send messages to partitions, then close producer

C. Enable idempotence, initialize and begin transactions, send messages, then commit transaction

D. Use synchronous sends without transactions and rely on consumer deduplication

Solution

1. Step 1: Identify EOS requirements for atomic writes

   Enable idempotence and use transactions to ensure atomic writes across partitions.

2. Step 2: Follow correct transaction steps

   Initialize transactions, begin transaction, send messages, then commit transaction to finalize.

3. Step 3: Evaluate other options

   Options B, C, and D lack idempotence or transactions, so they cannot guarantee exactly-once atomic writes.

4. Final Answer:

   Enable idempotence, initialize and begin transactions, send messages, then commit transaction -> Option C

5. Quick Check:

   Idempotence + transactions = atomic exactly-once writes [OK]

Hint: Use idempotence + transactions for atomic exactly-once writes [OK]

Common Mistakes:

• Ignoring transaction lifecycle steps
• Relying on consumer deduplication only
• Disabling idempotence for performance