Exactly-once stream processing in Kafka - Commands & Configuration

Introduction

When processing data streams, sometimes messages can be processed more than once due to failures or retries. Exactly-once stream processing ensures each message affects the system only once, avoiding duplicates and data errors.

When you need to update a database from a stream without creating duplicate records.

When processing financial transactions where double processing causes incorrect balances.

When aggregating metrics from event streams and accuracy is critical.

When retrying failed processing but want to avoid counting the same event multiple times.

When building real-time dashboards that must reflect precise data without duplication.

Config File - kafka-streams.properties

kafka-streams.properties

application.id=exactly-once-app
bootstrap.servers=localhost:9092
processing.guarantee=exactly_once_v2
cache.max.bytes.buffering=0
commit.interval.ms=100
acks=all

application.id: Identifies the Kafka Streams application uniquely.

bootstrap.servers: Kafka broker addresses to connect to.

processing.guarantee: Enables exactly-once processing with the latest Kafka version (v2).

cache.max.bytes.buffering: Set to 0 to disable caching for immediate processing.

commit.interval.ms: Frequency of committing processed offsets.

acks: Ensures all replicas acknowledge writes for durability.

Commands

Create an input topic with 3 partitions to distribute the stream data.

Terminal

kafka-topics --create --topic input-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic input-topic.

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Create an output topic where processed results will be written exactly once.

Terminal

kafka-topics --create --topic output-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

Expected Output

Created topic output-topic.

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Send sample messages to the input topic to simulate streaming data.

Terminal

kafka-console-producer --topic input-topic --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send messages to

→

--bootstrap-server - Kafka broker address

Run the Kafka Streams application configured for exactly-once processing to consume from input-topic and produce to output-topic.

Terminal

java -jar exactly-once-streams-app.jar --config kafka-streams.properties

Expected Output

INFO  KafkaStreams: Started application exactly-once-app

INFO  KafkaStreams: State transition from CREATED to RUNNING

Read the processed messages from the output topic to verify exactly-once processing results.

Terminal

kafka-console-consumer --topic output-topic --bootstrap-server localhost:9092 --from-beginning

Expected Output

processed-message-1

processed-message-2

processed-message-3

→

--from-beginning - Read all messages from the start

Key Concept

If you remember nothing else from this pattern, remember: exactly-once processing guarantees each message affects the system only once, preventing duplicates even if retries happen.

Common Mistakes

Not setting processing.guarantee to exactly_once_v2 in the configuration.

Without this setting, Kafka Streams defaults to at-least-once processing, which can cause duplicate processing.

Always set processing.guarantee=exactly_once_v2 to enable exactly-once semantics.

Using acks=1 instead of acks=all in producer configuration.

acks=1 risks data loss if the leader broker fails before replicas confirm, breaking exactly-once guarantees.

Set acks=all to ensure all replicas confirm writes for durability.

Enabling caching (cache.max.bytes.buffering > 0) during exactly-once processing.

Caching delays commits and can cause state inconsistencies affecting exactly-once guarantees.

Set cache.max.bytes.buffering=0 to disable caching for immediate processing.

Summary

Create input and output Kafka topics with proper partitions and replication.

Configure Kafka Streams with processing.guarantee=exactly_once_v2 and acks=all for durability.

Run the Kafka Streams app to process messages exactly once from input to output topic.

Use console producer and consumer to send and verify messages ensuring no duplicates.

Practice

1. What is the main benefit of exactly-once stream processing in Kafka?

easy

A. It ensures messages are processed only once, avoiding duplicates.

B. It speeds up message delivery by skipping acknowledgments.

C. It allows messages to be processed multiple times for safety.

D. It disables message ordering to improve throughput.

Solution

1. Step 1: Understand exactly-once processing goal

   Exactly-once processing means each message affects the system only once, preventing duplicates.

2. Step 2: Compare options with this goal

   Only It ensures messages are processed only once, avoiding duplicates. describes avoiding duplicates by processing messages once.

3. Final Answer:

   It ensures messages are processed only once, avoiding duplicates. -> Option A

4. Quick Check:

   Exactly-once = no duplicates [OK]

Hint: Exactly-once means no duplicate effects on data [OK]

Common Mistakes:

• Confusing exactly-once with at-least-once processing
• Thinking it speeds up delivery by skipping steps
• Assuming it allows multiple processing for safety

2. Which Kafka producer configuration enables exactly-once semantics?

easy

A. compression.type = none

B. enable.idempotence = true

C. acks = 0

D. max.in.flight.requests.per.connection = 10

Solution

1. Step 1: Identify config for exactly-once

   Enabling idempotence ensures the producer can retry without duplicates.

2. Step 2: Match config options

   Only 'enable.idempotence = true' enables this feature.

3. Final Answer:

   enable.idempotence = true -> Option B

4. Quick Check:

   Idempotence enabled = exactly-once [OK]

Hint: Idempotence true enables exactly-once in Kafka producer [OK]

Common Mistakes:

• Choosing acks=0 which disables acknowledgments
• Confusing compression with processing guarantees
• Setting max.in.flight.requests without idempotence

3. Given this Kafka Streams code snippet, what is the output if a failure occurs before commit?

streams.start();
try {
  streams.process(record);
  streams.commitTransaction();
} catch (Exception e) {
  streams.abortTransaction();
}

medium

A. Processed record is committed multiple times.

B. Processed record may be duplicated after restart.

C. Processed record is committed exactly once.

D. Processed record is lost and not committed.

Solution

1. Step 1: Analyze transaction handling

   If an exception occurs before commit, abortTransaction() is called, rolling back changes.

2. Step 2: Determine output effect

   Rollback means the processed record is not committed and effectively lost in this run.

3. Final Answer:

   Processed record is lost and not committed. -> Option D

4. Quick Check:

   Abort transaction = no commit = lost record [OK]

Hint: Abort transaction means no commit, so data is lost [OK]

Common Mistakes:

• Assuming record is committed despite failure
• Thinking abortTransaction duplicates records
• Confusing commitTransaction with abortTransaction

4. Identify the error in this Kafka Streams code snippet for exactly-once processing:

props.put("enable.idempotence", "true");
props.put("transactional.id", "txn-1");
KafkaProducer producer = new KafkaProducer(props);
producer.beginTransaction();
producer.send(record);
// Missing commit or abort here
producer.close();

medium

A. Missing commitTransaction or abortTransaction before close.

B. enable.idempotence should be false for transactions.

C. transactional.id must be unique per message.

D. KafkaProducer cannot be closed after sending messages.

Solution

1. Step 1: Review transaction lifecycle

   After beginTransaction and send, a commit or abort must finalize the transaction.

2. Step 2: Identify missing step

   The code closes producer without committing or aborting, causing incomplete transaction.

3. Final Answer:

   Missing commitTransaction or abortTransaction before close. -> Option A

4. Quick Check:

   Transactions need commit or abort before close [OK]

Hint: Always commit or abort before closing producer [OK]

Common Mistakes:

• Thinking idempotence false is needed for transactions
• Believing transactional.id changes per message
• Assuming producer close auto-commits transactions

5. You want to build a Kafka Streams application that updates user balances exactly once, even if failures occur. Which approach ensures this behavior?

hard

A. Process messages without committing offsets to avoid duplicates.

B. Disable transactions and rely on at-least-once processing with retries.

C. Use idempotent producer with transactions and commit offsets atomically.

D. Use multiple producers without transactions to increase throughput.

Solution

1. Step 1: Understand exactly-once requirements

   Exactly-once needs atomic commit of data and offsets to avoid duplicates or loss.

2. Step 2: Match approach to requirements

   Using idempotent producer with transactions and atomic offset commits ensures exactly-once updates.

3. Final Answer:

   Use idempotent producer with transactions and commit offsets atomically. -> Option C

4. Quick Check:

   Idempotent + transactions + atomic commit = exactly-once [OK]

Hint: Atomic commit with idempotent producer ensures exactly-once [OK]

Common Mistakes:

• Relying on retries without transactions causes duplicates
• Using multiple producers without coordination breaks exactly-once
• Not committing offsets leads to message reprocessing