Exactly-once stream processing in Kafka - Step-by-Step Execution

Process Flow - Exactly-once stream processing

Start Stream Processing

↓

Read Input Record

↓

Process Record

↓

Begin Transaction

↓

Write Output Record

↓

Acknowledge Input

↓

Commit Transaction

↓

Next Record or End

The flow reads each record, processes it, writes output within a transaction, acknowledges input by sending offsets to the transaction, commits to ensure exactly-once, then moves on.

Execution Sample

Kafka

producer.initTransactions();
producer.beginTransaction();
producer.send(outputRecord);
producer.sendOffsetsToTransaction(consumer.assignment(), groupMetadata);
producer.commitTransaction();

This code snippet shows how a Kafka producer uses transactions, including output records and input offsets, to ensure exactly-once stream processing.

Process Table

Step	Action	Transaction State	Output Produced	Input Acknowledged
1	Initialize transactions	Initialized	None	No
2	Begin transaction	Active	None	No
3	Send output record	Active	Output record buffered	No
4	Send offsets to transaction	Active	Output record buffered	No
5	Commit transaction	Committed	Output record committed	Yes
6	Process next record	None	None	No

💡 Processing stops when no more input records are available.

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 5	Final
transactionState	None	Active	Active	Committed	None
outputBuffer	Empty	Empty	Buffered output record	Committed output record	Empty
inputAcknowledged	No	No	No	Yes	No

Key Moments - 3 Insights

Why do we need to begin and commit a transaction around sending output?

Because the transaction groups the output and input acknowledgment so they happen exactly once together, as shown in steps 2 to 5 in the execution table.

What happens if the transaction is not committed?

The output record is not saved permanently and the input is not acknowledged, so the record will be reprocessed, ensuring no data loss or duplicates.

Why send offsets to the transaction before committing?

Sending offsets before commit ensures that the output and input acknowledgment are stored atomically, preventing data loss or duplicates (see step 5).

Visual Quiz - 3 Questions

Look at the execution table, what is the transaction state after step 3?

A Initialized

B Committed

C Active

D None

💡 Hint

Check the 'Transaction State' column at step 3 in the execution table.

At which step is the input record acknowledged?

A Step 2

B Step 5

C Step 4

D Step 6

💡 Hint

Look at the 'Input Acknowledged' column in the execution table.

If the transaction is not committed, what happens to the output record?

A It is buffered but not saved

B It is permanently saved

C It is deleted immediately

D It is acknowledged as sent

💡 Hint

Refer to the 'Output Produced' column before and after commit in the execution table.

Concept Snapshot

Exactly-once stream processing in Kafka uses transactions.
Start a transaction before sending output.
Send output records inside the transaction.
Send input offsets to the transaction.
Commit the transaction to save output and offsets atomically.
This ensures each input is processed exactly once.

Full Transcript

Exactly-once stream processing means each input record is processed one time only, no duplicates or losses. Kafka achieves this by using transactions. The process starts by initializing transactions. For each input record, a transaction begins. The output record is sent inside this transaction but not visible outside yet. Input offsets are sent to the transaction. When the transaction commits, the output and offsets are saved atomically. If a failure happens before commit, the output is discarded and input is not acknowledged, so the record will be retried. This flow guarantees exactly-once processing by linking output and input acknowledgment in one atomic transaction.

Practice

1. What is the main benefit of exactly-once stream processing in Kafka?

easy

A. It ensures messages are processed only once, avoiding duplicates.

B. It speeds up message delivery by skipping acknowledgments.

C. It allows messages to be processed multiple times for safety.

D. It disables message ordering to improve throughput.

Solution

1. Step 1: Understand exactly-once processing goal

   Exactly-once processing means each message affects the system only once, preventing duplicates.

2. Step 2: Compare options with this goal

   Only It ensures messages are processed only once, avoiding duplicates. describes avoiding duplicates by processing messages once.

3. Final Answer:

   It ensures messages are processed only once, avoiding duplicates. -> Option A

4. Quick Check:

   Exactly-once = no duplicates [OK]

Hint: Exactly-once means no duplicate effects on data [OK]

Common Mistakes:

• Confusing exactly-once with at-least-once processing
• Thinking it speeds up delivery by skipping steps
• Assuming it allows multiple processing for safety

2. Which Kafka producer configuration enables exactly-once semantics?

easy

A. compression.type = none

B. enable.idempotence = true

C. acks = 0

D. max.in.flight.requests.per.connection = 10

Solution

1. Step 1: Identify config for exactly-once

   Enabling idempotence ensures the producer can retry without duplicates.

2. Step 2: Match config options

   Only 'enable.idempotence = true' enables this feature.

3. Final Answer:

   enable.idempotence = true -> Option B

4. Quick Check:

   Idempotence enabled = exactly-once [OK]

Hint: Idempotence true enables exactly-once in Kafka producer [OK]

Common Mistakes:

• Choosing acks=0 which disables acknowledgments
• Confusing compression with processing guarantees
• Setting max.in.flight.requests without idempotence

3. Given this Kafka Streams code snippet, what is the output if a failure occurs before commit?

streams.start();
try {
  streams.process(record);
  streams.commitTransaction();
} catch (Exception e) {
  streams.abortTransaction();
}

medium

A. Processed record is committed multiple times.

B. Processed record may be duplicated after restart.

C. Processed record is committed exactly once.

D. Processed record is lost and not committed.

Solution

1. Step 1: Analyze transaction handling

   If an exception occurs before commit, abortTransaction() is called, rolling back changes.

2. Step 2: Determine output effect

   Rollback means the processed record is not committed and effectively lost in this run.

3. Final Answer:

   Processed record is lost and not committed. -> Option D

4. Quick Check:

   Abort transaction = no commit = lost record [OK]

Hint: Abort transaction means no commit, so data is lost [OK]

Common Mistakes:

• Assuming record is committed despite failure
• Thinking abortTransaction duplicates records
• Confusing commitTransaction with abortTransaction

4. Identify the error in this Kafka Streams code snippet for exactly-once processing:

props.put("enable.idempotence", "true");
props.put("transactional.id", "txn-1");
KafkaProducer producer = new KafkaProducer(props);
producer.beginTransaction();
producer.send(record);
// Missing commit or abort here
producer.close();

medium

A. Missing commitTransaction or abortTransaction before close.

B. enable.idempotence should be false for transactions.

C. transactional.id must be unique per message.

D. KafkaProducer cannot be closed after sending messages.

Solution

1. Step 1: Review transaction lifecycle

   After beginTransaction and send, a commit or abort must finalize the transaction.

2. Step 2: Identify missing step

   The code closes producer without committing or aborting, causing incomplete transaction.

3. Final Answer:

   Missing commitTransaction or abortTransaction before close. -> Option A

4. Quick Check:

   Transactions need commit or abort before close [OK]

Hint: Always commit or abort before closing producer [OK]

Common Mistakes:

• Thinking idempotence false is needed for transactions
• Believing transactional.id changes per message
• Assuming producer close auto-commits transactions

5. You want to build a Kafka Streams application that updates user balances exactly once, even if failures occur. Which approach ensures this behavior?

hard

A. Process messages without committing offsets to avoid duplicates.

B. Disable transactions and rely on at-least-once processing with retries.

C. Use idempotent producer with transactions and commit offsets atomically.

D. Use multiple producers without transactions to increase throughput.

Solution

1. Step 1: Understand exactly-once requirements

   Exactly-once needs atomic commit of data and offsets to avoid duplicates or loss.

2. Step 2: Match approach to requirements

   Using idempotent producer with transactions and atomic offset commits ensures exactly-once updates.

3. Final Answer:

   Use idempotent producer with transactions and commit offsets atomically. -> Option C

4. Quick Check:

   Idempotent + transactions + atomic commit = exactly-once [OK]

Hint: Atomic commit with idempotent producer ensures exactly-once [OK]

Common Mistakes:

• Relying on retries without transactions causes duplicates
• Using multiple producers without coordination breaks exactly-once
• Not committing offsets leads to message reprocessing