Exactly-once semantics (EOS) in Kafka - Step-by-Step Execution

Process Flow - Exactly-once semantics (EOS)

Start Producer

↓

Begin Transaction

↓

Send Messages

↓

Commit Transaction

↓

Broker stores messages atomically

↓

Consumer reads messages once

↓

End

The producer starts a transaction, sends messages, commits atomically, ensuring messages are stored and consumed exactly once.

Execution Sample

Kafka

producer.initTransactions();
producer.beginTransaction();
producer.send(record1);
producer.send(record2);
producer.commitTransaction();

This code shows a Kafka producer sending two messages within a transaction to guarantee exactly-once delivery.

Process Table

Step	Action	State Change	Result
1	Initialize transactions	Producer ready for transactions	No messages sent yet
2	Begin transaction	Transaction started	Messages will be sent atomically
3	Send record1	Record1 buffered in transaction	Not visible to consumers yet
4	Send record2	Record2 buffered in transaction	Not visible to consumers yet
5	Commit transaction	Transaction committed atomically	Both records become visible exactly once
6	Consumer reads messages	Reads record1 and record2 once	No duplicates or missing messages
7	End	Transaction complete	Exactly-once semantics ensured

💡 Transaction committed, messages stored atomically, ensuring exactly-once delivery

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
transactionState	none	started	started	started	committed	committed
bufferedMessages	[]	[]	[record1]	[record1, record2]	[]	[]
messagesVisibleToConsumer	[]	[]	[]	[]	[record1, record2]	[record1, record2]

Key Moments - 3 Insights

Why are messages not visible to consumers before commit?

Because messages are buffered in the transaction (see steps 3 and 4 in execution_table), they become visible only after commit (step 5), ensuring atomic visibility.

What happens if the transaction is aborted instead of committed?

If aborted, buffered messages are discarded and never become visible, preventing duplicates or partial writes (not shown in this trace but important for EOS).

How does exactly-once semantics prevent duplicates?

By committing messages atomically and tracking offsets, Kafka ensures consumers read each message once after commit (step 6), avoiding duplicates.

Visual Quiz - 3 Questions

Look at the execution_table, what is the state of bufferedMessages after step 4?

A []

B [record1]

C [record1, record2]

D [record2]

💡 Hint

Check the 'bufferedMessages' variable in variable_tracker after step 4

At which step do messages become visible to consumers?

A Step 3

B Step 5

C Step 2

D Step 4

💡 Hint

Look at 'messagesVisibleToConsumer' in variable_tracker and execution_table rows

If the transaction was aborted instead of committed at step 5, what would happen to bufferedMessages?

A They would be discarded and not sent

B They would be sent to consumers immediately

C They would remain buffered indefinitely

D They would be duplicated

💡 Hint

Recall the explanation in key_moments about aborting transactions

Concept Snapshot

Exactly-once semantics (EOS) in Kafka:
- Producer starts a transaction
- Sends messages buffered in transaction
- Commits transaction atomically
- Messages become visible once committed
- Consumers read messages exactly once
- Prevents duplicates and data loss

Full Transcript

Exactly-once semantics (EOS) in Kafka means messages are sent and stored so that consumers see each message only once. The producer starts a transaction, sends messages buffered inside it, and commits the transaction atomically. Before commit, messages are not visible to consumers. After commit, all messages become visible together. This prevents duplicates and partial writes. If the transaction aborts, messages are discarded. Consumers read committed messages exactly once, ensuring reliable processing.