At-least-once delivery in Kafka - Step-by-Step Execution

Process Flow - At-least-once delivery

Producer sends message

↓

Message stored in Kafka topic

↓

Consumer reads message

↓

Consumer processes message

↓

Consumer commits offset

↓

If commit fails or consumer crashes

↓

Message re-delivered

↓

Consumer processes message again

↓

At-least-once delivery achieved

This flow shows how Kafka ensures messages are delivered at least once by re-delivering if acknowledgments fail.

Execution Sample

Kafka

producer.send(topic, message)
consumer.poll()
process(message)
consumer.commitSync()

Producer sends a message, consumer reads and processes it, then commits offset to mark message as processed.

Process Table

Step	Action	Message State	Offset Commit	Result
1	Producer sends message	Message stored in topic	No	Message ready for consumption
2	Consumer polls message	Message fetched	No	Message available to process
3	Consumer processes message	Processing	No	Message being handled
4	Consumer commits offset	Processed	Commit sent	Offset saved
5	Consumer crashes before commit	Processed	No commit	Offset not saved, message will be re-delivered
6	Consumer restarts and polls	Message fetched again	No	Message re-delivered
7	Consumer processes message again	Processing	No	Message handled again
8	Consumer commits offset	Processed	Commit sent	Offset saved, no more re-delivery

💡 Consumer commits offset successfully, so message is not re-delivered again.

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 5	After Step 6	After Step 8
message_state	Not sent	Fetched	Processed	Processed	Fetched again	Processed
offset_committed	False	False	True	False	False	True

Key Moments - 2 Insights

Why does the message get processed twice sometimes?

If the consumer crashes before committing the offset (see step 5 in execution_table), Kafka will re-deliver the message to ensure it is not lost, causing duplicate processing.

What does committing the offset mean?

Committing the offset (step 4 and 8) tells Kafka the consumer has successfully processed the message and it can move on, preventing re-delivery.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the consumer fail to commit the offset?

A Step 5

B Step 4

C Step 6

D Step 8

💡 Hint

Check the 'Offset Commit' column for 'No commit' status.

According to variable_tracker, what is the offset_committed value after step 6?

A True

B False

C Unknown

D Not applicable

💡 Hint

Look at the 'offset_committed' row under 'After Step 6' column.

If the consumer commits offset immediately after processing, what changes in the execution_table?

A Message will be lost

B Consumer will crash more often

C Message will never be re-delivered

D Message will be processed twice

💡 Hint

Refer to the flow where committing offset prevents re-delivery.

Concept Snapshot

At-least-once delivery in Kafka means messages may be delivered more than once but never lost.
Producer sends messages to topic.
Consumer reads and processes messages.
Consumer commits offset after processing.
If commit fails, message is re-delivered.
This ensures reliability but can cause duplicates.

Full Transcript

At-least-once delivery means Kafka guarantees each message is delivered to the consumer at least once. The producer sends a message to a Kafka topic. The consumer polls and processes the message. After processing, the consumer commits the offset to mark the message as done. If the consumer crashes before committing, Kafka will re-deliver the message when the consumer restarts. This causes the message to be processed again, ensuring no message is lost but duplicates can happen. Committing the offset is key to preventing re-delivery. This flow ensures reliability in message delivery.