At-least-once delivery in Kafka - Step-by-Step Execution

Process Flow - At-least-once delivery

Producer sends message

↓

Message stored in Kafka topic

↓

Consumer reads message

↓

Consumer processes message

↓

Consumer commits offset

↓

If commit fails or consumer crashes

↓

Message re-delivered

↓

Consumer processes message again

↓

At-least-once delivery achieved

This flow shows how Kafka ensures messages are delivered at least once by re-delivering if acknowledgments fail.

Execution Sample

Kafka

producer.send(topic, message)
consumer.poll()
process(message)
consumer.commitSync()

Producer sends a message, consumer reads and processes it, then commits offset to mark message as processed.

Process Table

Step	Action	Message State	Offset Commit	Result
1	Producer sends message	Message stored in topic	No	Message ready for consumption
2	Consumer polls message	Message fetched	No	Message available to process
3	Consumer processes message	Processing	No	Message being handled
4	Consumer commits offset	Processed	Commit sent	Offset saved
5	Consumer crashes before commit	Processed	No commit	Offset not saved, message will be re-delivered
6	Consumer restarts and polls	Message fetched again	No	Message re-delivered
7	Consumer processes message again	Processing	No	Message handled again
8	Consumer commits offset	Processed	Commit sent	Offset saved, no more re-delivery

💡 Consumer commits offset successfully, so message is not re-delivered again.

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 5	After Step 6	After Step 8
message_state	Not sent	Fetched	Processed	Processed	Fetched again	Processed
offset_committed	False	False	True	False	False	True

Key Moments - 2 Insights

Why does the message get processed twice sometimes?

If the consumer crashes before committing the offset (see step 5 in execution_table), Kafka will re-deliver the message to ensure it is not lost, causing duplicate processing.

What does committing the offset mean?

Committing the offset (step 4 and 8) tells Kafka the consumer has successfully processed the message and it can move on, preventing re-delivery.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the consumer fail to commit the offset?

A Step 5

B Step 4

C Step 6

D Step 8

💡 Hint

Check the 'Offset Commit' column for 'No commit' status.

According to variable_tracker, what is the offset_committed value after step 6?

A True

B False

C Unknown

D Not applicable

💡 Hint

Look at the 'offset_committed' row under 'After Step 6' column.

If the consumer commits offset immediately after processing, what changes in the execution_table?

A Message will be lost

B Consumer will crash more often

C Message will never be re-delivered

D Message will be processed twice

💡 Hint

Refer to the flow where committing offset prevents re-delivery.

Concept Snapshot

At-least-once delivery in Kafka means messages may be delivered more than once but never lost.
Producer sends messages to topic.
Consumer reads and processes messages.
Consumer commits offset after processing.
If commit fails, message is re-delivered.
This ensures reliability but can cause duplicates.

Full Transcript

At-least-once delivery means Kafka guarantees each message is delivered to the consumer at least once. The producer sends a message to a Kafka topic. The consumer polls and processes the message. After processing, the consumer commits the offset to mark the message as done. If the consumer crashes before committing, Kafka will re-deliver the message when the consumer restarts. This causes the message to be processed again, ensuring no message is lost but duplicates can happen. Committing the offset is key to preventing re-delivery. This flow ensures reliability in message delivery.

Practice

1. What does at-least-once delivery guarantee in Kafka?

easy

A. Messages are delivered only once and never retried

B. Messages are delivered exactly once with no duplicates

C. Messages are never lost but might be delivered more than once

D. Messages might be lost but never duplicated

Solution

1. Step 1: Understand delivery guarantees

   At-least-once means Kafka ensures messages reach the consumer at least once, so no loss occurs.

2. Step 2: Consider duplicates possibility

   Because retries can happen, duplicates may occur, so messages might be delivered multiple times.

3. Final Answer:

   Messages are never lost but might be delivered more than once -> Option C

4. Quick Check:

   At-least-once = no loss, possible duplicates [OK]

Hint: At-least-once means no loss, duplicates possible [OK]

Common Mistakes:

• Confusing at-least-once with exactly-once delivery
• Thinking messages can be lost
• Assuming no duplicates occur

2. Which Kafka consumer configuration is essential for implementing at-least-once delivery?

easy

A. enable.auto.commit = false

B. enable.auto.commit = true

C. auto.offset.reset = earliest

D. max.poll.records = 1

Solution

1. Step 1: Identify offset commit behavior

   At-least-once requires manual control of offset commits to avoid losing messages.

2. Step 2: Choose correct config for manual commit

   Setting enable.auto.commit = false disables automatic commits, allowing manual commits after processing.

3. Final Answer:

   enable.auto.commit = false -> Option A

4. Quick Check:

   Manual commit enabled by disabling auto commit [OK]

Hint: Disable auto commit for manual offset control [OK]

Common Mistakes:

• Leaving auto commit enabled causes possible message loss
• Confusing auto.offset.reset with commit control
• Assuming max.poll.records affects delivery guarantee

3. Given this Kafka consumer code snippet, what is the expected behavior regarding message delivery?

consumer.subscribe(['topic'])
for message in consumer:
    process(message)
    consumer.commit()

medium

A. Messages are processed at least once, duplicates possible

B. Messages may be lost if commit happens before processing

C. Messages are processed exactly once with no duplicates

D. Messages are never committed, causing infinite reprocessing

Solution

1. Step 1: Analyze commit timing

   Commit happens after processing each message, so offsets are saved only after successful processing.

2. Step 2: Understand delivery guarantee

   If a failure occurs before commit, message will be reprocessed, causing duplicates but no loss.

3. Final Answer:

   Messages are processed at least once, duplicates possible -> Option A

4. Quick Check:

   Commit after processing = at-least-once delivery [OK]

Hint: Commit after processing ensures no loss, duplicates possible [OK]

Common Mistakes:

• Assuming commit before processing
• Thinking duplicates cannot happen
• Believing messages are lost on failure

4. Identify the bug in this Kafka consumer code that aims for at-least-once delivery:

consumer.subscribe(['topic'])
for message in consumer:
    consumer.commit()
    process(message)

medium

A. Using subscribe instead of assign causes duplicates

B. Committing offset before processing causes message loss on failure

C. Not committing offsets causes infinite reprocessing

D. process() function is missing a return statement

Solution

1. Step 1: Check commit position relative to processing

   Offsets are committed before processing, so if processing fails, message is lost.

2. Step 2: Understand at-least-once requirement

   To guarantee at-least-once, commit must happen after successful processing to avoid loss.

3. Final Answer:

   Committing offset before processing causes message loss on failure -> Option B

4. Quick Check:

   Commit after processing to avoid loss [OK]

Hint: Commit offsets only after processing completes [OK]

Common Mistakes:

• Committing before processing causes data loss
• Ignoring commit timing importance
• Confusing subscribe with assign usage

5. You want to ensure at-least-once delivery in Kafka but avoid processing duplicates as much as possible. Which approach best achieves this?

hard

A. Use exactly-once semantics without any manual offset control

B. Enable auto commit and process messages without tracking offsets

C. Commit offsets before processing to speed up throughput

D. Manually commit offsets after processing and implement idempotent processing logic

Solution

1. Step 1: Combine manual commit with processing logic

   Manual commit after processing ensures no message loss; idempotent logic handles duplicates safely.

2. Step 2: Evaluate other options

   Auto commit risks loss; committing before processing risks loss; exactly-once semantics require special setup.

3. Final Answer:

   Manually commit offsets after processing and implement idempotent processing logic -> Option D

4. Quick Check:

   Manual commit + idempotency = best at-least-once practice [OK]

Hint: Use manual commit plus idempotent processing to reduce duplicates [OK]

Common Mistakes:

• Relying on auto commit for at-least-once
• Committing offsets before processing
• Ignoring idempotent processing importance