At-most-once delivery in Kafka - Step-by-Step Execution

Process Flow - At-most-once delivery

Message Produced

↓

Send to Kafka Broker

↓

Message Stored in Partition

↓

Consumer Polls Message

↓

Commit Offset

↓

Process Message

↓

Next Message

At-most-once delivery means messages are sent and processed zero or one time, allowing possible message loss but no duplicates.

Execution Sample

Kafka

producer.send(message)
// no retries
consumer.poll()
consumer.commit()
// commit before process
process(message)

This code sends a message without retries, polls and commits offset before processing, risking message loss if crash after commit before process but no duplicates.

Process Table

Step	Action	Message State	Offset Commit	Result
1	Producer sends message	Message sent to broker	No	Message may be lost if failure occurs
2	Consumer polls message	Message received by consumer	No	Consumer has message
3	Commit offset	Offset committed	Yes	Prevents re-delivery on restart
4	Consumer crashes before process	Message not processed	Yes	Message lost, no re-delivery
5	Consumer restarts, next poll	No message re-delivered	Yes	At-most-once delivery achieved

💡 Consumer crash after offset commit (before process) causes message loss, no re-delivery.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
message_state	Not sent	Sent to broker	Received by consumer	Offset committed	Not processed (crash)	Lost (skipped)
offset_commit	Not committed	Not committed	Not committed	Committed	Committed	Committed

Key Moments - 3 Insights

Why can messages be lost in at-most-once delivery?

Because the consumer may commit the offset after polling but crash before processing, so Kafka skips the message on restart (see execution_table step 4).

Does at-most-once delivery guarantee no duplicate messages?

Yes, because offset committed before processing prevents re-delivery even if processing fails, no producer retries (see execution_table step 5).

What happens if the producer retries sending messages?

Retries can cause duplicates, so at-most-once delivery usually disables retries to avoid duplicates (not shown in this trace).

Visual Quiz - 3 Questions

Look at the execution table, at which step does the consumer crash (after commit before process)?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Check the 'Consumer crashes before process' description in step 4 of the execution_table.

According to the variable tracker, what is the offset_commit state after step 3?

A Not committed

B Committed

C Partially committed

D Unknown

💡 Hint

Look at the offset_commit row in variable_tracker after step 3.

If the producer enabled retries, how would the at-most-once delivery behavior change?

A Duplicates could occur

B No change in delivery guarantees

C Messages might be lost more often

D Messages would be delivered exactly once

💡 Hint

Refer to key_moments about retries causing duplicates.

Concept Snapshot

At-most-once delivery:
- Messages sent once, no retries
- Consumer commits offset before process
- Possible message loss, no duplicates
- Fast but less reliable
- Use when losing messages is acceptable

Full Transcript

At-most-once delivery in Kafka means messages are sent and processed zero or one time. The producer sends messages without retries. The consumer polls, commits offset before processing. If the consumer crashes after commit before processing, the message is lost and not re-delivered. This avoids duplicates but risks losing messages. The execution table shows each step from sending to the crash scenario. Variable tracking shows message state and offset commit status. Key moments clarify why messages can be lost and why duplicates do not occur. The quiz tests understanding of crash timing, offset commit state, and retry effects. This delivery mode is fast but less reliable, suitable when losing some messages is acceptable.

Practice

1.

What does at-most-once delivery guarantee in Kafka messaging?

easy

A. Messages are sent at least once, possibly duplicated

B. Messages are sent exactly once with no loss

C. Messages are sent only once but some may be lost

D. Messages are sent multiple times intentionally

Solution

1. Step 1: Understand at-most-once delivery meaning

   At-most-once delivery means messages are sent once without retries, so duplicates are avoided but some messages might be lost.

2. Step 2: Compare with other delivery guarantees

   At-least-once allows duplicates, exactly-once avoids loss and duplicates, so only at-most-once fits the description of possible loss but no duplicates.

3. Final Answer:

   Messages are sent only once but some may be lost -> Option C

4. Quick Check:

   At-most-once = no duplicates, possible loss [OK]

Hint: At-most-once means no retries, so possible message loss [OK]

Common Mistakes:

• Confusing at-most-once with exactly-once delivery
• Assuming no messages are ever lost
• Thinking duplicates can occur in at-most-once

2.

Which Kafka producer configuration enables at-most-once delivery?

producer = KafkaProducer(acks=?, retries=? )

easy

A. acks=all, retries=5

B. acks=0, retries=0

C. acks=1, retries=3

D. acks=all, retries=0

Solution

1. Step 1: Identify settings for at-most-once

   At-most-once requires no retries and no acknowledgments, so acks=0 and retries=0 are needed.

2. Step 2: Check other options

   Other options enable retries or acknowledgments, which allow retries and possible duplicates, not at-most-once.

3. Final Answer:

   acks=0, retries=0 -> Option B

4. Quick Check:

   acks=0 & retries=0 = at-most-once [OK]

Hint: Set acks and retries to zero for at-most-once [OK]

Common Mistakes:

• Setting retries > 0 allows retries, not at-most-once
• Using acks=all causes waiting for confirmation
• Confusing acks=1 with acks=0

3.

Given the following Kafka producer code snippet, what is the expected delivery guarantee?

producer = KafkaProducer(acks=0, retries=0)
producer.send('topic', b'message')

medium

A. At-most-once delivery with possible message loss

B. No message will be sent

C. Exactly-once delivery with no duplicates

D. At-least-once delivery with possible duplicates

Solution

1. Step 1: Analyze producer settings

   With acks=0 and retries=0, the producer does not wait for acknowledgment and does not retry on failure.

2. Step 2: Determine delivery guarantee

   This means messages may be lost if sending fails, but duplicates won't occur, matching at-most-once delivery.

3. Final Answer:

   At-most-once delivery with possible message loss -> Option A

4. Quick Check:

   acks=0 & retries=0 = at-most-once [OK]

Hint: acks=0 and retries=0 means no retries, so at-most-once [OK]

Common Mistakes:

• Assuming messages are always delivered
• Confusing at-most-once with exactly-once
• Thinking duplicates can happen here

4.

Identify the error in this Kafka producer configuration for at-most-once delivery:

producer = KafkaProducer(acks=1, retries=0)
producer.send('topic', b'message')

medium

A. acks=1 causes waiting for leader ack, not at-most-once

B. retries=0 causes infinite retries

C. Missing enable.idempotence=True

D. Topic name must be uppercase

Solution

1. Step 1: Check acks setting

   acks=1 means the producer waits for leader acknowledgment, allowing retries and possible duplicates, not at-most-once.

2. Step 2: Verify retries and other settings

   Retries=0 disables retries, but acks=1 still waits for confirmation, so this is not pure at-most-once.

3. Final Answer:

   acks=1 causes waiting for leader ack, not at-most-once -> Option A

4. Quick Check:

   acks must be 0 for at-most-once [OK]

Hint: acks must be 0 for at-most-once delivery [OK]

Common Mistakes:

• Thinking retries=0 means no retries but ignoring acks effect
• Assuming idempotence is required for at-most-once
• Believing topic names must be uppercase

5.

You want to configure a Kafka producer for at-most-once delivery but also need to minimize message loss. Which combination of settings is best?

Options:
A) acks=1, retries=0
B) acks=0, retries=0
C) acks=0, retries=3
D) acks=all, retries=0

hard

A. acks=all, retries=0 (wait for all acks, no retries)

B. acks=0, retries=0 (pure at-most-once, possible loss)

C. acks=0, retries=3 (retries with no ack, risky duplicates)

D. acks=1, retries=0 (wait for ack, no retries, less loss)

Solution

1. Step 1: Understand trade-offs

   Pure at-most-once (acks=0, retries=0) risks message loss. Waiting for ack reduces loss but may not be pure at-most-once.

2. Step 2: Evaluate options

   acks=1, retries=0 (wait for ack, no retries, less loss) waits for leader ack but disables retries, reducing loss while avoiding duplicates, closest to at-most-once with less loss.

3. Final Answer:

   acks=1, retries=0 (wait for ack, no retries, less loss) -> Option D

4. Quick Check:

   acks=1 & retries=0 balances loss and duplicates [OK]

Hint: Wait for ack but disable retries to reduce loss in at-most-once [OK]

Common Mistakes:

• Choosing retries > 0 breaks at-most-once
• Ignoring ack setting impact on loss
• Assuming acks=all with no retries is at-most-once