At-least-once delivery in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does At-least-once delivery mean in Kafka?

It means each message is delivered to the consumer one or more times, ensuring no message is lost but duplicates can occur.

intermediate

How does Kafka achieve At-least-once delivery?

By committing offsets only after processing messages, so if a failure happens before commit, messages are reprocessed.

beginner

What is the risk of At-least-once delivery?

The risk is duplicate message processing, because messages may be delivered more than once.

intermediate

Which Kafka consumer setting helps control At-least-once delivery behavior?

The enable.auto.commit setting controls if offsets are committed automatically or manually, affecting delivery guarantees.

beginner

Why might you choose At-least-once delivery over At-most-once?

Because it guarantees no message loss, which is important for critical data even if duplicates happen.

What does At-least-once delivery guarantee in Kafka?

A Messages may be lost but never duplicated

B No message is lost, but duplicates can occur

C Each message is delivered exactly once

D Messages are delivered only once without fail

At-least-once delivery ensures messages are never lost but may be delivered multiple times.

Which Kafka feature helps avoid message loss in At-least-once delivery?

A Auto-deleting messages immediately

B Disabling consumer groups

C Manual offset commit after processing

D Using only one partition

Manual offset commit after processing ensures messages are reprocessed if failure occurs before commit.

What is a common side effect of At-least-once delivery?

A Duplicate message processing

B Message loss

C Faster message delivery

D No need to commit offsets

Duplicates happen because messages may be delivered again if offsets are not committed.

If enable.auto.commit is set to true, what happens?

A Consumer crashes immediately

B Offsets are never committed

C Messages are always duplicated

D Offsets are committed automatically, possibly causing At-most-once delivery

Auto commit can cause offsets to be committed before processing, risking message loss (At-most-once).

Why is At-least-once delivery preferred for critical data?

A It prevents message loss even if duplicates occur

B It guarantees no duplicates

C It is faster than other methods

D It requires no offset management

Critical data needs no loss, so At-least-once is safer despite duplicates.

Explain how Kafka ensures At-least-once delivery and what trade-offs it involves.

Think about when offsets are saved and what happens if a failure occurs.

Describe the difference between At-least-once and At-most-once delivery in Kafka.

Consider when offsets are committed relative to message processing.

Practice

1. What does at-least-once delivery guarantee in Kafka?

easy

A. Messages are delivered only once and never retried

B. Messages are delivered exactly once with no duplicates

C. Messages are never lost but might be delivered more than once

D. Messages might be lost but never duplicated

Solution

1. Step 1: Understand delivery guarantees

   At-least-once means Kafka ensures messages reach the consumer at least once, so no loss occurs.

2. Step 2: Consider duplicates possibility

   Because retries can happen, duplicates may occur, so messages might be delivered multiple times.

3. Final Answer:

   Messages are never lost but might be delivered more than once -> Option C

4. Quick Check:

   At-least-once = no loss, possible duplicates [OK]

Hint: At-least-once means no loss, duplicates possible [OK]

Common Mistakes:

• Confusing at-least-once with exactly-once delivery
• Thinking messages can be lost
• Assuming no duplicates occur

2. Which Kafka consumer configuration is essential for implementing at-least-once delivery?

easy

A. enable.auto.commit = false

B. enable.auto.commit = true

C. auto.offset.reset = earliest

D. max.poll.records = 1

Solution

1. Step 1: Identify offset commit behavior

   At-least-once requires manual control of offset commits to avoid losing messages.

2. Step 2: Choose correct config for manual commit

   Setting enable.auto.commit = false disables automatic commits, allowing manual commits after processing.

3. Final Answer:

   enable.auto.commit = false -> Option A

4. Quick Check:

   Manual commit enabled by disabling auto commit [OK]

Hint: Disable auto commit for manual offset control [OK]

Common Mistakes:

• Leaving auto commit enabled causes possible message loss
• Confusing auto.offset.reset with commit control
• Assuming max.poll.records affects delivery guarantee

3. Given this Kafka consumer code snippet, what is the expected behavior regarding message delivery?

consumer.subscribe(['topic'])
for message in consumer:
    process(message)
    consumer.commit()

medium

A. Messages are processed at least once, duplicates possible

B. Messages may be lost if commit happens before processing

C. Messages are processed exactly once with no duplicates

D. Messages are never committed, causing infinite reprocessing

Solution

1. Step 1: Analyze commit timing

   Commit happens after processing each message, so offsets are saved only after successful processing.

2. Step 2: Understand delivery guarantee

   If a failure occurs before commit, message will be reprocessed, causing duplicates but no loss.

3. Final Answer:

   Messages are processed at least once, duplicates possible -> Option A

4. Quick Check:

   Commit after processing = at-least-once delivery [OK]

Hint: Commit after processing ensures no loss, duplicates possible [OK]

Common Mistakes:

• Assuming commit before processing
• Thinking duplicates cannot happen
• Believing messages are lost on failure

4. Identify the bug in this Kafka consumer code that aims for at-least-once delivery:

consumer.subscribe(['topic'])
for message in consumer:
    consumer.commit()
    process(message)

medium

A. Using subscribe instead of assign causes duplicates

B. Committing offset before processing causes message loss on failure

C. Not committing offsets causes infinite reprocessing

D. process() function is missing a return statement

Solution

1. Step 1: Check commit position relative to processing

   Offsets are committed before processing, so if processing fails, message is lost.

2. Step 2: Understand at-least-once requirement

   To guarantee at-least-once, commit must happen after successful processing to avoid loss.

3. Final Answer:

   Committing offset before processing causes message loss on failure -> Option B

4. Quick Check:

   Commit after processing to avoid loss [OK]

Hint: Commit offsets only after processing completes [OK]

Common Mistakes:

• Committing before processing causes data loss
• Ignoring commit timing importance
• Confusing subscribe with assign usage

5. You want to ensure at-least-once delivery in Kafka but avoid processing duplicates as much as possible. Which approach best achieves this?

hard

A. Use exactly-once semantics without any manual offset control

B. Enable auto commit and process messages without tracking offsets

C. Commit offsets before processing to speed up throughput

D. Manually commit offsets after processing and implement idempotent processing logic

Solution

1. Step 1: Combine manual commit with processing logic

   Manual commit after processing ensures no message loss; idempotent logic handles duplicates safely.

2. Step 2: Evaluate other options

   Auto commit risks loss; committing before processing risks loss; exactly-once semantics require special setup.

3. Final Answer:

   Manually commit offsets after processing and implement idempotent processing logic -> Option D

4. Quick Check:

   Manual commit + idempotency = best at-least-once practice [OK]

Hint: Use manual commit plus idempotent processing to reduce duplicates [OK]

Common Mistakes:

• Relying on auto commit for at-least-once
• Committing offsets before processing
• Ignoring idempotent processing importance