At-least-once delivery in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka consumer commit example?

Consider a Kafka consumer configured for at-least-once delivery. What will be the output of the following code snippet when consuming messages?

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer('my_topic', group_id='group1', enable_auto_commit=False)

for message in consumer:
    print(f"Received: {message.value.decode('utf-8')}")
    consumer.commit()
    break

A Prints the first message once and commits offset immediately.

B Raises an error because commit() is called without arguments.

C Prints the first message multiple times without committing offset.

D Prints no messages because auto commit is disabled.

💡 Hint

Think about what happens when auto commit is disabled and commit() is called manually after processing a message.

Since auto commit is disabled, the consumer must commit offsets manually. The code commits after processing the first message, so it prints it once and commits the offset, ensuring at-least-once delivery.

🧠 Conceptual

intermediate

Which Kafka configuration ensures at-least-once delivery?

Which configuration option in Kafka consumer guarantees at-least-once message delivery?

A enable.auto.commit=true with auto commit interval set to 1 ms

B enable.auto.commit=true with auto commit interval set to 1 hour

C enable.auto.commit=false and manual commit after processing

D enable.auto.commit=false and never commit offsets

💡 Hint

At-least-once means messages can be processed more than once but never lost.

Disabling auto commit and committing offsets manually after processing ensures that messages are not lost but might be processed multiple times, which is at-least-once delivery.

🔧 Debug

advanced

Why does this Kafka consumer cause message loss despite manual commits?

Given the following code, why might some messages be lost even though manual commits are used?

Kafka

consumer = KafkaConsumer('topic', group_id='g1', enable_auto_commit=False)

for msg in consumer:
    process(msg)
    if some_condition():
        consumer.commit()

A The consumer group id is missing, causing offset commits to fail.

B Manual commit is not supported in Kafka consumers.

C The consumer must call commit_async() instead of commit().

D Offsets are only committed conditionally, so some processed messages are not committed.

💡 Hint

Think about when offsets are committed in relation to message processing.

Since commits happen only if some_condition() is true, messages processed when the condition is false are not committed, causing potential message loss on restart.

📝 Syntax

advanced

Which option correctly commits offsets after processing in Kafka consumer?

Choose the correct code snippet that commits offsets manually after processing each message.

Kafka

consumer = KafkaConsumer('topic', group_id='g1', enable_auto_commit=False)

for msg in consumer:
    process(msg)
    # commit offset here

A consumer.commit()

B consumer.commit(offsets={msg.topic: msg.offset})

C consumer.commit_async()

D consumer.commit(offsets={TopicPartition(msg.topic, msg.partition): OffsetAndMetadata(msg.offset, None)})

💡 Hint

consumer.commit() without arguments is the standard way to manually commit offsets after processing a message.

Option A calls consumer.commit(), which automatically commits the tracked offsets up to the next offset after the last processed message. This is the correct and standard way for manual commits in kafka-python.

🚀 Application

expert

How to implement at-least-once delivery with Kafka producer and consumer?

You want to ensure at-least-once delivery in a Kafka system. Which combination of producer and consumer settings and code logic achieves this?

A Producer disables retries; consumer commits offsets before processing messages.

B Producer uses acks='all' and retries; consumer disables auto commit and commits offsets after processing each message.

C Producer uses acks=1; consumer never commits offsets.

D Producer uses acks=0; consumer enables auto commit with interval 1000ms.

💡 Hint

At-least-once means messages are never lost but may be duplicated.

Producer with acks='all' and retries ensures messages are not lost on send. Consumer disabling auto commit and committing offsets after processing ensures messages are processed at least once.

Practice

1. What does at-least-once delivery guarantee in Kafka?

easy

A. Messages are delivered only once and never retried

B. Messages are delivered exactly once with no duplicates

C. Messages are never lost but might be delivered more than once

D. Messages might be lost but never duplicated

Solution

1. Step 1: Understand delivery guarantees

   At-least-once means Kafka ensures messages reach the consumer at least once, so no loss occurs.

2. Step 2: Consider duplicates possibility

   Because retries can happen, duplicates may occur, so messages might be delivered multiple times.

3. Final Answer:

   Messages are never lost but might be delivered more than once -> Option C

4. Quick Check:

   At-least-once = no loss, possible duplicates [OK]

Hint: At-least-once means no loss, duplicates possible [OK]

Common Mistakes:

• Confusing at-least-once with exactly-once delivery
• Thinking messages can be lost
• Assuming no duplicates occur

2. Which Kafka consumer configuration is essential for implementing at-least-once delivery?

easy

A. enable.auto.commit = false

B. enable.auto.commit = true

C. auto.offset.reset = earliest

D. max.poll.records = 1

Solution

1. Step 1: Identify offset commit behavior

   At-least-once requires manual control of offset commits to avoid losing messages.

2. Step 2: Choose correct config for manual commit

   Setting enable.auto.commit = false disables automatic commits, allowing manual commits after processing.

3. Final Answer:

   enable.auto.commit = false -> Option A

4. Quick Check:

   Manual commit enabled by disabling auto commit [OK]

Hint: Disable auto commit for manual offset control [OK]

Common Mistakes:

• Leaving auto commit enabled causes possible message loss
• Confusing auto.offset.reset with commit control
• Assuming max.poll.records affects delivery guarantee

3. Given this Kafka consumer code snippet, what is the expected behavior regarding message delivery?

consumer.subscribe(['topic'])
for message in consumer:
    process(message)
    consumer.commit()

medium

A. Messages are processed at least once, duplicates possible

B. Messages may be lost if commit happens before processing

C. Messages are processed exactly once with no duplicates

D. Messages are never committed, causing infinite reprocessing

Solution

1. Step 1: Analyze commit timing

   Commit happens after processing each message, so offsets are saved only after successful processing.

2. Step 2: Understand delivery guarantee

   If a failure occurs before commit, message will be reprocessed, causing duplicates but no loss.

3. Final Answer:

   Messages are processed at least once, duplicates possible -> Option A

4. Quick Check:

   Commit after processing = at-least-once delivery [OK]

Hint: Commit after processing ensures no loss, duplicates possible [OK]

Common Mistakes:

• Assuming commit before processing
• Thinking duplicates cannot happen
• Believing messages are lost on failure

4. Identify the bug in this Kafka consumer code that aims for at-least-once delivery:

consumer.subscribe(['topic'])
for message in consumer:
    consumer.commit()
    process(message)

medium

A. Using subscribe instead of assign causes duplicates

B. Committing offset before processing causes message loss on failure

C. Not committing offsets causes infinite reprocessing

D. process() function is missing a return statement

Solution

1. Step 1: Check commit position relative to processing

   Offsets are committed before processing, so if processing fails, message is lost.

2. Step 2: Understand at-least-once requirement

   To guarantee at-least-once, commit must happen after successful processing to avoid loss.

3. Final Answer:

   Committing offset before processing causes message loss on failure -> Option B

4. Quick Check:

   Commit after processing to avoid loss [OK]

Hint: Commit offsets only after processing completes [OK]

Common Mistakes:

• Committing before processing causes data loss
• Ignoring commit timing importance
• Confusing subscribe with assign usage

5. You want to ensure at-least-once delivery in Kafka but avoid processing duplicates as much as possible. Which approach best achieves this?

hard

A. Use exactly-once semantics without any manual offset control

B. Enable auto commit and process messages without tracking offsets

C. Commit offsets before processing to speed up throughput

D. Manually commit offsets after processing and implement idempotent processing logic

Solution

1. Step 1: Combine manual commit with processing logic

   Manual commit after processing ensures no message loss; idempotent logic handles duplicates safely.

2. Step 2: Evaluate other options

   Auto commit risks loss; committing before processing risks loss; exactly-once semantics require special setup.

3. Final Answer:

   Manually commit offsets after processing and implement idempotent processing logic -> Option D

4. Quick Check:

   Manual commit + idempotency = best at-least-once practice [OK]

Hint: Use manual commit plus idempotent processing to reduce duplicates [OK]

Common Mistakes:

• Relying on auto commit for at-least-once
• Committing offsets before processing
• Ignoring idempotent processing importance