At-least-once delivery in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka consumer commit example?

Consider a Kafka consumer configured for at-least-once delivery. What will be the output of the following code snippet when consuming messages?

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer('my_topic', group_id='group1', enable_auto_commit=False)

for message in consumer:
    print(f"Received: {message.value.decode('utf-8')}")
    consumer.commit()
    break

A Prints the first message once and commits offset immediately.

B Raises an error because commit() is called without arguments.

C Prints the first message multiple times without committing offset.

D Prints no messages because auto commit is disabled.

💡 Hint

Think about what happens when auto commit is disabled and commit() is called manually after processing a message.

Since auto commit is disabled, the consumer must commit offsets manually. The code commits after processing the first message, so it prints it once and commits the offset, ensuring at-least-once delivery.

🧠 Conceptual

intermediate

Which Kafka configuration ensures at-least-once delivery?

Which configuration option in Kafka consumer guarantees at-least-once message delivery?

A enable.auto.commit=true with auto commit interval set to 1 ms

B enable.auto.commit=true with auto commit interval set to 1 hour

C enable.auto.commit=false and manual commit after processing

D enable.auto.commit=false and never commit offsets

💡 Hint

At-least-once means messages can be processed more than once but never lost.

Disabling auto commit and committing offsets manually after processing ensures that messages are not lost but might be processed multiple times, which is at-least-once delivery.

🔧 Debug

advanced

Why does this Kafka consumer cause message loss despite manual commits?

Given the following code, why might some messages be lost even though manual commits are used?

Kafka

consumer = KafkaConsumer('topic', group_id='g1', enable_auto_commit=False)

for msg in consumer:
    process(msg)
    if some_condition():
        consumer.commit()

A The consumer group id is missing, causing offset commits to fail.

B Manual commit is not supported in Kafka consumers.

C The consumer must call commit_async() instead of commit().

D Offsets are only committed conditionally, so some processed messages are not committed.

💡 Hint

Think about when offsets are committed in relation to message processing.

Since commits happen only if some_condition() is true, messages processed when the condition is false are not committed, causing potential message loss on restart.

📝 Syntax

advanced

Which option correctly commits offsets after processing in Kafka consumer?

Choose the correct code snippet that commits offsets manually after processing each message.

Kafka

consumer = KafkaConsumer('topic', group_id='g1', enable_auto_commit=False)

for msg in consumer:
    process(msg)
    # commit offset here

A consumer.commit()

B consumer.commit(offsets={msg.topic: msg.offset})

C consumer.commit_async()

D consumer.commit(offsets={TopicPartition(msg.topic, msg.partition): OffsetAndMetadata(msg.offset, None)})

💡 Hint

consumer.commit() without arguments is the standard way to manually commit offsets after processing a message.

Option A calls consumer.commit(), which automatically commits the tracked offsets up to the next offset after the last processed message. This is the correct and standard way for manual commits in kafka-python.

🚀 Application

expert

How to implement at-least-once delivery with Kafka producer and consumer?

You want to ensure at-least-once delivery in a Kafka system. Which combination of producer and consumer settings and code logic achieves this?

A Producer disables retries; consumer commits offsets before processing messages.

B Producer uses acks='all' and retries; consumer disables auto commit and commits offsets after processing each message.

C Producer uses acks=1; consumer never commits offsets.

D Producer uses acks=0; consumer enables auto commit with interval 1000ms.

💡 Hint

At-least-once means messages are never lost but may be duplicated.

Producer with acks='all' and retries ensures messages are not lost on send. Consumer disabling auto commit and committing offsets after processing ensures messages are processed at least once.