At-least-once delivery in Kafka - Commands & Configuration

Introduction

At-least-once delivery ensures that every message sent to Kafka is processed at least one time. This means messages might be delivered more than once but never lost, which helps avoid missing important data.

When you want to make sure no messages are lost even if the consumer crashes.

When processing financial transactions where missing a message is worse than processing duplicates.

When collecting logs or metrics where completeness is more important than avoiding duplicates.

When your system can handle duplicate messages safely or has a way to detect duplicates.

When you want reliable message delivery but can accept occasional repeated processing.

Config File - producer.properties

producer.properties

acks=all
retries=3
enable.idempotence=false

This configuration file sets the Kafka producer to wait for all replicas to acknowledge a message (acks=all) to ensure durability.

Retries=3 allows the producer to resend messages if sending fails temporarily.

Idempotence is disabled here to allow at-least-once delivery, which may cause duplicates.

Commands

This command starts a Kafka producer that sends messages to 'example-topic' using the configuration file to ensure at-least-once delivery.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --producer.config producer.properties

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the topic to send messages to.

→

--producer.config - Loads producer settings from the given configuration file.

This command starts a Kafka consumer that reads all messages from the beginning of 'example-topic' to verify message delivery.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning

Expected Output

message1

message2

message3

→

--bootstrap-server - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the topic to read messages from.

→

--from-beginning - Reads all messages from the start of the topic.

Key Concept

If you remember nothing else from this pattern, remember: at-least-once delivery guarantees no message loss but may cause duplicates.

Common Mistakes

Setting 'acks=0' in producer configuration.

This causes the producer to not wait for any acknowledgment, risking message loss.

Set 'acks=all' to ensure the message is fully committed before continuing.

Not enabling retries in the producer.

Without retries, temporary network issues can cause message loss.

Set 'retries' to a positive number like 3 to resend failed messages.

Expecting exactly-once delivery without enabling idempotence.

Without idempotence, duplicates can occur, so exactly-once is not guaranteed.

Accept duplicates or enable idempotence for exactly-once semantics.

Summary

Configure the Kafka producer with 'acks=all' and retries to ensure messages are not lost.

Use the console producer to send messages to a topic with this configuration.

Use the console consumer to read messages from the topic and verify delivery.

At-least-once delivery means messages may be duplicated but never lost.

Practice

1. What does at-least-once delivery guarantee in Kafka?

easy

A. Messages are delivered only once and never retried

B. Messages are delivered exactly once with no duplicates

C. Messages are never lost but might be delivered more than once

D. Messages might be lost but never duplicated

Solution

1. Step 1: Understand delivery guarantees

   At-least-once means Kafka ensures messages reach the consumer at least once, so no loss occurs.

2. Step 2: Consider duplicates possibility

   Because retries can happen, duplicates may occur, so messages might be delivered multiple times.

3. Final Answer:

   Messages are never lost but might be delivered more than once -> Option C

4. Quick Check:

   At-least-once = no loss, possible duplicates [OK]

Hint: At-least-once means no loss, duplicates possible [OK]

Common Mistakes:

• Confusing at-least-once with exactly-once delivery
• Thinking messages can be lost
• Assuming no duplicates occur

2. Which Kafka consumer configuration is essential for implementing at-least-once delivery?

easy

A. enable.auto.commit = false

B. enable.auto.commit = true

C. auto.offset.reset = earliest

D. max.poll.records = 1

Solution

1. Step 1: Identify offset commit behavior

   At-least-once requires manual control of offset commits to avoid losing messages.

2. Step 2: Choose correct config for manual commit

   Setting enable.auto.commit = false disables automatic commits, allowing manual commits after processing.

3. Final Answer:

   enable.auto.commit = false -> Option A

4. Quick Check:

   Manual commit enabled by disabling auto commit [OK]

Hint: Disable auto commit for manual offset control [OK]

Common Mistakes:

• Leaving auto commit enabled causes possible message loss
• Confusing auto.offset.reset with commit control
• Assuming max.poll.records affects delivery guarantee

3. Given this Kafka consumer code snippet, what is the expected behavior regarding message delivery?

consumer.subscribe(['topic'])
for message in consumer:
    process(message)
    consumer.commit()

medium

A. Messages are processed at least once, duplicates possible

B. Messages may be lost if commit happens before processing

C. Messages are processed exactly once with no duplicates

D. Messages are never committed, causing infinite reprocessing

Solution

1. Step 1: Analyze commit timing

   Commit happens after processing each message, so offsets are saved only after successful processing.

2. Step 2: Understand delivery guarantee

   If a failure occurs before commit, message will be reprocessed, causing duplicates but no loss.

3. Final Answer:

   Messages are processed at least once, duplicates possible -> Option A

4. Quick Check:

   Commit after processing = at-least-once delivery [OK]

Hint: Commit after processing ensures no loss, duplicates possible [OK]

Common Mistakes:

• Assuming commit before processing
• Thinking duplicates cannot happen
• Believing messages are lost on failure

4. Identify the bug in this Kafka consumer code that aims for at-least-once delivery:

consumer.subscribe(['topic'])
for message in consumer:
    consumer.commit()
    process(message)

medium

A. Using subscribe instead of assign causes duplicates

B. Committing offset before processing causes message loss on failure

C. Not committing offsets causes infinite reprocessing

D. process() function is missing a return statement

Solution

1. Step 1: Check commit position relative to processing

   Offsets are committed before processing, so if processing fails, message is lost.

2. Step 2: Understand at-least-once requirement

   To guarantee at-least-once, commit must happen after successful processing to avoid loss.

3. Final Answer:

   Committing offset before processing causes message loss on failure -> Option B

4. Quick Check:

   Commit after processing to avoid loss [OK]

Hint: Commit offsets only after processing completes [OK]

Common Mistakes:

• Committing before processing causes data loss
• Ignoring commit timing importance
• Confusing subscribe with assign usage

5. You want to ensure at-least-once delivery in Kafka but avoid processing duplicates as much as possible. Which approach best achieves this?

hard

A. Use exactly-once semantics without any manual offset control

B. Enable auto commit and process messages without tracking offsets

C. Commit offsets before processing to speed up throughput

D. Manually commit offsets after processing and implement idempotent processing logic

Solution

1. Step 1: Combine manual commit with processing logic

   Manual commit after processing ensures no message loss; idempotent logic handles duplicates safely.

2. Step 2: Evaluate other options

   Auto commit risks loss; committing before processing risks loss; exactly-once semantics require special setup.

3. Final Answer:

   Manually commit offsets after processing and implement idempotent processing logic -> Option D

4. Quick Check:

   Manual commit + idempotency = best at-least-once practice [OK]

Hint: Use manual commit plus idempotent processing to reduce duplicates [OK]

Common Mistakes:

• Relying on auto commit for at-least-once
• Committing offsets before processing
• Ignoring idempotent processing importance