At-least-once delivery in Kafka - Mini Project: Build & Apply

At-least-once delivery with Kafka

📖 Scenario: You are building a simple message processing system using Kafka. You want to make sure that every message sent to a topic is processed at least once, even if the consumer crashes or restarts.

🎯 Goal: Build a Kafka consumer that processes messages from a topic with at-least-once delivery guarantee by committing offsets after processing each message.

📋 What You'll Learn

Create a Kafka topic named orders with some sample messages.

Configure the consumer to disable auto-commit of offsets.

Consume messages from the orders topic and commit offsets manually after processing each message.

Print each processed message to the console.

💡 Why This Matters

🌍 Real World

At-least-once delivery is important in systems where missing any message could cause data loss or incorrect results, such as order processing or financial transactions.

💼 Career

Kafka is widely used in industry for building reliable data pipelines and event-driven systems. Understanding delivery guarantees is key for roles like backend developer, data engineer, and site reliability engineer.

1

Create the Kafka topic and produce messages

Create a Kafka topic named orders and produce these exact messages to it: order1, order2, order3.

Kafka

# Use Kafka CLI or client to create topic and produce messages
# Your code here

Hint

Use Kafka command line tools to create the topic and send messages exactly named order1, order2, and order3.

2

Configure the Kafka consumer for manual offset commit

Create a Kafka consumer configuration dictionary named consumer_config with bootstrap.servers set to localhost:9092, group.id set to order_group, and enable.auto.commit set to false.

Kafka

# Create topic 'orders' and produce messages

consumer_config = {}
# Your code here

Hint

Set enable.auto.commit to false to disable automatic offset commits.

3

Consume messages and commit offsets manually

Write code to create a Kafka consumer named consumer using consumer_config, subscribe it to the orders topic, then use a for loop to consume messages. For each message, print its value and commit the offset manually using consumer.commit().

Kafka

# Create topic 'orders' and produce messages

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'order_group',
    'enable.auto.commit': false
}

# Create consumer, subscribe to 'orders', consume messages, print and commit offsets
# Your code here

Hint

Use consumer.commit() after processing each message to ensure at-least-once delivery.

4

Print the processed messages

Run the consumer code and print each processed message exactly as received from the orders topic.

Kafka

# Run the consumer code to print messages
# Your code here

Hint

The output should show each message printed on its own line exactly as sent.

Practice

1. What does at-least-once delivery guarantee in Kafka?

easy

A. Messages are delivered only once and never retried

B. Messages are delivered exactly once with no duplicates

C. Messages are never lost but might be delivered more than once

D. Messages might be lost but never duplicated

Solution

1. Step 1: Understand delivery guarantees

   At-least-once means Kafka ensures messages reach the consumer at least once, so no loss occurs.

2. Step 2: Consider duplicates possibility

   Because retries can happen, duplicates may occur, so messages might be delivered multiple times.

3. Final Answer:

   Messages are never lost but might be delivered more than once -> Option C

4. Quick Check:

   At-least-once = no loss, possible duplicates [OK]

Hint: At-least-once means no loss, duplicates possible [OK]

Common Mistakes:

• Confusing at-least-once with exactly-once delivery
• Thinking messages can be lost
• Assuming no duplicates occur

2. Which Kafka consumer configuration is essential for implementing at-least-once delivery?

easy

A. enable.auto.commit = false

B. enable.auto.commit = true

C. auto.offset.reset = earliest

D. max.poll.records = 1

Solution

1. Step 1: Identify offset commit behavior

   At-least-once requires manual control of offset commits to avoid losing messages.

2. Step 2: Choose correct config for manual commit

   Setting enable.auto.commit = false disables automatic commits, allowing manual commits after processing.

3. Final Answer:

   enable.auto.commit = false -> Option A

4. Quick Check:

   Manual commit enabled by disabling auto commit [OK]

Hint: Disable auto commit for manual offset control [OK]

Common Mistakes:

• Leaving auto commit enabled causes possible message loss
• Confusing auto.offset.reset with commit control
• Assuming max.poll.records affects delivery guarantee

3. Given this Kafka consumer code snippet, what is the expected behavior regarding message delivery?

consumer.subscribe(['topic'])
for message in consumer:
    process(message)
    consumer.commit()

medium

A. Messages are processed at least once, duplicates possible

B. Messages may be lost if commit happens before processing

C. Messages are processed exactly once with no duplicates

D. Messages are never committed, causing infinite reprocessing

Solution

1. Step 1: Analyze commit timing

   Commit happens after processing each message, so offsets are saved only after successful processing.

2. Step 2: Understand delivery guarantee

   If a failure occurs before commit, message will be reprocessed, causing duplicates but no loss.

3. Final Answer:

   Messages are processed at least once, duplicates possible -> Option A

4. Quick Check:

   Commit after processing = at-least-once delivery [OK]

Hint: Commit after processing ensures no loss, duplicates possible [OK]

Common Mistakes:

• Assuming commit before processing
• Thinking duplicates cannot happen
• Believing messages are lost on failure

4. Identify the bug in this Kafka consumer code that aims for at-least-once delivery:

consumer.subscribe(['topic'])
for message in consumer:
    consumer.commit()
    process(message)

medium

A. Using subscribe instead of assign causes duplicates

B. Committing offset before processing causes message loss on failure

C. Not committing offsets causes infinite reprocessing

D. process() function is missing a return statement

Solution

1. Step 1: Check commit position relative to processing

   Offsets are committed before processing, so if processing fails, message is lost.

2. Step 2: Understand at-least-once requirement

   To guarantee at-least-once, commit must happen after successful processing to avoid loss.

3. Final Answer:

   Committing offset before processing causes message loss on failure -> Option B

4. Quick Check:

   Commit after processing to avoid loss [OK]

Hint: Commit offsets only after processing completes [OK]

Common Mistakes:

• Committing before processing causes data loss
• Ignoring commit timing importance
• Confusing subscribe with assign usage

5. You want to ensure at-least-once delivery in Kafka but avoid processing duplicates as much as possible. Which approach best achieves this?

hard

A. Use exactly-once semantics without any manual offset control

B. Enable auto commit and process messages without tracking offsets

C. Commit offsets before processing to speed up throughput

D. Manually commit offsets after processing and implement idempotent processing logic

Solution

1. Step 1: Combine manual commit with processing logic

   Manual commit after processing ensures no message loss; idempotent logic handles duplicates safely.

2. Step 2: Evaluate other options

   Auto commit risks loss; committing before processing risks loss; exactly-once semantics require special setup.

3. Final Answer:

   Manually commit offsets after processing and implement idempotent processing logic -> Option D

4. Quick Check:

   Manual commit + idempotency = best at-least-once practice [OK]

Hint: Use manual commit plus idempotent processing to reduce duplicates [OK]

Common Mistakes:

• Relying on auto commit for at-least-once
• Committing offsets before processing
• Ignoring idempotent processing importance