At-least-once delivery in Kafka - Mini Project: Build & Apply

At-least-once delivery with Kafka

📖 Scenario: You are building a simple message processing system using Kafka. You want to make sure that every message sent to a topic is processed at least once, even if the consumer crashes or restarts.

🎯 Goal: Build a Kafka consumer that processes messages from a topic with at-least-once delivery guarantee by committing offsets after processing each message.

📋 What You'll Learn

Create a Kafka topic named orders with some sample messages.

Configure the consumer to disable auto-commit of offsets.

Consume messages from the orders topic and commit offsets manually after processing each message.

Print each processed message to the console.

💡 Why This Matters

🌍 Real World

At-least-once delivery is important in systems where missing any message could cause data loss or incorrect results, such as order processing or financial transactions.

💼 Career

Kafka is widely used in industry for building reliable data pipelines and event-driven systems. Understanding delivery guarantees is key for roles like backend developer, data engineer, and site reliability engineer.

1

Create the Kafka topic and produce messages

Create a Kafka topic named orders and produce these exact messages to it: order1, order2, order3.

Kafka

# Use Kafka CLI or client to create topic and produce messages
# Your code here

Need a hint?

Use Kafka command line tools to create the topic and send messages exactly named order1, order2, and order3.

2

Configure the Kafka consumer for manual offset commit

Create a Kafka consumer configuration dictionary named consumer_config with bootstrap.servers set to localhost:9092, group.id set to order_group, and enable.auto.commit set to false.

Kafka

# Create topic 'orders' and produce messages

consumer_config = {}
# Your code here

Need a hint?

Set enable.auto.commit to false to disable automatic offset commits.

3

Consume messages and commit offsets manually

Write code to create a Kafka consumer named consumer using consumer_config, subscribe it to the orders topic, then use a for loop to consume messages. For each message, print its value and commit the offset manually using consumer.commit().

Kafka

# Create topic 'orders' and produce messages

consumer_config = {
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'order_group',
    'enable.auto.commit': false
}

# Create consumer, subscribe to 'orders', consume messages, print and commit offsets
# Your code here

Need a hint?

Use consumer.commit() after processing each message to ensure at-least-once delivery.

4

Print the processed messages

Run the consumer code and print each processed message exactly as received from the orders topic.

Kafka

# Run the consumer code to print messages
# Your code here

Need a hint?

The output should show each message printed on its own line exactly as sent.