At-most-once delivery in Kafka - Mini Project: Build & Apply

At-most-once delivery with Kafka

📖 Scenario: You are building a simple message producer and consumer using Kafka. Your goal is to ensure messages are delivered at most once, meaning no message is processed more than once, but some messages might be lost if failures occur.

🎯 Goal: Create a Kafka producer and consumer setup that demonstrates at-most-once delivery by disabling retries and committing offsets before processing messages.

📋 What You'll Learn

Create a Kafka producer that sends three specific messages to a topic named orders.

Create a configuration variable enable_auto_commit set to True to enable automatic offset commits.

Create a Kafka consumer that consumes messages from the orders topic with enable_auto_commit enabled.

Print each consumed message's value to show the output.

💡 Why This Matters

🌍 Real World

At-most-once delivery is useful when you want fast processing and can tolerate some message loss, such as logging or metrics collection.

💼 Career

Understanding Kafka delivery guarantees helps you design reliable data pipelines and streaming applications in real-world jobs.

1

Create Kafka producer with messages

Create a Kafka producer named producer that sends these exact messages to the topic orders: 'order1', 'order2', and 'order3'. Use the send method for each message.

Kafka

# Create a Kafka producer named producer
# Send 'order1', 'order2', and 'order3' to topic 'orders'
# Your code here

Need a hint?

Use KafkaProducer from the kafka library and call send with the topic and message bytes.

2

Set consumer configuration for auto commit

Create a variable called enable_auto_commit and set it to True to enable automatic offset commits for the consumer.

Kafka

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('orders', b'order1')
producer.send('orders', b'order2')
producer.send('orders', b'order3')
producer.flush()

# Create enable_auto_commit variable and set to True
# Your code here

Need a hint?

Just create a variable named enable_auto_commit and assign True.

3

Create Kafka consumer with auto commit enabled

Create a Kafka consumer named consumer that subscribes to the orders topic. Use enable_auto_commit to set the enable_auto_commit parameter in the consumer configuration.

Kafka

from kafka import KafkaProducer, KafkaConsumer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('orders', b'order1')
producer.send('orders', b'order2')
producer.send('orders', b'order3')
producer.flush()

enable_auto_commit = True

# Create Kafka consumer named consumer with enable_auto_commit set
# Your code here

Need a hint?

Use KafkaConsumer with enable_auto_commit=enable_auto_commit and subscribe to 'orders'.

4

Print consumed messages

Use a for loop to iterate over consumer and print the value of each message decoded as UTF-8.

Kafka

from kafka import KafkaProducer, KafkaConsumer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('orders', b'order1')
producer.send('orders', b'order2')
producer.send('orders', b'order3')
producer.flush()

enable_auto_commit = True

consumer = KafkaConsumer(
    'orders',
    bootstrap_servers='localhost:9092',
    enable_auto_commit=enable_auto_commit,
    auto_offset_reset='earliest'
)

# Print each message value decoded as UTF-8
# Your code here

Need a hint?

Use a for loop over consumer and print message.value.decode('utf-8').