At-most-once delivery in Kafka - Mini Project: Build & Apply

At-most-once delivery with Kafka

📖 Scenario: You are building a simple message producer and consumer using Kafka. Your goal is to ensure messages are delivered at most once, meaning no message is processed more than once, but some messages might be lost if failures occur.

🎯 Goal: Create a Kafka producer and consumer setup that demonstrates at-most-once delivery by disabling retries and committing offsets before processing messages.

📋 What You'll Learn

Create a Kafka producer that sends three specific messages to a topic named orders.

Create a configuration variable enable_auto_commit set to True to enable automatic offset commits.

Create a Kafka consumer that consumes messages from the orders topic with enable_auto_commit enabled.

Print each consumed message's value to show the output.

💡 Why This Matters

🌍 Real World

At-most-once delivery is useful when you want fast processing and can tolerate some message loss, such as logging or metrics collection.

💼 Career

Understanding Kafka delivery guarantees helps you design reliable data pipelines and streaming applications in real-world jobs.

1

Create Kafka producer with messages

Create a Kafka producer named producer that sends these exact messages to the topic orders: 'order1', 'order2', and 'order3'. Use the send method for each message.

Kafka

# Create a Kafka producer named producer
# Send 'order1', 'order2', and 'order3' to topic 'orders'
# Your code here

Hint

Use KafkaProducer from the kafka library and call send with the topic and message bytes.

2

Set consumer configuration for auto commit

Create a variable called enable_auto_commit and set it to True to enable automatic offset commits for the consumer.

Kafka

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('orders', b'order1')
producer.send('orders', b'order2')
producer.send('orders', b'order3')
producer.flush()

# Create enable_auto_commit variable and set to True
# Your code here

Hint

Just create a variable named enable_auto_commit and assign True.

3

Create Kafka consumer with auto commit enabled

Create a Kafka consumer named consumer that subscribes to the orders topic. Use enable_auto_commit to set the enable_auto_commit parameter in the consumer configuration.

Kafka

from kafka import KafkaProducer, KafkaConsumer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('orders', b'order1')
producer.send('orders', b'order2')
producer.send('orders', b'order3')
producer.flush()

enable_auto_commit = True

# Create Kafka consumer named consumer with enable_auto_commit set
# Your code here

Hint

Use KafkaConsumer with enable_auto_commit=enable_auto_commit and subscribe to 'orders'.

4

Print consumed messages

Use a for loop to iterate over consumer and print the value of each message decoded as UTF-8.

Kafka

from kafka import KafkaProducer, KafkaConsumer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('orders', b'order1')
producer.send('orders', b'order2')
producer.send('orders', b'order3')
producer.flush()

enable_auto_commit = True

consumer = KafkaConsumer(
    'orders',
    bootstrap_servers='localhost:9092',
    enable_auto_commit=enable_auto_commit,
    auto_offset_reset='earliest'
)

# Print each message value decoded as UTF-8
# Your code here

Hint

Use a for loop over consumer and print message.value.decode('utf-8').

Practice

1.

What does at-most-once delivery guarantee in Kafka messaging?

easy

A. Messages are sent at least once, possibly duplicated

B. Messages are sent exactly once with no loss

C. Messages are sent only once but some may be lost

D. Messages are sent multiple times intentionally

Solution

1. Step 1: Understand at-most-once delivery meaning

   At-most-once delivery means messages are sent once without retries, so duplicates are avoided but some messages might be lost.

2. Step 2: Compare with other delivery guarantees

   At-least-once allows duplicates, exactly-once avoids loss and duplicates, so only at-most-once fits the description of possible loss but no duplicates.

3. Final Answer:

   Messages are sent only once but some may be lost -> Option C

4. Quick Check:

   At-most-once = no duplicates, possible loss [OK]

Hint: At-most-once means no retries, so possible message loss [OK]

Common Mistakes:

• Confusing at-most-once with exactly-once delivery
• Assuming no messages are ever lost
• Thinking duplicates can occur in at-most-once

2.

Which Kafka producer configuration enables at-most-once delivery?

producer = KafkaProducer(acks=?, retries=? )

easy

A. acks=all, retries=5

B. acks=0, retries=0

C. acks=1, retries=3

D. acks=all, retries=0

Solution

1. Step 1: Identify settings for at-most-once

   At-most-once requires no retries and no acknowledgments, so acks=0 and retries=0 are needed.

2. Step 2: Check other options

   Other options enable retries or acknowledgments, which allow retries and possible duplicates, not at-most-once.

3. Final Answer:

   acks=0, retries=0 -> Option B

4. Quick Check:

   acks=0 & retries=0 = at-most-once [OK]

Hint: Set acks and retries to zero for at-most-once [OK]

Common Mistakes:

• Setting retries > 0 allows retries, not at-most-once
• Using acks=all causes waiting for confirmation
• Confusing acks=1 with acks=0

3.

Given the following Kafka producer code snippet, what is the expected delivery guarantee?

producer = KafkaProducer(acks=0, retries=0)
producer.send('topic', b'message')

medium

A. At-most-once delivery with possible message loss

B. No message will be sent

C. Exactly-once delivery with no duplicates

D. At-least-once delivery with possible duplicates

Solution

1. Step 1: Analyze producer settings

   With acks=0 and retries=0, the producer does not wait for acknowledgment and does not retry on failure.

2. Step 2: Determine delivery guarantee

   This means messages may be lost if sending fails, but duplicates won't occur, matching at-most-once delivery.

3. Final Answer:

   At-most-once delivery with possible message loss -> Option A

4. Quick Check:

   acks=0 & retries=0 = at-most-once [OK]

Hint: acks=0 and retries=0 means no retries, so at-most-once [OK]

Common Mistakes:

• Assuming messages are always delivered
• Confusing at-most-once with exactly-once
• Thinking duplicates can happen here

4.

Identify the error in this Kafka producer configuration for at-most-once delivery:

producer = KafkaProducer(acks=1, retries=0)
producer.send('topic', b'message')

medium

A. acks=1 causes waiting for leader ack, not at-most-once

B. retries=0 causes infinite retries

C. Missing enable.idempotence=True

D. Topic name must be uppercase

Solution

1. Step 1: Check acks setting

   acks=1 means the producer waits for leader acknowledgment, allowing retries and possible duplicates, not at-most-once.

2. Step 2: Verify retries and other settings

   Retries=0 disables retries, but acks=1 still waits for confirmation, so this is not pure at-most-once.

3. Final Answer:

   acks=1 causes waiting for leader ack, not at-most-once -> Option A

4. Quick Check:

   acks must be 0 for at-most-once [OK]

Hint: acks must be 0 for at-most-once delivery [OK]

Common Mistakes:

• Thinking retries=0 means no retries but ignoring acks effect
• Assuming idempotence is required for at-most-once
• Believing topic names must be uppercase

5.

You want to configure a Kafka producer for at-most-once delivery but also need to minimize message loss. Which combination of settings is best?

Options:
A) acks=1, retries=0
B) acks=0, retries=0
C) acks=0, retries=3
D) acks=all, retries=0

hard

A. acks=all, retries=0 (wait for all acks, no retries)

B. acks=0, retries=0 (pure at-most-once, possible loss)

C. acks=0, retries=3 (retries with no ack, risky duplicates)

D. acks=1, retries=0 (wait for ack, no retries, less loss)

Solution

1. Step 1: Understand trade-offs

   Pure at-most-once (acks=0, retries=0) risks message loss. Waiting for ack reduces loss but may not be pure at-most-once.

2. Step 2: Evaluate options

   acks=1, retries=0 (wait for ack, no retries, less loss) waits for leader ack but disables retries, reducing loss while avoiding duplicates, closest to at-most-once with less loss.

3. Final Answer:

   acks=1, retries=0 (wait for ack, no retries, less loss) -> Option D

4. Quick Check:

   acks=1 & retries=0 balances loss and duplicates [OK]

Hint: Wait for ack but disable retries to reduce loss in at-most-once [OK]

Common Mistakes:

• Choosing retries > 0 breaks at-most-once
• Ignoring ack setting impact on loss
• Assuming acks=all with no retries is at-most-once