At-least-once delivery in Kafka - Deep Dive

Overview - At-least-once delivery

What is it?

At-least-once delivery is a message delivery guarantee in systems like Kafka that ensures every message is delivered to the consumer one or more times. It means no message is lost, but duplicates can happen. This approach prioritizes reliability over avoiding repeated messages.

Why it matters

Without at-least-once delivery, messages could be lost during transmission or processing, causing data gaps or missed events. This can lead to incorrect results or system failures in real-world applications like financial transactions or monitoring. Ensuring messages arrive at least once protects data integrity and system correctness.

Where it fits

Learners should first understand basic messaging concepts and Kafka architecture. After mastering at-least-once delivery, they can explore exactly-once delivery and idempotent processing to handle duplicates and improve efficiency.

Mental Model

Core Idea

At-least-once delivery guarantees that every message reaches the consumer at least once, accepting possible duplicates to avoid loss.

Think of it like...

It's like sending a letter with a request for a delivery receipt and resending it if no receipt arrives, so the recipient gets the letter at least once, even if they get duplicates.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Producer    │──────▶│   Kafka Log   │──────▶│   Consumer    │
└───────────────┘       └───────────────┘       └───────────────┘
       │                      ▲                      │
       │                      │                      │
       └───── retries on send ─┘                      │
                                                      │
                                      ┌───────────────┐
                                      │  Acknowledges │
                                      └───────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding message delivery basics

Concept: Learn what message delivery means in distributed systems and the basic guarantees.

In messaging systems, delivery means sending data from one place (producer) to another (consumer). There are three common guarantees: at-most-once (messages may be lost), at-least-once (messages never lost but may repeat), and exactly-once (messages delivered once, no loss or duplicates).

Result

You understand the basic types of delivery guarantees and their tradeoffs.

Knowing these guarantees helps you choose the right approach for your system's needs.

2

Foundation

Kafka's role in message delivery

Concept: Kafka stores messages durably and allows consumers to read them, enabling delivery guarantees.

Kafka writes messages to a log stored on disk. Producers send messages to Kafka, which stores them safely. Consumers read messages from Kafka at their own pace. Kafka tracks offsets to know which messages have been read.

Result

You see how Kafka's log storage supports reliable message delivery.

Understanding Kafka's log is key to grasping how delivery guarantees work.

3

Intermediate

How at-least-once delivery works in Kafka

🤔Before reading on: do you think Kafka retries sending messages automatically or relies on the producer to retry? Commit to your answer.

Concept: At-least-once delivery happens because producers retry sending messages until Kafka confirms receipt, and consumers may reprocess messages if acknowledgments are delayed.

Producers send messages and wait for Kafka's acknowledgment. If no ack arrives, they resend. Consumers read messages and commit offsets after processing. If a consumer crashes before committing, it may re-read messages, causing duplicates.

Result

Messages are never lost but can be delivered multiple times to consumers.

Knowing retries and offset commits cause duplicates helps design consumer logic to handle them.

4

Intermediate

Consumer offset management and duplicates

🤔Before reading on: do you think committing offsets before or after processing affects duplicates? Commit to your answer.

Concept: When consumers commit offsets after processing, duplicates can occur if they crash before committing. Committing offsets before processing risks message loss.

Consumers typically process a message, then commit the offset. If they crash after processing but before committing, Kafka will resend the message on restart. This causes duplicates but ensures no message is lost.

Result

Offset commit timing directly impacts message duplication and loss risks.

Understanding offset commit timing is crucial to balancing reliability and duplication.

5

Advanced

Handling duplicates with idempotent consumers

🤔Before reading on: do you think consumers can ignore duplicates safely without special logic? Commit to your answer.

Concept: Idempotent consumers process messages so that repeated processing has no harmful effect, solving duplicate issues from at-least-once delivery.

Consumers track processed message IDs or use operations that produce the same result if repeated. For example, updating a database record with the same data multiple times or using unique keys prevents duplicate side effects.

Result

Systems can safely handle duplicates without losing data correctness.

Knowing how to build idempotent consumers is key to practical use of at-least-once delivery.

6

Expert

Tradeoffs and pitfalls of at-least-once delivery

🤔Before reading on: do you think at-least-once delivery always guarantees no data loss without any downsides? Commit to your answer.

Concept: At-least-once delivery avoids data loss but can cause duplicates, requiring extra handling and sometimes increasing latency or complexity.

Retries increase message traffic and processing load. Duplicates can cause errors if consumers are not idempotent. Exactly-once delivery is harder and more resource-intensive but avoids duplicates. Choosing at-least-once is a tradeoff favoring reliability over simplicity.

Result

You understand the practical limits and costs of at-least-once delivery.

Recognizing tradeoffs helps you design systems that balance reliability, performance, and complexity.

Under the Hood

Kafka stores messages in partitions as an ordered log on disk. Producers send messages and wait for acknowledgments based on configured durability levels. If no ack arrives, producers retry sending. Consumers read messages by offset and commit offsets after processing. If a consumer fails before committing, it re-reads messages, causing duplicates. This retry and offset commit mechanism ensures no message is lost but allows duplicates.

Why designed this way?

Kafka was designed for high throughput and durability with simple storage. At-least-once delivery is a natural outcome of its append-only log and offset tracking. Alternatives like exactly-once require complex coordination and state management, which Kafka added later as optional features. The design favors reliability and scalability first.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Producer    │──────▶│   Kafka Log   │──────▶│   Consumer    │
│ (sends data)  │       │ (stores data) │       │ (reads data)  │
└───────┬───────┘       └───────┬───────┘       └───────┬───────┘
        │                       │                       │
        │  retries if no ack     │                       │
        │◀──────────────────────┘                       │
        │                                               │
        │                               commits offset │
        │──────────────────────────────────────────────▶│
        │                                               │
        │                consumer crash/restart        │
        │◀─────────────────────────────────────────────┘
        │                                               │
        │               re-reads messages from offset  │

Myth Busters - 4 Common Misconceptions

Quick: Does at-least-once delivery guarantee no duplicate messages? Commit yes or no.

Common Belief: At-least-once delivery means each message is delivered exactly once, no duplicates.

Reality: At-least-once delivery guarantees messages are delivered one or more times, so duplicates can occur.

Why it matters: Assuming no duplicates leads to bugs if consumers do not handle repeated messages properly.

Quick: Do producers in Kafka automatically retry sending messages forever by default? Commit yes or no.

Common Belief: Kafka producers always retry sending messages automatically until success.

Reality: Kafka producers retry based on configuration; retries can be limited or disabled.

Why it matters: Misconfiguring retries can cause message loss or delays, breaking delivery guarantees.

Quick: If a consumer commits offset before processing, does it prevent duplicates? Commit yes or no.

Common Belief: Committing offsets before processing messages prevents duplicates completely.

Reality: Committing offsets before processing risks message loss if the consumer crashes after commit but before processing.

Why it matters: Incorrect offset commit timing can cause lost messages, violating reliability.

Quick: Is at-least-once delivery always the best choice for all systems? Commit yes or no.

Common Belief: At-least-once delivery is always the safest and best option.

Reality: At-least-once delivery can cause duplicates and overhead; sometimes exactly-once or at-most-once is better.

Why it matters: Choosing the wrong delivery guarantee can cause inefficiency or data errors.

Expert Zone

1

Kafka's at-least-once delivery depends heavily on producer acks and consumer offset commit strategies, which can be tuned for different tradeoffs.

2

Idempotency in consumers is often implemented with external storage or unique keys, but this adds complexity and latency that experts must balance.

3

Kafka's newer exactly-once semantics build on at-least-once delivery but require transactional APIs and careful coordination, which many systems avoid due to complexity.

When NOT to use

At-least-once delivery is not ideal when duplicates cause critical errors or when system simplicity is paramount. In such cases, use exactly-once delivery with Kafka transactions or at-most-once delivery if occasional data loss is acceptable.

Production Patterns

In production, teams combine at-least-once delivery with idempotent consumer logic and monitoring for duplicates. They tune producer retries and consumer commit intervals to balance latency and reliability. Exactly-once delivery is used selectively for critical workflows.

Connections

Database Transactions

Both handle reliability and consistency under failure conditions.

Understanding how databases commit or rollback transactions helps grasp why Kafka consumers must carefully commit offsets to avoid duplicates or loss.

Email Delivery Systems

Both use retries and acknowledgments to ensure messages arrive at least once.

Knowing how email servers resend undelivered mail clarifies why Kafka producers retry sending messages until acknowledged.

Human Communication

At-least-once delivery is like repeating important instructions to ensure understanding, even if repeated.

Recognizing this pattern in everyday life helps appreciate the tradeoff between reliability and repetition in messaging systems.

Common Pitfalls

#1 Committing consumer offsets before processing messages.

Wrong approach:

consumer.commitSync() // called before processing message

Correct approach:

processMessage();
consumer.commitSync(); // commit after processing

Root cause:

Misunderstanding that committing offsets early can cause message loss if the consumer crashes before processing.

#2 Disabling producer retries leading to message loss.

Wrong approach:

props.put("retries", 0); // no retries configured

Correct approach:

props.put("retries", Integer.MAX_VALUE); // enable retries

Root cause:

Not realizing retries are essential to guarantee message delivery in case of transient failures.

#3 Not handling duplicate messages in consumer logic.

Wrong approach:

processMessage(message); // no duplicate check or idempotency

Correct approach:

if (!alreadyProcessed(message.id)) {
processMessage(message);
markProcessed(message.id);
}

Root cause:

Assuming at-least-once means no duplicates, so no duplicate handling is needed.

Key Takeaways

At-least-once delivery ensures no message is lost but allows duplicates, requiring careful consumer design.

Kafka achieves at-least-once delivery through producer retries and consumer offset commits after processing.

Offset commit timing is critical: committing too early risks loss, too late risks duplicates.

Idempotent consumer processing is essential to handle duplicates safely in production.

Choosing delivery guarantees involves tradeoffs between reliability, complexity, and performance.