Message delivery guarantees in HLD - System Design Guide

Problem Statement

When messages are sent between services or components, they can be lost, duplicated, or delivered out of order. This causes failures like missing transactions, repeated actions, or inconsistent data states, which break user trust and system correctness.

Solution

Message delivery guarantees define how a system ensures messages reach their destination reliably. They use acknowledgments, retries, and ordering controls to prevent loss, duplication, or disorder. Different levels of guarantees balance reliability and performance based on system needs.

Architecture

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Producer    │──────▶│ Message Queue │──────▶│   Consumer    │
└───────────────┘       └───────────────┘       └───────────────┘
       │                      ▲   │                    │
       │                      │   │                    │
       │                      │   └───── Acknowledgment ─┘
       └───── Send Message ───┘

This diagram shows a producer sending messages to a message queue, which then delivers them to a consumer. The consumer sends acknowledgments back to confirm receipt, enabling delivery guarantees.

Trade-offs

✓ Pros

→

Prevents message loss, ensuring critical data is not missed.

→

Avoids duplicate processing by detecting repeated messages.

→

Maintains message order when required for consistency.

→

Improves system reliability and user trust.

✗ Cons

→

Higher delivery guarantees add latency due to acknowledgments and retries.

→

Complexity increases with mechanisms like deduplication and ordering.

→

Resource usage grows with message tracking and storage.

Use strong delivery guarantees when data correctness is critical, such as financial transactions or inventory updates, especially at scales above thousands of messages per second.

Avoid strict guarantees in low-scale or latency-sensitive systems where occasional message loss or duplication is acceptable, such as real-time analytics or logs.

Real World Examples

Amazon

Amazon SQS provides at-least-once delivery with deduplication options to ensure order processing messages are reliably handled without loss.

Netflix

Netflix uses Kafka with exactly-once semantics to guarantee event delivery for billing and user activity tracking, preventing duplicates and data loss.

Uber

Uber employs at-least-once delivery in their dispatch system to ensure ride requests are never lost, retrying message delivery until confirmed.

Alternatives

At-most-once delivery

Messages are sent once without retries or acknowledgments, risking loss but minimizing latency.

Use when: Choose when low latency is critical and occasional message loss is acceptable, such as telemetry data.

At-least-once delivery

Messages are retried until acknowledged, ensuring no loss but allowing duplicates.

Use when: Choose when message loss is unacceptable but duplicates can be handled, like order processing.

Exactly-once delivery

Combines retries and deduplication to ensure each message is processed once and only once.

Use when: Choose when both loss and duplicates are unacceptable, such as financial transactions.

Summary

Message delivery guarantees prevent loss, duplication, and disorder of messages between components.

Different levels like at-most-once, at-least-once, and exactly-once balance reliability and performance.

Choosing the right guarantee depends on system criticality, scale, and tolerance for duplicates or loss.

Practice

1. Which message delivery guarantee ensures that a message is never delivered more than once, but some messages might be lost?

easy

A. Exactly once

B. At most once

C. At least once

D. None of the above

Solution

1. Step 1: Understand 'At most once' guarantee

   This guarantee means messages can be lost but never duplicated.

2. Step 2: Compare with other guarantees

   'At least once' may duplicate messages, 'Exactly once' avoids loss and duplication but is complex.

3. Final Answer:

   At most once -> Option B

4. Quick Check:

   At most once = no duplicates, possible loss [OK]

Hint: No duplicates but possible loss means 'At most once' [OK]

Common Mistakes:

• Confusing 'At most once' with 'At least once'
• Thinking 'Exactly once' is simple
• Assuming 'At most once' never loses messages

2. Which of the following is the correct description of the 'Exactly once' message delivery guarantee?

easy

A. Messages are delivered once without loss or duplication

B. Messages may be lost but never duplicated

C. Messages may be duplicated but never lost

D. Messages are delivered at least once, possibly duplicated

Solution

1. Step 1: Define 'Exactly once' guarantee

   This guarantee ensures each message is delivered once, no loss, no duplication.

2. Step 2: Eliminate other options

   'Messages may be lost but never duplicated' is 'At most once', 'Messages may be duplicated but never lost' and 'delivered at least once, possibly duplicated' describe 'At least once'.

3. Final Answer:

   Messages are delivered once without loss or duplication -> Option A

4. Quick Check:

   Exactly once = no loss, no duplicates [OK]

Hint: Exactly once means no loss and no duplicates [OK]

Common Mistakes:

• Mixing 'Exactly once' with 'At least once'
• Believing 'Exactly once' allows duplicates
• Confusing 'At most once' with 'Exactly once'

3. Consider a message queue system that uses 'At least once' delivery. If a message is sent and the receiver crashes before acknowledging, what is the likely outcome?

medium

A. The message is lost and never delivered

B. The message is delivered once without duplication

C. The message is delivered at most once

D. The message may be delivered multiple times

Solution

1. Step 1: Understand 'At least once' behavior on receiver crash

   If receiver crashes before ack, sender retries, causing possible duplicates.

2. Step 2: Analyze options

   The message is not lost but sender retries on no ack, so may be delivered multiple times to ensure at least once.

3. Final Answer:

   The message may be delivered multiple times -> Option D

4. Quick Check:

   At least once = possible duplicates [OK]

Hint: Receiver crash before ack causes duplicates in 'At least once' [OK]

Common Mistakes:

• Assuming message is lost on crash
• Confusing 'At least once' with 'At most once'
• Ignoring possibility of duplicates

4. A system claims to provide 'Exactly once' delivery but sometimes messages are duplicated. What is the most likely cause?

medium

A. The system uses 'At least once' delivery without deduplication

B. The system uses 'At most once' delivery internally

C. The system does not handle acknowledgments properly

D. The system drops messages on network failure

Solution

1. Step 1: Identify cause of duplicates in 'Exactly once'

   Duplicates usually happen if 'At least once' is used without deduplication.

2. Step 2: Evaluate other options

   'At most once' causes losses not duplicates, improper ack handling or dropping messages cause losses or retries but exactly once needs deduplication atop at least once.

3. Final Answer:

   The system uses 'At least once' delivery without deduplication -> Option A

4. Quick Check:

   Duplicates mean missing deduplication in 'At least once' [OK]

Hint: Duplicates in 'Exactly once' mean missing deduplication [OK]

Common Mistakes:

• Assuming 'At most once' causes duplicates
• Ignoring deduplication step
• Confusing message loss with duplication

5. You are designing a payment processing system that must never lose or duplicate transactions. Which message delivery guarantee should you choose and why?

hard

A. At most once, because it avoids duplicates

B. At least once, because it ensures no message loss

C. Exactly once, because it guarantees no loss and no duplicates

D. None, because message delivery guarantees are not relevant

Solution

1. Step 1: Identify system requirements

   Payment processing must avoid losing or duplicating transactions.

2. Step 2: Match guarantee to requirements

   'Exactly once' ensures messages are delivered once without loss or duplication, fitting the need.

3. Final Answer:

   Exactly once, because it guarantees no loss and no duplicates -> Option C

4. Quick Check:

   Payment systems need exactly once delivery [OK]

Hint: No loss or duplicates means choose 'Exactly once' [OK]

Common Mistakes:

• Choosing 'At most once' and risking loss
• Choosing 'At least once' and risking duplicates
• Ignoring delivery guarantees in critical systems