Message delivery guarantees in HLD - Scalability & System Analysis

Scalability Analysis - Message delivery guarantees

Growth Table: Message Delivery Guarantees

Users / Messages	100 users	10K users	1M users	100M users
Message volume per second	~100 msg/s	~10K msg/s	~1M msg/s	~100M msg/s
System components	Single broker, single DB	Multiple brokers, DB replicas	Partitioned brokers, sharded DB	Global distributed brokers, multi-region DB shards
Delivery guarantee complexity	Simple ack, retry	Idempotency, deduplication	Exactly-once semantics, ordering	Cross-region consistency, geo-replication
Latency	Low, <100ms	Moderate, 100-200ms	Higher, 200-500ms	Variable, 500ms+
Failure handling	Basic retry	Dead-letter queues, monitoring	Advanced retry policies, transactional logs	Multi-region failover, disaster recovery

First Bottleneck

The first bottleneck is the message broker's throughput and storage capacity. At low scale, a single broker handles message delivery with simple acknowledgments. As users grow, the broker's ability to process and persist messages reliably becomes limited. The database or storage system backing the broker also becomes a bottleneck due to write throughput and consistency requirements for delivery guarantees.

Scaling Solutions

• Horizontal scaling: Add more message brokers and partition topics to distribute load.
• Replication: Use broker clusters with replication for fault tolerance and availability.
• Caching: Use in-memory caches for quick message state checks to reduce DB load.
• Sharding: Partition databases by user or topic to scale storage and throughput.
• Idempotency and deduplication: Implement to ensure exactly-once delivery despite retries.
• Dead-letter queues: Handle undeliverable messages separately to avoid blocking.
• Geo-distribution: Deploy brokers and storage in multiple regions for latency and disaster recovery.

Back-of-Envelope Cost Analysis

• At 10K messages/sec, a single broker can handle ~5K-10K msg/sec, so 2 brokers needed.
• Database write throughput must support message persistence; ~10K writes/sec at 10K msg/sec.
• Storage grows with message retention; 10K msg/sec * 86,400 sec/day = ~864M messages/day.
• Network bandwidth: 10K msg/sec * average message size (e.g., 1KB) = ~10MB/s (~80Mbps).
• At 1M msg/sec, partitioning and sharding are mandatory; storage and network scale accordingly.

Interview Tip

Start by clarifying the delivery guarantees needed: at-most-once, at-least-once, or exactly-once. Discuss how these affect system design and complexity. Then, outline scaling steps from single broker to distributed clusters, focusing on bottlenecks and solutions. Use real numbers to justify design choices and show understanding of trade-offs.

Self Check

Your database handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add read replicas and implement caching to reduce load on the primary database. Also, consider sharding the database to distribute writes and scale horizontally.

Key Result

Message delivery systems first hit bottlenecks at the message broker and database throughput. Scaling requires partitioning, replication, and careful handling of delivery semantics to maintain guarantees at high volume.

Practice

1. Which message delivery guarantee ensures that a message is never delivered more than once, but some messages might be lost?

easy

A. Exactly once

B. At most once

C. At least once

D. None of the above

Solution

1. Step 1: Understand 'At most once' guarantee

   This guarantee means messages can be lost but never duplicated.

2. Step 2: Compare with other guarantees

   'At least once' may duplicate messages, 'Exactly once' avoids loss and duplication but is complex.

3. Final Answer:

   At most once -> Option B

4. Quick Check:

   At most once = no duplicates, possible loss [OK]

Hint: No duplicates but possible loss means 'At most once' [OK]

Common Mistakes:

• Confusing 'At most once' with 'At least once'
• Thinking 'Exactly once' is simple
• Assuming 'At most once' never loses messages

2. Which of the following is the correct description of the 'Exactly once' message delivery guarantee?

easy

A. Messages are delivered once without loss or duplication

B. Messages may be lost but never duplicated

C. Messages may be duplicated but never lost

D. Messages are delivered at least once, possibly duplicated

Solution

1. Step 1: Define 'Exactly once' guarantee

   This guarantee ensures each message is delivered once, no loss, no duplication.

2. Step 2: Eliminate other options

   'Messages may be lost but never duplicated' is 'At most once', 'Messages may be duplicated but never lost' and 'delivered at least once, possibly duplicated' describe 'At least once'.

3. Final Answer:

   Messages are delivered once without loss or duplication -> Option A

4. Quick Check:

   Exactly once = no loss, no duplicates [OK]

Hint: Exactly once means no loss and no duplicates [OK]

Common Mistakes:

• Mixing 'Exactly once' with 'At least once'
• Believing 'Exactly once' allows duplicates
• Confusing 'At most once' with 'Exactly once'

3. Consider a message queue system that uses 'At least once' delivery. If a message is sent and the receiver crashes before acknowledging, what is the likely outcome?

medium

A. The message is lost and never delivered

B. The message is delivered once without duplication

C. The message is delivered at most once

D. The message may be delivered multiple times

Solution

1. Step 1: Understand 'At least once' behavior on receiver crash

   If receiver crashes before ack, sender retries, causing possible duplicates.

2. Step 2: Analyze options

   The message is not lost but sender retries on no ack, so may be delivered multiple times to ensure at least once.

3. Final Answer:

   The message may be delivered multiple times -> Option D

4. Quick Check:

   At least once = possible duplicates [OK]

Hint: Receiver crash before ack causes duplicates in 'At least once' [OK]

Common Mistakes:

• Assuming message is lost on crash
• Confusing 'At least once' with 'At most once'
• Ignoring possibility of duplicates

4. A system claims to provide 'Exactly once' delivery but sometimes messages are duplicated. What is the most likely cause?

medium

A. The system uses 'At least once' delivery without deduplication

B. The system uses 'At most once' delivery internally

C. The system does not handle acknowledgments properly

D. The system drops messages on network failure

Solution

1. Step 1: Identify cause of duplicates in 'Exactly once'

   Duplicates usually happen if 'At least once' is used without deduplication.

2. Step 2: Evaluate other options

   'At most once' causes losses not duplicates, improper ack handling or dropping messages cause losses or retries but exactly once needs deduplication atop at least once.

3. Final Answer:

   The system uses 'At least once' delivery without deduplication -> Option A

4. Quick Check:

   Duplicates mean missing deduplication in 'At least once' [OK]

Hint: Duplicates in 'Exactly once' mean missing deduplication [OK]

Common Mistakes:

• Assuming 'At most once' causes duplicates
• Ignoring deduplication step
• Confusing message loss with duplication

5. You are designing a payment processing system that must never lose or duplicate transactions. Which message delivery guarantee should you choose and why?

hard

A. At most once, because it avoids duplicates

B. At least once, because it ensures no message loss

C. Exactly once, because it guarantees no loss and no duplicates

D. None, because message delivery guarantees are not relevant

Solution

1. Step 1: Identify system requirements

   Payment processing must avoid losing or duplicating transactions.

2. Step 2: Match guarantee to requirements

   'Exactly once' ensures messages are delivered once without loss or duplication, fitting the need.

3. Final Answer:

   Exactly once, because it guarantees no loss and no duplicates -> Option C

4. Quick Check:

   Payment systems need exactly once delivery [OK]

Hint: No loss or duplicates means choose 'Exactly once' [OK]

Common Mistakes:

• Choosing 'At most once' and risking loss
• Choosing 'At least once' and risking duplicates
• Ignoring delivery guarantees in critical systems