Bird
Raised Fist0

Message Delivery Guarantees - At Most Once, At Least Once, Exactly Once

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Message Delivery Guarantees - At Most Once, At Least Once, Exactly Once
Designing message delivery guarantees with trade-offs between latency, duplication, and consistency
mediumMessage Queues and Event StreamingFreq: common
⚙️
Functional Requirements
  • Support at-most-once delivery: messages delivered zero or one time, no duplicates
  • Support at-least-once delivery: messages delivered one or more times, duplicates possible
  • Support exactly-once delivery: messages delivered exactly once, no duplicates
  • Allow clients to choose delivery guarantee per message or topic
  • Provide message ordering within partitions or topics
  • Expose APIs for producing and consuming messages with delivery semantics
📊
Non-Functional Requirements
  • Handle 2 billion daily active users (DAU)
  • Support peak write throughput of 925,000 messages per second
  • Ensure 99.99% availability for message delivery APIs
  • Achieve 100ms P99 end-to-end latency for message delivery
  • Retain messages for 7 days with replication for durability
❓
Clarifying Questions to Ask
  • What is the expected average message size?
  • Are messages ordered globally or only within partitions?
  • Is message duplication acceptable for at-least-once delivery?
  • What is the maximum acceptable latency for message delivery?
  • Should the system support transactional message processing?
  • What failure scenarios must the system tolerate (e.g., broker crashes)?
Out of Scope
✗ Message content encryption and end-to-end security
✗ Complex message transformations or filtering
✗ Multi-region geo-replication consistency models
✗ User authentication and authorization mechanisms
📌
Assumptions
  • 2 billion DAU
  • 40 messages per user per day
  • Average message size 100 bytes
  • Message retention period 7 days
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 msg/sec
peak write qpsAssuming 2× peak factor → 1,851,852 msg/sec
read qpsAssuming 1.5× read to write ratio → 1,388,889 msg/sec
💾
Storage Estimate
daily storage gb(925,926 msg/sec × 100 bytes × 86,400 sec) / (1024³) ≈ 7.46 TB/day
retention storage gb7.46 TB/day × 7 days ≈ 52.2 TB
with replication gb52.2 TB × 3 replicas ≈ 156.6 TB
🖥️
Server Estimate
broker nodesAssuming Kafka broker handles 20,000 msg/sec → 1,851,852 / 20,000 ≈ 93 brokers
storage nodesAssuming Cassandra node stores 2 TB data → 156.6 TB / 2 ≈ 79 nodes
🏗️
Architecture Diagram
Architecture Diagram
          +-------------+
          |  Producers  |
          +------+------+
                 |
                 v
          +------+-------+
          |   Kafka      |
          |  Brokers     |
          +------+-------+
                 |
                 v
          +------+-------+
          |  Delivery    |
          |  Workers     |
          +------+-------+
                 |
                 v
          +------+-------+
          |  Cassandra   |
          |  Storage     |
          +------+-------+
                 |
                 v
          +------+-------+
          |  Consumers   |
          +--------------+
🧩
Components
Kafka Brokers
Durable, partitioned message queue supporting high throughput and ordering
Why thisKafka supports partitioned logs with configurable delivery semantics and scales to millions of msgs/sec
Why notRabbitMQ lacks partitioning and throughput at 925K msg/sec; Redis Streams lacks durability guarantees
Delivery Workers
Implement delivery guarantees logic (at-most-once, at-least-once, exactly-once) and message deduplication
Why thisStateless workers can scale horizontally and coordinate with Kafka offsets and Cassandra for idempotency
Why notEmbedding logic in Kafka Connectors limits flexibility; using only Kafka transactions is insufficient for exactly-once at scale
Cassandra Storage
Store message metadata and deduplication state with high write throughput and linear scalability
Why thisCassandra's wide-column model and tunable consistency enable fast writes and idempotency checks at scale
Why notPostgreSQL requires manual sharding and cannot sustain 925K writes/sec; HBase has higher operational complexity
Producers
Clients that publish messages with specified delivery guarantees and partition keys
Why thisProducers embed delivery semantics in message headers enabling downstream processing
Why notSimple fire-and-forget clients cannot support exactly-once semantics without coordination
Consumers
Clients that consume messages and acknowledge processing to enable offset commits
Why thisConsumers coordinate with delivery workers to ensure correct offset commits per delivery guarantee
Why notStateless consumers without offset tracking cause duplicate or lost messages
🔄
Data Flow
1
Producer → Kafka Broker
Publish message with delivery guarantee metadata
Kafka partitions message by key, persists to log
2
Kafka Broker → Delivery Worker
Consume message from partition log
Delivery worker reads messages in order for processing
3
Delivery Worker → Cassandra Storage
Check deduplication state and write message metadata
Ensures idempotency for at-least-once and exactly-once
4
Delivery Worker → Consumer
Deliver message according to guarantee semantics
At-most-once: deliver once; At-least-once: retry on failure; Exactly-once: deduplicate
5
Consumer → Delivery Worker
Acknowledge message processing
Triggers offset commit and state update
6
Delivery Worker → Kafka Broker
Commit consumer offset
Ensures no re-delivery beyond guarantee

Practice

(1/5)
1. A web crawler must avoid overloading a single domain by spacing requests appropriately. Which component is primarily responsible for enforcing this constraint?
easy
A. Politeness Policy Manager, which schedules requests respecting domain-specific delays
B. Scheduler, which balances load across multiple crawler instances
C. Duplicate Detection module, which filters repeated URLs to reduce load
D. URL Frontier, which stores URLs to be crawled next without timing control

Solution

  1. Step 1: Identify the role of Politeness Policy Manager

    This component enforces delays between requests to the same domain to prevent overloading servers.

  2. Step 2: Eliminate other options

    Duplicate Detection filters URLs but does not control timing; Scheduler balances load but does not enforce domain-specific delays; URL Frontier queues URLs but lacks timing logic.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Politeness Policy controls request timing per domain [check]
Hint: Politeness Policy controls request timing per domain [OK]
Common Mistakes:
2. Without the seat lock expiration mechanism, what exact failure occurs during concurrent bookings?
medium
A. Users receive immediate errors due to deadlock detection in the database
B. Seats remain locked indefinitely, causing availability to drop and bookings to stall
C. Payment processing fails because seat status is inconsistent across services
D. The system automatically cancels bookings after a fixed timeout regardless of payment

Solution

  1. Step 1: Understand role of seat lock expiration

    Expiration releases seats if payment or booking is abandoned, preventing indefinite holds.

  2. Step 2: Trace failure without expiration

    Locks never expire, so seats remain reserved even if users abandon payment, reducing available seats and causing booking stalls.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    No TTL -> stale locks -> seats unavailable indefinitely [check]
Hint: No expiration -> stale locks -> seats blocked forever [OK]
Common Mistakes:
3. If the Duplicate Detection component fails silently and stops filtering URLs, what is the most immediate failure observed in the crawler system?
medium
A. The URL Frontier crashes because it cannot handle duplicate URLs
B. The Politeness Policy Manager blocks all requests due to unexpected URL patterns
C. The Scheduler rejects all URLs due to overload and stops scheduling new crawls
D. The crawler repeatedly fetches the same URLs, wasting bandwidth and storage

Solution

  1. Step 1: Trace failure impact of Duplicate Detection silent failure

    Without filtering, duplicate URLs flood the system causing repeated fetches.

  2. Step 2: Eliminate other options

    Scheduler overload is secondary; Politeness Policy unaffected by duplicates; URL Frontier typically handles duplicates gracefully.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Duplicate Detection failure causes repeated URL fetches [check]
Hint: No duplicate filtering -> repeated URL fetches waste resources [OK]
Common Mistakes:
4. At 10x scale, which component is most likely to become the first bottleneck in the stock exchange system, and why?
hard
A. The network layer, because TCP connections for order submission exceed server socket limits
B. The Market Data Feed, because broadcasting to millions of subscribers saturates network bandwidth
C. The Order Book storage, because the relational database cannot handle the write throughput
D. The Matching Engine, because partitioned order books require cross-partition coordination for large orders

Solution

  1. Step 1: Analyze scaling challenges

    Matching Engine partitions order books per stock symbol, but large cross-symbol or large orders require coordination, causing latency and bottlenecks at scale.

  2. Step 2: Evaluate other components

    Market Data Feed scales with multicast or sharding; Order Book storage uses scalable Kafka or NoSQL; network layer scales with load balancers and horizontal scaling.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Cross-partition coordination -> Matching Engine bottleneck at scale [check]
Hint: Cross-partition coordination bottlenecks matching engine [check]
Common Mistakes:
5. At 10x traffic scale, which component in a GeoDNS-based global traffic distribution system is most likely to fail first and why?
hard
A. Authoritative DNS servers, because they maintain stateful session info for each client
B. GeoDNS servers, because DNS query rate increases and caching TTLs limit cache effectiveness
C. Data center load balancers, because they cannot handle increased TCP connection counts
D. Network routers, because they cannot route Anycast IPs at scale

Solution

  1. Step 1: Identify GeoDNS scaling bottleneck

    GeoDNS servers handle DNS queries globally; at 10x scale, query volume spikes, and short TTLs reduce cache hits, increasing load.

  2. Step 2: Eliminate other options

    Authoritative DNS servers (A) are stateless per query. Data center load balancers (C) scale horizontally and are not the first bottleneck. Network routers (D) handle Anycast well at scale.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    GeoDNS query load spikes with limited caching cause first failure [check]
Hint: GeoDNS query load spikes first due to limited caching [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard