Bird
Raised Fist0

Message Delivery Guarantees - At Most Once, At Least Once, Exactly Once

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Message Delivery Guarantees - At Most Once, At Least Once, Exactly Once
45 min · senior
senior

You are designing the backend message delivery system for a startup with 50 million daily active users (DAU) sending chat messages. The system must support three delivery guarantees: at-most-once, at-least-once, and exactly-once. Messages must be encrypted end-to-end. Offline delivery is required so users receive messages sent while they were offline. The system must maintain a P99 latency under 200ms for message delivery when users are online. The system should scale to handle peak message rates of approximately 1 million messages per second.

Constraints
  • End-to-end encryption mandatory for all messages
  • Offline delivery support with message persistence
  • P99 message delivery latency under 200ms for online users
  • Support three delivery guarantees selectable per message: at-most-once, at-least-once, exactly-once
  • Scale to 50M DAU with peak 1M msg/sec throughput
📋
Guided Steps
1/5
Step 1
Choosing the Messaging Backbone
Which messaging infrastructure will you use to handle high throughput and ordering guarantees for different delivery semantics?
💡 Hint
Consider partitioning by conversation to maintain order and use a distributed log system that supports high throughput and durability.
Key Decision
Use Apache Kafka with conversation_id as the partition key - Kafka’s partitioning ensures ordering per conversation and scales linearly with partitions.
Interviewer Expectation
Mid-level: Names Kafka for messaging. Senior: Explains why conversation_id as partition key ensures ordering and why sender_id partitioning fails due to out-of-order delivery.
⚠️Common Wrong Answer
Using RabbitMQ or Redis Pub/Sub - these systems do not scale linearly to 1M msg/sec and lack strong ordering guarantees at this scale.
Step 2
Message Persistence and Offline Delivery
How will you persist messages to support offline delivery and efficient sync for reconnecting clients?
💡 Hint
Choose a storage system optimized for high write throughput and range queries by conversation for cursor-based sync.
Key Decision
Use Apache Cassandra with partition_key=conversation_id - Cassandra’s LSM-tree design supports high write throughput and efficient range scans for message sync.
Interviewer Expectation
Mid-level: Suggests a NoSQL store. Senior: Explains why Cassandra’s partition key by conversation_id enables O(1) range scans and why relational DBs or MongoDB fail at this scale.
⚠️Common Wrong Answer
Using relational databases for message storage - they cannot handle the write volume and range query latency at 50M DAU.
Step 3
Implementing Delivery Guarantees
How will you implement at-most-once, at-least-once, and exactly-once delivery semantics in the system?
💡 Hint
Consider idempotency keys, message deduplication, and transactional writes combined with acknowledgments from clients.
Key Decision
Use Kafka’s exactly-once semantics with transactional producers and consumers, combined with client ACKs stored in Cassandra to track delivery state and deduplicate messages.
Interviewer Expectation
Mid-level: Mentions retries for at-least-once and no retries for at-most-once. Senior: Details Kafka transactions, idempotent producers, and client-side ACK tracking for exactly-once.
⚠️Common Wrong Answer
Relying solely on client retries without deduplication - leads to duplicate messages and breaks exactly-once guarantees.
Step 4
Ensuring End-to-End Encryption
How will you ensure messages are encrypted end-to-end while still supporting offline delivery and server-side processing?
💡 Hint
Consider client-side encryption with keys managed outside the server and storing encrypted blobs without server access to plaintext.
Key Decision
Implement client-side encryption with keys managed by clients; servers store and forward opaque encrypted blobs without decrypting, enabling offline delivery without compromising privacy.
Interviewer Expectation
Mid-level: Mentions TLS for transport encryption. Senior: Explains client-side encryption prevents server access to plaintext and supports offline delivery by storing encrypted messages.
⚠️Common Wrong Answer
Encrypting only in transit (TLS) but storing plaintext on servers - violates end-to-end encryption and privacy requirements.
Step 5
API Design for Message Flow
Design three core API endpoints for sending messages, syncing missed messages, and acknowledging receipt to support delivery guarantees.
💡 Hint
Consider how clients specify delivery guarantees on send, how they fetch messages after offline periods, and how they confirm receipt for exactly-once.
Key Decision
POST /sendMessage with deliveryGuarantee param; GET /syncMessages with conversation_id and cursor; POST /ackMessage with message_id and delivery status.
Interviewer Expectation
Mid-level: Names endpoints for send and receive. Senior: Details parameters supporting delivery guarantees, cursor-based sync, and ACKs enabling deduplication.
⚠️Common Wrong Answer
Using simple polling endpoints without cursors or ACKs - causes high load and breaks exactly-once semantics.
🔌
API Design Challenge
Sketch 3 core API endpoints for the messaging system that support sending messages, syncing messages for offline users, and acknowledging message receipt.
Hints
  • How does the client send a message and specify delivery guarantee?
  • How does the client sync missed messages after reconnecting?
  • How does the client acknowledge message receipt to enable exactly-once delivery?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with conversation_id partition key for ordered, scalable messaging
  • Use Cassandra with conversation_id partition key for message persistence and offline delivery
  • Implement exactly-once semantics using Kafka transactions and client ACK tracking
  • Design APIs supporting delivery guarantees, cursor-based sync, and acknowledgments
Good to Have - 30%
  • Client-side encryption for end-to-end message confidentiality
  • Idempotency keys and transactional writes to prevent duplicates
  • Explain trade-offs between delivery guarantees and latency
Bonus - 10%
  • Discuss failure modes and recovery for message duplication and loss
  • Explain scaling Kafka partitions and Cassandra clusters for 50M DAU
Anti-patterns to Avoid
Using relational databases for message storage causes write bottlenecks and high latency at scale, making them unsuitable for 50M DAU messaging workloads.
Relying on HTTP polling for message sync at 50M DAU results in 50 million reconnects per minute just to check for messages, which collapses under load and causes latency spikes.
Encrypting only in transit (TLS) but storing plaintext on servers violates end-to-end encryption and compromises user privacy.
✅
Model Answer
▼ Reveal

The system uses Apache Kafka partitioned by conversation_id to ensure ordered, scalable message streaming. Messages are persisted in Apache Cassandra partitioned by conversation_id to support offline delivery and efficient sync. Delivery guarantees are implemented using Kafka's exactly-once semantics with transactional producers and consumers, combined with client ACK tracking stored in Cassandra for deduplication. Client-side encryption ensures end-to-end confidentiality while allowing offline delivery. The API includes endpoints for sending messages with delivery guarantees, syncing messages with cursor-based pagination, and acknowledging message receipt to support exactly-once semantics.

Key Decisions
  • Use Kafka with conversation_id as partition key to maintain message order per conversation and scale to 1M msg/sec
  • Persist messages in Cassandra partitioned by conversation_id for high write throughput and efficient range queries for offline sync
  • Implement delivery guarantees using Kafka's transactional producers and consumers for exactly-once semantics
  • Track client acknowledgments in Cassandra to enable deduplication and support exactly-once delivery
  • Encrypt messages client-side with keys managed by clients to ensure end-to-end encryption without server access to plaintext
  • Design APIs with parameters to specify delivery guarantees, cursor-based sync for efficient offline message retrieval, and ACK endpoints to confirm receipt
  • Apache Kafka with conversation_id as partition key for ordered, scalable message streaming
  • Apache Cassandra with conversation_id partition key for high-throughput message persistence and efficient range queries
  • Kafka exactly-once semantics with transactional producers and consumers for delivery guarantees
  • Client-side end-to-end encryption to protect message privacy while enabling offline delivery
  • Client ACK tracking stored in Cassandra to enable deduplication and exactly-once delivery
  • API endpoints: POST /sendMessage (with delivery guarantee), GET /syncMessages (cursor-based), POST /ackMessage (for ACKs)
  • Use of idempotency keys and transactional writes to prevent duplicates in at-least-once and exactly-once modes
  • TLS for transport security combined with client-side encryption for end-to-end confidentiality

Practice

(1/5)
1. In the timeline generation flow, at step 4 the Delivery Worker reads from Kafka and checks the follower's timeline cache TTL. What happens if the cache TTL has expired for a follower at this step?
easy
A. The Delivery Worker triggers a timeline rebuild for that follower before delivering
B. The Delivery Worker retries reading the cache until TTL refresh completes
C. The Delivery Worker writes the tweet directly to the follower's timeline DB without cache refresh
D. The Delivery Worker skips the follower and marks the message as delivered

Solution

  1. Step 1: Understand Delivery Worker cache TTL check

    If TTL expired, the cached timeline is stale and must be rebuilt to ensure fresh data.

  2. Step 2: Evaluate actions on expired TTL

    Skipping (A) loses data; direct DB write (C) bypasses cache rebuild; retrying cache read (D) causes blocking and delays.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Expired TTL triggers timeline rebuild to maintain freshness [check]
Hint: Expired cache TTL triggers timeline rebuild [check]
Common Mistakes:
2. A distributed cache system needs to minimize data reshuffling when nodes are added or removed. Which hashing strategy best supports this requirement?
easy
A. Use consistent hashing with a ring and virtual nodes
B. Use modular hashing with a fixed number of buckets equal to the number of nodes
C. Use a simple hash mod large prime number unrelated to node count
D. Use random hashing with a centralized directory mapping keys to nodes

Solution

  1. Step 1: Identify modular hashing limitations

    Modular hashing requires rehashing all keys when node count changes, causing massive reshuffling.

  2. Step 2: Understand consistent hashing benefits

    Consistent hashing maps nodes and keys on a ring, minimizing key movement to only neighbors when nodes change.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Consistent hashing minimizes reshuffling on node changes [check]
Hint: Consistent hashing minimizes reshuffle on node changes [check]
Common Mistakes:
3. In a system where a load balancer forwards requests to backend servers, at step 4 the backend server checks its health status before processing the request. What happens if the health check fails at this step?
easy
A. The backend server rejects the request and signals the load balancer to reroute
B. The backend server returns an error response directly to the client
C. The load balancer retries routing the request to a different healthy backend
D. The backend server processes the request anyway, risking errors

Solution

  1. Step 1: Trace backend server health check at Step 4

    If the backend server health check fails, it should not process the request (The backend server processes the request anyway, risking errors is incorrect). Returning an error directly to the client (The backend server returns an error response directly to the client) bypasses the load balancer's retry logic. The load balancer retrying without backend signaling (The load balancer retries routing the request to a different healthy backend) is incomplete.

  2. Step 2: Understand backend rejection and signaling

    The backend server rejects the request and signals the load balancer to reroute traffic to a healthy backend (The backend server rejects the request and signals the load balancer to reroute), ensuring availability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Backend signals load balancer to reroute on health check fail [check]
Hint: Backend signals reroute on health failure [check]
Common Mistakes:
4. If the authentication service in a client-server system becomes unavailable, what is the exact failure mode experienced by clients attempting to log in?
medium
A. Clients receive a generic server error but can retry immediately without data loss
B. Clients are authenticated locally by the server, so login proceeds without interruption
C. Clients' login requests hang until timeout, then receive authentication failure with no retry
D. Clients are redirected to a backup authentication service automatically

Solution

  1. Step 1: Understand authentication dependency

    The server relies on the authentication service for validating credentials; if unavailable, requests cannot be processed.

  2. Step 2: Trace failure impact

    Without authentication service, login requests hang waiting for response until timeout, then fail with no automatic retry or fallback.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Unavailable auth service causes login hang then failure [check]
Hint: No auth service -> login hangs then fails, no fallback [check]
Common Mistakes:
5. At 10x scale, which component of a distributed token bucket rate limiter is most likely to fail first and why?
hard
A. The distributed cache storing token counts, because it cannot handle the increased read/write throughput
B. The centralized token refill scheduler, because it becomes a single point of contention and latency bottleneck
C. The client-side token bucket implementation, because local memory limits token storage capacity
D. The network between API gateways and the token store, because increased traffic causes packet loss and delays

Solution

  1. Step 1: Identify centralized token refill scheduler role

    It coordinates token replenishment globally, requiring strong consistency and frequent updates.

  2. Step 2: Analyze scaling impact

    At 10x scale, centralized scheduler becomes a contention bottleneck and latency hotspot, limiting throughput.

  3. Step 3: Compare other components

    Distributed cache scales better with sharding; client-side buckets are local and scale naturally; network issues are secondary to scheduler bottleneck.

  4. Final Answer:

    Option B -> Option B

  5. Quick Check:

    Centralized scheduler bottlenecks first at scale [check]
Hint: Centralized scheduler bottlenecks first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard