Bird
Raised Fist0

Message Delivery Guarantees - At Most Once, At Least Once, Exactly Once

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/7
Producers

Send message

Producer sends message to Kafka Brokers

Protocol:Kafka Produce
BeforeMessage exists only on Producer device
AfterMessage appended to Kafka topic partition
Kafka Brokers

Store message

Kafka brokers persist message with offset

Protocol:Kafka internal
BeforeMessage received from Producer but not stored
AfterMessage durably stored with offset assigned
Delivery Worker

Consume message

Delivery Worker reads message from Kafka topic

Protocol:Kafka Consume
BeforeMessage stored in Kafka but not processed
AfterMessage loaded into Delivery Worker for processing
Delivery Worker

Write delivery state

Delivery Worker writes message delivery metadata to Cassandra

Protocol:Cassandra Write
BeforeDelivery Worker has message but no delivery record
AfterDelivery state recorded in Cassandra for tracking
Delivery Worker

Read delivery state

Delivery Worker reads delivery metadata from Cassandra before delivery

Protocol:Cassandra Read
BeforeDelivery state recorded in Cassandra for tracking
AfterDelivery Worker confirms delivery status from Cassandra
Delivery Worker

Deliver message

Delivery Worker sends message to Consumer

Protocol:Push Notification / WebSocket
BeforeMessage delivery state recorded but not delivered
AfterConsumer receives message; delivery acknowledged or retried
Consumers

Acknowledge message

Consumer sends ack back to Delivery Worker

Protocol:Ack over WebSocket
BeforeMessage received but not acknowledged
AfterDelivery Worker marks message as delivered in Cassandra
🏗️
Architecture Diagram
Architecture Diagram
┌─────────────┐
│  Producers  │
└──────┬──────┘
       │ Kafka Produce
       ▼
┌─────────────┐
│Kafka Brokers│
└──────┬──────┘
       │ Kafka Consume
       ▼
┌───────────────┐
│Delivery Worker│
└──────┬────────┘
       │ Write
       ▼
┌───────────────┐
│Cassandra Store│
└──────┬────────┘
       │ Read
       ▼
┌─────────────┐
│  Consumers  │
└─────────────┘
Design Rationale

This architecture illustrates message delivery guarantees using Kafka as a durable, ordered queue. Producers send messages to Kafka Brokers, which persist them with offsets. Delivery Workers consume messages from Kafka, write delivery state to Cassandra for tracking, and push messages to Consumers. Consumers acknowledge receipt, enabling Delivery Workers to mark messages as delivered. This flow supports at-most-once by skipping retries, at-least-once by retrying undelivered messages, and exactly-once by combining Kafka offsets with Cassandra state to avoid duplicates. The vertical layout clarifies the sequential processing steps and component roles, emphasizing how durable storage and acknowledgement interplay to achieve delivery guarantees.

Primary Focus

✓

Practice

(1/5)
1. A web application experiences increasing CPU load on a single database server, causing query latency spikes. Which scaling approach best addresses this bottleneck without changing the application code?
easy
A. Distribute the database load by sharding data across multiple servers (horizontal scaling)
B. Add more CPU cores and RAM to the existing database server (vertical scaling)
C. Use a caching layer in front of the database to reduce read load
D. Switch from SQL to NoSQL database to improve write throughput

Solution

  1. Step 1: Identify the bottleneck as CPU and memory on a single DB server

    Vertical scaling adds resources to the existing server to handle more load without architectural changes.

  2. Step 2: Evaluate horizontal scaling and other options

    Horizontal scaling requires application changes for sharding; caching reduces reads but not CPU-bound writes; switching DB types is a different trade-off.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Vertical scaling adds resources to one server without code changes [check]
Hint: Vertical scaling adds power to one machine [check]
Common Mistakes:
2. For a mobile client calling a flaky backend service, which approach best balances responsiveness and failure isolation?
medium
A. Use synchronous calls with retries and exponential backoff
B. Rely on client-side caching to avoid backend calls during failures
C. Implement a circuit breaker with a short timeout and fallback response
D. Use a message queue to buffer requests and process asynchronously

Solution

  1. Step 1: Evaluate synchronous retries (A)

    Retries increase latency and risk cascading failures, bad for mobile responsiveness.

  2. Step 2: Compare asynchronous queue (C) and caching (D)

    Queues add delay and complexity; caching avoids calls but may serve stale data, not ideal for dynamic data.

  3. Step 3: Circuit breaker with short timeout and fallback (B)

    Provides quick failure detection, prevents cascading failures, and returns fallback to maintain responsiveness.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Circuit breaker with fallback balances isolation and responsiveness [check]
Hint: Circuit breaker with fallback keeps mobile responsive and isolated [OK]
Common Mistakes:
3. At 10x scale, which component in the Ticket Server pattern is most likely to become the bottleneck first, and why?
hard
A. Ticket Server, because it handles all block allocations and cannot be sharded easily
B. Clients, because local ID block exhaustion causes frequent server requests
C. Network, because increased traffic saturates bandwidth between clients and server
D. Database storing allocated blocks, because it cannot handle concurrent writes at scale

Solution

  1. Step 1: Identify centralization bottleneck

    The Ticket Server centrally allocates ID blocks and maintains allocation state, limiting horizontal scaling.

  2. Step 2: Compare other components

    Clients scale horizontally; network scales with infrastructure; database load is less critical if Ticket Server caches state.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Ticket Server centralization -> first bottleneck at 10x scale [check]
Hint: Central Ticket Server state limits horizontal scaling first [check]
Common Mistakes:
4. At 10x scale, which component in a modular hashing distributed cache cluster is most likely to fail first and why?
hard
A. The centralized metadata service, due to bottleneck in updating key-node mappings
B. The network bandwidth between nodes, due to increased replication traffic
C. The client-side hashing logic, due to inability to handle large keyspace efficiently
D. The cache nodes themselves, due to uneven load causing hotspots and memory exhaustion

Solution

  1. Step 1: Identify modular hashing load distribution

    Modular hashing assigns keys unevenly when node count changes, causing hotspots on some cache nodes.

  2. Step 2: Analyze scaling impact

    At 10x scale, hotspots cause memory exhaustion and node failures before network or metadata bottlenecks.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Modular hashing hotspots cause node failures first at scale [check]
Hint: Modular hashing hotspots cause node failures first at scale [check]
Common Mistakes:
5. At 10x scale, which component in a Cassandra cluster is most likely to fail first and why?
hard
A. The coordinator nodes due to increased load managing large partitions and routing requests
B. The storage nodes due to disk space exhaustion
C. The client application due to increased request latency
D. The network switches due to bandwidth saturation

Solution

  1. Step 1: Identify bottlenecks at 10x scale

    Coordinator nodes handle query routing and aggregation; increased scale causes them to manage larger partitions and more requests.

  2. Step 2: Compare other components

    Storage nodes scale horizontally and add disk; network bandwidth scales with infrastructure upgrades; client latency is secondary to cluster bottlenecks.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Coordinator overload -> first failure point at 10x scale [check]
Hint: Coordinator nodes bottleneck first at scale due to routing load [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard