Bird
Raised Fist0

Why Async Messaging and What Synchronous Calls Cannot Handle

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Why Async Messaging and What Synchronous Calls Cannot Handle
45 min · senior
senior

You are designing the communication backbone for a 50 million daily active user (DAU) social media startup. The system must support real-time messaging with end-to-end encryption, offline message delivery, and maintain a P99 latency under 200ms for message delivery acknowledgments. The business requires that users can send messages even when recipients are offline, and messages must be reliably delivered once recipients reconnect. Synchronous HTTP calls have been causing bottlenecks and failures during peak loads.

Constraints
  • End-to-end encryption mandatory for all messages
  • Offline message delivery with guaranteed eventual consistency
  • P99 message delivery acknowledgment latency under 200ms
  • System must handle peak load of 10 million concurrent connections
  • No message loss allowed under any failure scenario
📋
Guided Steps
1/5
Step 1
Choosing the Messaging Pattern
Why is asynchronous messaging preferred over synchronous calls for this system?
💡 Hint
Consider how synchronous HTTP calls behave under high concurrency and network delays.
Key Decision
Use Apache Kafka as the asynchronous messaging backbone to decouple sender and receiver, enabling high throughput and fault tolerance.
Interviewer Expectation
Mid: Names Kafka for async messaging. Senior: Explains how Kafka partitions by conversation_id ensure ordered delivery and how synchronous HTTP calls cause head-of-line blocking and resource exhaustion at 10M concurrent users.
⚠️Common Wrong Answer
Using synchronous HTTP calls for message delivery, which leads to thread exhaustion and increased latency under peak load.
Step 2
Message Storage and Partitioning
How do you store and partition messages to support offline delivery and ordered retrieval?
💡 Hint
Think about a storage system optimized for write-heavy workloads and range queries by conversation.
Key Decision
Use Apache Cassandra with partition_key=conversation_id to enable linear scalable writes and efficient cursor-based range queries for offline sync.
Interviewer Expectation
Mid: Suggests Cassandra for storage. Senior: Explains why conversation_id as partition key ensures all messages in a conversation are on the same node for ordered retrieval, rejecting sender_id partitioning due to out-of-order delivery.
⚠️Common Wrong Answer
Using relational databases with sender_id partitioning causing cross-node queries and inconsistent message order.
Step 3
Ensuring End-to-End Encryption
How do you design the system to guarantee end-to-end encryption without breaking offline delivery?
💡 Hint
Consider where encryption and decryption happen and how keys are managed for offline recipients.
Key Decision
Implement client-side encryption with keys managed via a secure key management service (AWS KMS) and store only encrypted payloads in Kafka and Cassandra.
Interviewer Expectation
Mid: Mentions client-side encryption. Senior: Details key rotation, secure key exchange protocols, and how encrypted messages are stored and delivered without server-side decryption.
⚠️Common Wrong Answer
Encrypting messages only on the server side, exposing plaintext during offline storage and violating end-to-end encryption.
Step 4
Handling Offline Message Delivery and Sync
How does the system ensure reliable delivery and synchronization of messages when recipients are offline?
💡 Hint
Think about message queues, delivery workers, and client sync protocols.
Key Decision
Use Kafka consumer groups with dedicated delivery workers that track offsets per user and push notifications via WebSocket or push services upon reconnect.
Interviewer Expectation
Mid: Describes Kafka consumers for delivery. Senior: Explains offset tracking per user, exactly-once delivery semantics, and fallback to push notifications for offline devices.
⚠️Common Wrong Answer
Polling the database for new messages causing excessive load and high latency.
Step 5
API Design for Messaging
Design three core API endpoints that enable sending messages, syncing offline messages, and acknowledging receipt.
💡 Hint
How does the client send messages asynchronously?How does the client request missed messages after reconnecting?How does the client confirm message receipt to avoid duplicates?
Key Decision
Define POST /messages/send for async message submission, GET /messages/sync with cursor for offline sync, and POST /messages/ack for delivery acknowledgment.
Interviewer Expectation
Mid: Names endpoints and basic functionality. Senior: Details request/response payloads, idempotency keys, cursor-based pagination, and how these APIs integrate with Kafka and Cassandra.
⚠️Common Wrong Answer
Using synchronous blocking endpoints without idempotency or cursor support, leading to poor scalability and inconsistent state.
🔌
API Design Challenge
Sketch 3 core API endpoints for the messaging system that enable sending messages, syncing offline messages, and acknowledging receipt.
Hints
  • How does the client send a message asynchronously to the server?
  • How does the client synchronize messages missed while offline?
  • How does the recipient acknowledge message receipt to ensure reliability?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of Apache Kafka for asynchronous messaging with conversation_id partitioning
  • Use of Apache Cassandra for message storage with conversation_id partition key
  • Client-side end-to-end encryption with secure key management
  • Design of APIs supporting async send, offline sync, and acknowledgments
Good to Have - 30%
  • Explanation of failure modes of synchronous HTTP calls at 10M concurrent users
  • Use of Kafka consumer groups and offset tracking for exactly-once delivery
  • Fallback mechanisms like WebSocket and push notifications for offline users
Bonus - 10%
  • Discussion of idempotency keys and cursor-based pagination in API design
  • Deep failure analysis of message loss scenarios and recovery
Anti-patterns to Avoid
Synchronous HTTP calls causing thread exhaustion and increased latency under peak load, leading to system crashes and degraded user experience
Polling the database for new messages leading to excessive load and poor scalability, causing high latency and potential database outages
Server-side encryption exposing plaintext during offline storage violating end-to-end encryption, risking user data privacy
Partitioning messages by sender_id causing out-of-order delivery and cross-node queries, resulting in inconsistent message ordering and increased latency
✅
Model Answer
▼ Reveal

This design uses Apache Kafka for asynchronous messaging to handle 10M concurrent users without blocking, Apache Cassandra for scalable message storage partitioned by conversation_id to maintain order and enable offline sync, and client-side end-to-end encryption with AWS KMS to secure messages even when offline. Delivery workers consume Kafka topics with offset tracking to ensure exactly-once delivery, falling back to WebSocket or push notifications for offline users. APIs are designed with idempotency keys and cursor-based pagination to support async send, offline sync, and acknowledgments efficiently.

Key Decisions
  • Use Apache Kafka for asynchronous messaging with conversation_id partitioning to ensure ordered, scalable message delivery
  • Store messages in Apache Cassandra partitioned by conversation_id for efficient writes and offline sync
  • Implement client-side end-to-end encryption with AWS KMS to protect message confidentiality even during offline storage
  • Use Kafka consumer groups with delivery workers tracking per-user offsets for exactly-once delivery semantics
  • Design APIs with idempotency keys and cursor-based pagination to handle retries and efficient offline message synchronization
  • Reject synchronous HTTP calls due to thread exhaustion and head-of-line blocking at 10M concurrent users
  • Avoid polling the database for new messages to prevent excessive load and high latency
  • Use WebSocket and push notification fallback mechanisms for real-time and offline message delivery
  • Apache Kafka with conversation_id as partition key to guarantee ordered, scalable async messaging
  • Apache Cassandra with partition_key=conversation_id for linear scalable writes and efficient offline sync
  • Client-side end-to-end encryption with AWS KMS for secure key management and offline message confidentiality
  • Kafka consumer groups with delivery workers tracking offsets per user for exactly-once offline delivery
  • WebSocket and push notification fallback for real-time and offline message delivery
  • API endpoints: POST /messages/send, GET /messages/sync with cursor, POST /messages/ack with idempotency keys
  • Reject synchronous HTTP calls due to thread exhaustion and head-of-line blocking at 10M concurrent users
  • Avoid polling-based sync to prevent database overload and high latency
  • Use cursor-based pagination for efficient offline message synchronization
  • Implement idempotency keys in APIs to handle retries without duplicate message processing

Practice

(1/5)
1. You must design a multi-primary replication system for a geo-distributed database with low write latency and conflict detection. Which approach is best?
medium
A. Use asynchronous replication with vector clocks for conflict detection and resolution
B. Use a single global primary with read replicas in each region
C. Use synchronous replication between primaries to ensure consistency
D. Use asynchronous replication with last-write-wins conflict resolution

Solution

  1. Step 1: Analyze latency and conflict requirements

    Low write latency requires asynchronous replication; conflict detection needs metadata like vector clocks.

  2. Step 2: Evaluate options

    Synchronous replication (A) adds latency; single primary (C) violates multi-primary requirement; last-write-wins (D) ignores conflicts.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Async replication + vector clocks -> low latency + conflict detection [check]
Hint: Vector clocks detect conflicts in async multi-primary [check]
Common Mistakes:
2. For a pub-sub system requiring low latency fan-out to millions of subscribers with dynamic topic subscriptions, which architecture is best suited?
medium
A. Use a centralized broker that pushes messages directly to all subscribers synchronously
B. Use a multicast network protocol to broadcast messages to all subscribers
C. Use a polling-based system where subscribers fetch messages periodically from a database
D. Use a distributed log system with topic partitions and consumer groups for parallel consumption

Solution

  1. Step 1: Evaluate synchronous push and multicast

    Centralized synchronous push (Use a centralized broker that pushes messages directly to all subscribers synchronously) causes bottlenecks and backpressure. Multicast (Use a multicast network protocol to broadcast messages to all subscribers) lacks reliability and fine-grained subscription control.

  2. Step 2: Assess polling and distributed log

    Polling (Use a polling-based system where subscribers fetch messages periodically from a database) adds latency and load. Distributed log with partitions and consumer groups (Use a distributed log system with topic partitions and consumer groups for parallel consumption) supports scalable, parallel, and dynamic subscriptions with low latency.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Distributed logs scale fan-out with parallel consumers [check]
Hint: Distributed logs + consumer groups = scalable low-latency fan-out [check]
Common Mistakes:
3. At 10x traffic scale, which load balancer component is most likely to become the bottleneck first and why?
hard
A. The IP Hash function, because hashing client IPs becomes computationally expensive at scale
B. The weighted server list, because updating weights dynamically causes synchronization delays
C. The centralized connection tracker, because Least Connections requires global state updates for every connection change
D. The Round Robin pointer, because cycling through servers sequentially cannot handle high concurrency

Solution

  1. Step 1: Identify statefulness in load balancing algorithms

    Least Connections requires tracking active connections per server globally, needing frequent updates.

  2. Step 2: Analyze scaling impact

    At 10x scale, the centralized connection tracker becomes a bottleneck due to synchronization overhead and contention.

  3. Step 3: Evaluate other components

    IP Hash is stateless and fast; weighted lists update less frequently; Round Robin pointer is lightweight and easily sharded.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Global connection tracking bottlenecks Least Connections at high scale [check]
Hint: Global connection tracking bottlenecks Least Connections at scale [check]
Common Mistakes:
4. At 10x traffic scale, which component is most likely to fail first in a layered rate limiting system and why?
hard
A. API Gateway, because in-memory counters consume excessive memory and CPU per connection
B. Database, because synchronous counter updates cause write bottlenecks
C. Service Layer, because distributed caches cannot synchronize counters fast enough
D. Network Load Balancer, because it cannot handle the increased connection churn

Solution

  1. Step 1: Analyze API Gateway resource usage

    API Gateway maintains in-memory counters per user/request, which grow linearly with traffic, exhausting memory and CPU first.

  2. Step 2: Compare other components

    Databases can be sharded and scaled; service caches sync asynchronously; load balancers handle connection churn better than API Gateway resource limits.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    API Gateway memory/CPU bottleneck at 10x scale [check]
Hint: API Gateway memory/CPU hits first at 10x scale [check]
Common Mistakes:
5. At 10x scale, which component in a Rendezvous Hashing-based distributed system is most likely to fail first and why?
hard
A. The client-side routing logic fails due to maintaining large node lists and computing scores
B. The central coordinator fails due to increased coordination overhead
C. The hash function computation becomes a CPU bottleneck due to increased node and key counts
D. The network bandwidth saturates because all nodes broadcast state to each other

Solution

  1. Step 1: Identify system components at scale

    Rendezvous Hashing is decentralized, so no central coordinator (The central coordinator fails due to increased coordination overhead) exists. Nodes do not broadcast state to all others (The network bandwidth saturates because all nodes broadcast state to each other) in typical implementations.

  2. Step 2: Analyze client-side routing impact

    Clients must compute scores for all nodes to pick highest; at 10x scale, this computation and memory for node lists become bottlenecks.

  3. Step 3: Consider hash computation cost

    Hash computations (The hash function computation becomes a CPU bottleneck due to increased node and key counts) are lightweight compared to maintaining large node lists and score computations.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Client routing logic scales poorly with node list size -> bottleneck [check]
Hint: Client routing logic chokes on large node lists [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard