Bird
Raised Fist0

Synchronous vs Asynchronous Communication - When Each Wins

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Synchronous vs Asynchronous Communication - When Each Wins
45 min · senior
senior

You are designing the communication layer for a messaging platform with 50 million daily active users (DAU). The system must support both synchronous (real-time) and asynchronous (offline) message delivery modes. Messages must be end-to-end encrypted, support offline delivery with guaranteed eventual consistency, and maintain a P99 latency under 200ms for synchronous interactions. The system should gracefully handle network partitions and user offline scenarios without data loss or message duplication.

Constraints
  • End-to-end encryption mandatory for all messages
  • Support offline message delivery with eventual consistency
  • P99 latency for synchronous communication must be under 200ms
  • System must handle network partitions without message loss
  • Scale to 50M DAU with peak message throughput of ~1M msg/sec
📋
Guided Steps
1/5
Step 1
Choosing Communication Mode per Use Case
How do you decide when to use synchronous vs asynchronous communication for message delivery?
💡 Hint
Consider user experience requirements like latency and availability during offline periods.
Key Decision
Use WebSocket-based synchronous communication for online users requiring sub-200ms P99 latency, and Kafka-backed asynchronous queues for offline delivery and eventual consistency.
Interviewer Expectation
Mid-level: Names WebSocket for real-time and Kafka for async. Senior: Explains latency constraints and offline user scenarios dictating mode choice, including fallback logic.
⚠️Common Wrong Answer
Using synchronous HTTP polling for all messages - causes massive load spikes and fails to meet latency at 50M DAU.
Step 2
Ensuring End-to-End Encryption Across Modes
How do you design encryption to work seamlessly for both synchronous and asynchronous communication?
💡 Hint
Think about where encryption/decryption happens and how keys are managed for offline delivery.
Key Decision
Implement client-side end-to-end encryption with keys managed via a secure key management service (AWS KMS), ensuring encrypted payloads are stored and transmitted without server-side decryption.
Interviewer Expectation
Mid-level: Mentions client-side encryption and key management. Senior: Details key rotation, offline key retrieval, and how encrypted Kafka messages preserve confidentiality.
⚠️Common Wrong Answer
Encrypting only at transport layer (TLS) - exposes message content on server and breaks offline delivery confidentiality.
Step 3
Designing Message Storage and Delivery Guarantees
What storage and delivery mechanisms ensure no message loss or duplication, especially during network partitions?
💡 Hint
Consider partition keys and idempotency in Kafka and durable storage for offline sync.
Key Decision
Use Kafka with conversation_id as partition key to guarantee ordered delivery per conversation, combined with Cassandra for durable message storage and idempotent consumer logic to avoid duplicates.
Interviewer Expectation
Mid-level: Names Kafka and Cassandra. Senior: Explains why conversation_id partitioning ensures ordering and how idempotent consumers handle retries and duplicates.
⚠️Common Wrong Answer
Using a single global Kafka topic partition - causes ordering issues and scalability bottlenecks.
Step 4
API Design for Sending, Syncing, and Acknowledging Messages
Design 3 core API endpoints to support sending messages, syncing offline messages, and acknowledging receipt.
💡 Hint
Think about request payloads, encryption, and idempotency tokens.
Key Decision
POST /messages/send with encrypted payload and delivery mode flag; GET /messages/sync with cursor-based pagination for offline sync; POST /messages/ack with message IDs for exactly-once delivery confirmation.
Interviewer Expectation
Mid-level: Defines basic endpoints. Senior: Details encryption payload structure, cursor-based sync for efficient offline catch-up, and ack semantics to prevent duplicates.
⚠️Common Wrong Answer
Using simple GET polling without cursors - leads to inefficient sync and high load.
Step 5
Handling Failure and Recovery in Both Modes
How does the system detect failures and recover message delivery without violating latency or consistency SLAs?
💡 Hint
Consider heartbeat mechanisms, TTL caches, and retry policies.
Key Decision
Use Redis with TTL-based presence keys for online user heartbeats; delivery workers check Redis before deciding sync vs async path; implement exponential backoff retries with Kafka consumer offsets for recovery.
Interviewer Expectation
Mid-level: Mentions heartbeat and retries. Senior: Explains Redis TTL expiry triggers offline mode, how Kafka offsets ensure no message loss, and backpressure handling.
⚠️Common Wrong Answer
No heartbeat detection - causes message delivery attempts to synchronous users who are offline, increasing latency.
🔌
API Design Challenge
Sketch 3 core API endpoints to support synchronous and asynchronous communication modes in this system.
Hints
  • How does the client send a message ensuring encryption and delivery mode?
  • How does the client synchronize messages after reconnecting from offline?
  • How does the recipient acknowledge message receipt to ensure exactly-once delivery?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Clear distinction between synchronous (WebSocket) and asynchronous (Kafka) communication modes
  • Client-side end-to-end encryption with secure key management
  • Use of conversation_id partition key in Kafka for ordering and scalability
  • API design covering send, sync, and ack with encryption and idempotency
Good to Have - 30%
  • Heartbeat detection using Redis TTL for presence and mode switching
  • Durable storage in Cassandra for offline message persistence
  • Exponential backoff retry and idempotent consumer logic
  • Cursor-based pagination for efficient offline sync
Bonus - 10%
  • Detailed failure recovery flow tracing message loss scenarios
  • Discussion of trade-offs in latency vs consistency under network partitions
Anti-patterns to Avoid
HTTP polling for synchronous communication - causes excessive load and latency spikes at 50M DAU, resulting in system collapse due to 50 million reconnects per minute just to check for messages
Encrypting only at transport layer (TLS) - breaks end-to-end confidentiality especially for offline messages, exposing message content on servers and risking data leaks
Using a single Kafka partition for all messages - leads to ordering issues and scalability bottlenecks, causing message delays and throughput degradation
No heartbeat or presence detection - results in wasted synchronous delivery attempts to offline users, increasing latency and resource consumption
Stateless message delivery without idempotency - causes duplicates and inconsistent state, leading to poor user experience and data corruption
Simple GET polling without cursors for offline sync - inefficient and high resource consumption, causing unnecessary load and slow client synchronization
✅
Model Answer
▼ Reveal

This design balances synchronous and asynchronous communication to meet strict latency and reliability requirements for a 50M DAU messaging platform. It uses WebSocket for real-time messaging with P99 latency under 200ms and Kafka for asynchronous offline delivery with eventual consistency. Client-side end-to-end encryption with AWS KMS ensures confidentiality across modes. Durable storage in Cassandra supports offline sync, while Redis TTL-based heartbeats enable presence detection and mode switching. Idempotent Kafka consumers and cursor-based APIs prevent duplicates and optimize offline sync. Exponential backoff retry policies ensure robust failure recovery without overload.

Key Decisions
  • Use WebSocket for synchronous communication to meet P99 latency < 200ms
  • Partition Kafka topics by conversation_id to guarantee ordered, scalable async delivery
  • Implement client-side end-to-end encryption with AWS KMS for secure key management and offline access
  • Store messages durably in Cassandra to support offline sync with range queries
  • Detect user presence via Redis TTL keys to switch between sync and async delivery modes
  • Design APIs with encrypted payloads, cursor-based pagination for sync, and idempotent acknowledgments
  • Apply exponential backoff retry policies and Kafka consumer offsets to ensure no message loss or duplication
  • Avoid HTTP polling and single Kafka partition to prevent load spikes and ordering bottlenecks
  • WebSocket for synchronous communication to meet P99 < 200ms latency
  • Kafka with conversation_id as partition key for ordered, scalable async delivery
  • Client-side end-to-end encryption with AWS KMS for secure offline and online messaging
  • Cassandra for durable message storage enabling offline sync with range queries
  • Redis TTL-based heartbeat presence detection to switch between sync and async modes
  • Idempotent Kafka consumers and message acknowledgments to prevent duplicates
  • Cursor-based pagination in sync API to efficiently fetch offline messages
  • Exponential backoff retry policies for robust failure recovery without overload

Practice

(1/5)
1. In the cache warming system, at step 3 the Cache Loader fetches data from the database. What happens if the database response is delayed beyond the configured timeout?
easy
A. Cache Loader retries indefinitely until data is fetched, blocking subsequent steps
B. Cache Loader falls back to a secondary cache layer to fetch the data synchronously
C. Cache Loader marks the key as stale and skips warming that key for this cycle
D. Cache Loader writes a null value to cache to prevent cache miss storms

Solution

  1. Step 1: Trace Cache Loader timeout behavior

    On timeout, Cache Loader skips warming that key to avoid blocking the entire warming process.

  2. Step 2: Eliminate retry and fallback options

    Indefinite retries block warming; fallback cache is not part of warming design; null writes risk stale data.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Timeout causes skipping key warming to maintain overall progress [check]
Hint: Timeout skips key warming to avoid blocking cache load [check]
Common Mistakes:
2. In a system where a user requests access to a protected resource, at step 4 the Authorization Service checks the user's access token. What happens if the token is expired at this step?
easy
A. The Authorization Service forwards the request to the Resource Server without validation
B. The Authorization Service grants access but logs a warning for token expiry
C. The Authorization Service refreshes the token automatically and proceeds
D. The Authorization Service rejects the request and returns an HTTP 401 Unauthorized error

Solution

  1. Step 1: Understand Authorization Service behavior at token validation

    The service validates token expiry and rejects expired tokens.

  2. Step 2: Eliminate incorrect behaviors

    Granting access with expired token violates security; automatic refresh is done by Authentication Service, not Authorization; forwarding without validation breaks security.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Expired tokens cause 401 Unauthorized rejection [check]
Hint: Expired token -> Authorization rejects with 401 Unauthorized [check]
Common Mistakes:
3. In a distributed lock system using ZooKeeper, at step 4 the client attempts to acquire the lock by creating an ephemeral sequential node. What happens if the client crashes immediately after creating this node but before confirming lock ownership?
easy
A. ZooKeeper automatically deletes the ephemeral node, releasing the lock
B. The ephemeral node remains until manually deleted, causing a deadlock
C. The client retains the lock until a timeout expires
D. The lock is transferred to the next waiting client immediately

Solution

  1. Step 1: Understand ephemeral node semantics

    Ephemeral nodes in ZooKeeper are automatically deleted when the client session ends (e.g., crash), preventing deadlocks (A) and stale locks.

  2. Step 2: Trace lock ownership after crash

    Since the ephemeral node is deleted, the next client waiting for the lock will be notified, but the lock is not immediately transferred (D) until they acquire it explicitly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Ephemeral node deletion on client crash -> lock released [check]
Hint: Ephemeral nodes vanish on client crash, freeing locks [check]
Common Mistakes:
4. If the Redis Cluster's slot-to-node mapping cache on the client is stale and not updated after a node failure, what is the most likely failure observed by the client?
medium
A. Client requests succeed but with increased latency due to retries
B. Client receives MOVED errors and automatically updates mapping without failure
C. Client silently writes data to wrong nodes, causing data inconsistency
D. Client requests are sent to a failed node, causing timeouts and request failures

Solution

  1. Step 1: Understand stale slot cache impact

    Stale cache causes client to send requests to nodes no longer responsible or down.

  2. Step 2: Trace failure consequences

    Requests to failed nodes time out or fail; client does not receive MOVED errors because node is down (Client requests are sent to a failed node, causing timeouts and request failures).

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stale cache -> requests to failed nodes -> timeouts/failures [check]
Hint: Stale slot cache -> requests to dead nodes -> timeouts, no auto-fix [OK]
Common Mistakes:
5. For a large-scale code repository system supporting millions of users, which branching strategy best balances performance and user flexibility?
medium
A. Use a centralized locking mechanism to serialize all branch creation and merges
B. Use lightweight branch pointers referencing immutable commits with eventual consistency
C. Use full branch copies stored separately for each user to avoid conflicts
D. Use synchronous replication of branch metadata across all servers before allowing merges

Solution

  1. Step 1: Analyze branching strategies for scale

    Lightweight branch pointers referencing immutable commits allow fast branch creation and merges without locking, supporting high concurrency.

  2. Step 2: Evaluate other options

    Centralized locking serializes operations, hurting performance; full branch copies waste storage and complicate merges; synchronous replication adds latency and reduces availability.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Immutable commits + lightweight pointers -> scalable branching [check]
Hint: Immutable commits enable lightweight, scalable branches [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard