Bird
Raised Fist0

Synchronous vs Asynchronous Communication - When Each Wins

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Synchronous vs Asynchronous Communication - When Each Wins
Designing distributed systems by choosing between synchronous and asynchronous communication based on scale, latency, and reliability requirements
mediumCore Distributed Systems ConceptsFreq: high
⚙️
Functional Requirements
  • Support request-response interactions with low latency (synchronous communication).
  • Support decoupled, eventual processing of tasks or events (asynchronous communication).
  • Provide reliable message delivery with retry and failure handling.
  • Allow clients to choose communication mode based on use case requirements.
  • Enable monitoring and alerting on communication failures and latencies.
📊
Non-Functional Requirements
  • System must handle up to 2 billion daily active users (DAU).
  • End-to-end latency for synchronous calls must be under 100ms P99.
  • System availability must be at least 99.99%.
  • Asynchronous message processing throughput must support 925K messages per second peak.
  • Message durability guarantees with at least once delivery semantics.
❓
Clarifying Questions to Ask
  • What are the typical latency requirements for synchronous calls?
  • What failure modes are acceptable for asynchronous processing?
  • Are there specific use cases that mandate synchronous communication?
  • What is the expected message size and payload characteristics?
  • Is ordering of messages important in asynchronous communication?
  • What are the retry policies and dead-letter queue requirements?
Out of Scope
✗ Designing the entire application business logic beyond communication patterns.
✗ Security and authentication mechanisms for communication channels.
✗ Detailed UI/UX design for client interaction.
✗ Cross-data center replication strategies.
📌
Assumptions
  • 2 billion DAU
  • 40 messages per user per day on average
  • 100 bytes average message size
  • Peak traffic is 3x average traffic
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 msg/sec
peak write qps925,926 × 3 ≈ 2,777,778 msg/sec
read qpsAssuming 2 reads per write: 2 × 925,926 ≈ 1,851,852 reads/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024^3) ≈ 7.45 TB/day ≈ 7,450 GB/day
retention days7
total storage gb7,450 × 7 = 52,150 GB
with replication 3x52,150 × 3 = 156,450 GB (≈ 156.5 TB)
🖥️
Server Estimate
api serversAssuming 1000 req/sec per server, need ~2,778 servers for peak write QPS
message brokersKafka cluster sized for 3M msg/sec peak with 100 partitions and 20 brokers
🏗️
Architecture Diagram
Architecture Diagram
Client
  |
  v
API Gateway (Nginx + Envoy)
  |
  v
Load Balancer
  |
  v
Synchronous Service (gRPC/HTTP2)
  |          \
  |           \
  v            v
Database (Cassandra)  Kafka Cluster
                      |
                      v
Asynchronous Worker Pool
                      |
                      v
Result Store (Redis + Cassandra)
🧩
Components
API Gateway (Nginx + Envoy)
Handles client requests, routes to synchronous or asynchronous services
Why thisEnvoy supports advanced routing, retries, and observability for microservices
Why notHAProxy lacks native gRPC support and observability features at scale
Synchronous Service (gRPC/HTTP2)
Processes low-latency request-response calls synchronously
Why thisgRPC provides efficient binary protocol with built-in deadlines and retries
Why notREST/HTTP1.1 has higher overhead and less efficient streaming support
Kafka Cluster
Decouples producers and consumers for asynchronous message processing
Why thisKafka supports high throughput, partitioning, replication, and durable storage
Why notRabbitMQ does not scale linearly beyond hundreds of thousands msg/sec
Asynchronous Worker Pool
Consumes Kafka messages and processes tasks asynchronously
Why thisKubernetes-managed stateless workers allow elastic scaling and fault tolerance
Why notMonolithic background jobs lack elasticity and fault isolation
Database (Cassandra)
Stores synchronous request data with high write throughput and availability
Why thisCassandra's write-optimized LSM storage and linear scalability fit 2.7M writes/sec
Why notPostgreSQL requires complex manual sharding and does not scale linearly
Result Store (Redis + Cassandra)
Caches asynchronous processing results for fast retrieval and durability
Why thisRedis TTL natively expires stale data with sub-millisecond lookup; Cassandra stores durable results
Why notMemcached lacks persistence and TTL management; single DB adds latency
🔄
Data Flow
1
Client → API Gateway
Send request specifying sync or async mode
Client chooses communication mode based on latency tolerance
2
API Gateway → Synchronous Service
Route synchronous requests for immediate processing
gRPC call with 100ms deadline enforced
3
Synchronous Service → Database (Cassandra)
Write request data synchronously
Ensure durable write with quorum consistency
4
API Gateway → Kafka Cluster
Publish asynchronous requests as Kafka messages
Decouples client from processing latency
5
Kafka Cluster → Asynchronous Worker Pool
Workers consume messages and process tasks
Allows elastic scaling and retries
6
Asynchronous Worker Pool → Result Store (Redis + Cassandra)
Store processing results with TTL caching
Fast retrieval for clients polling results
7
Client → Result Store
Poll or fetch asynchronous results
Clients retrieve results when ready
🗄️
SynchronousRequests
PK: request_id
request_id UUIDuser_id UUIDpayload texttimestamp timestampstatus text
Stores synchronous request data with fast writes and query by request_id and time
🗄️
AsyncProcessingResults
PK: user_id
user_id UUIDresult_id UUIDresult_data textresult_timestamp timestampttl int
Stores asynchronous processing results with TTL for cache expiration
🔌
POST
SubmitRequest
Request
{
  "user_id": "UUID",
  "payload": "string",
  "mode": "string (sync|async)"
}
Response
{
  "request_id": "UUID",
  "status": "string"
}
Client submits request specifying synchronous or asynchronous mode
🔌
GET
GetAsyncResult
Request
{
  "user_id": "UUID"
}
Response
{
  "results": "list of result objects with result_id, data, timestamp"
}
Client polls for asynchronous processing results
🔌
GET
GetSyncStatus
Request
{
  "request_id": "UUID"
}
Response
{
  "status": "string",
  "timestamp": "timestamp"
}
Client checks status of synchronous request

Practice

(1/5)
1. In the request flow of the location tracking system, at step 4 the WebSocket Server receives a location update from a device. What does the WebSocket Server do immediately after receiving this update?
easy
A. It directly writes the location update to the relational database synchronously
B. It broadcasts the location update to all connected clients without filtering
C. It publishes the location update to a Kafka topic for asynchronous processing
D. It caches the location update in local memory without persistence

Solution

  1. Step 1: Trace WebSocket Server action at Step 4

    The server must handle high throughput and avoid blocking; synchronous DB writes would cause latency.

  2. Step 2: Evaluate alternatives

    Broadcasting to all clients wastes bandwidth; caching without persistence risks data loss.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Publishing to Kafka decouples ingestion from processing [check]
Hint: WebSocket Server publishes updates asynchronously to Kafka [check]
Common Mistakes:
2. If the origin server becomes unreachable, what exactly breaks in a CDN using origin pull with TTL caching?
medium
A. Edge nodes continue serving stale cached content indefinitely without errors
B. Edge nodes serve stale content until TTL expires, then return errors for new requests
C. Edge nodes immediately return errors for all requests without cached content
D. Edge nodes switch to push mode automatically to fetch content from other origins

Solution

  1. Step 1: Understand origin pull with TTL caching failure

    When origin is unreachable, edge nodes serve cached content until TTL expires.

  2. Step 2: Identify behavior after TTL expiry

    After TTL expires, edge nodes cannot refresh content and must return errors for new requests.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Origin down + expired TTL -> edge nodes return errors [check]
Hint: Expired TTL + origin down -> errors returned [check]
Common Mistakes:
3. If the distributed priority queue component fails and becomes unavailable, what is the immediate failure mode observed in the task scheduler?
medium
A. Worker nodes block waiting for tasks, causing execution delays and potential timeouts
B. Tasks are silently dropped without retries, causing data loss
C. Tasks continue executing but with incorrect priority ordering
D. The cron scheduler automatically reschedules tasks on all nodes

Solution

  1. Step 1: Understand role of distributed priority queue

    The queue is the source of truth for pending tasks; if unavailable, new tasks cannot be fetched.

  2. Step 2: Analyze failure impact

    Without the queue, tasks submitted may be lost if not persisted elsewhere; workers cannot fetch tasks, so blocking (B) is unlikely if queue is down; cron scheduler (D) does not reschedule tasks automatically.

  3. Step 3: Evaluate silent drops (C)

    Without queue durability, tasks may be lost silently if producers cannot enqueue or retries are not implemented.

  4. Final Answer:

    Option B -> Option B

  5. Quick Check:

    Queue failure causes silent task loss without retries [check]
Hint: Queue failure causes silent task loss without retries [check]
Common Mistakes:
4. If the Layer 7 load balancer's HTTP header inspection module fails, what is the most likely failure mode for incoming requests?
medium
A. Requests are forwarded blindly to a default backend without routing based on URL
B. The load balancer drops all TCP connections silently without response
C. The load balancer falls back to Layer 4 forwarding, preserving source IP but ignoring HTTP routing
D. The load balancer encrypts requests incorrectly causing backend decryption errors

Solution

  1. Step 1: Understand failure impact on Layer 7 inspection

    If HTTP header inspection fails, the load balancer may fallback to Layer 4 forwarding to maintain availability.

  2. Step 2: Eliminate options inconsistent with graceful degradation

    Dropping all connections (B) is unlikely; forwarding blindly to default backend (A) loses load distribution; encryption errors (D) unrelated.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Layer 7 failure -> fallback to Layer 4 forwarding without HTTP routing [check]
Hint: Layer 7 fail -> fallback to Layer 4 forwarding [check]
Common Mistakes:
5. At 10x traffic scale, which component in the cache warming architecture is most likely to become the bottleneck first, and why?
hard
A. The database, because it cannot handle the increased read load from the warming service
B. The network bandwidth, because warming floods the network with data transfers
C. The cache servers, because they run out of memory storing warmed keys
D. The cache warming coordinator, because it maintains global state and cannot shard effectively

Solution

  1. Step 1: Identify coordinator role and scaling limits

    The warming coordinator manages global warming state and task assignment, which is hard to shard.

  2. Step 2: Compare other components

    Database can be scaled with replicas; cache servers scale horizontally; network scales with capacity upgrades.

  3. Step 3: Conclude bottleneck

    Coordinator's centralized state causes it to fail first at 10x scale.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Centralized coordinator state limits sharding, causing bottleneck [check]
Hint: Central coordinator state limits sharding, causing bottleneck [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard