Bird
Raised Fist0

Why Async Messaging and What Synchronous Calls Cannot Handle

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Why Async Messaging and What Synchronous Calls Cannot Handle
Understanding the architectural rationale for async messaging over synchronous calls at scale
mediumMessage Queues and Event StreamingFreq: common
⚙️
Functional Requirements
  • Support decoupled communication between microservices via asynchronous message passing
  • Guarantee at-least-once delivery semantics for critical events
  • Enable message ordering within partitions for event streams
  • Allow consumers to process messages at their own pace without blocking producers
  • Provide message retention for replay and audit purposes
  • Support backpressure handling to prevent system overload
📊
Non-Functional Requirements
  • Handle 2 billion daily active users generating 40 messages per user per day
  • Achieve 99.99% availability for message ingestion and delivery
  • Ensure end-to-end message latency under 100ms P99 for critical flows
  • Scale to 925,000 peak write messages per second
  • Support message retention of 7 days with replication for durability
❓
Clarifying Questions to Ask
  • What is the expected message size and variability?
  • Are messages required to be strictly ordered globally or only within partitions?
  • What are the SLAs for message delivery latency and durability?
  • Is exactly-once delivery required or is at-least-once acceptable?
  • What is the expected consumer processing speed relative to producer speed?
  • Are there any constraints on message schema evolution or compatibility?
Out of Scope
✗ Designing the internal business logic of microservices consuming messages
✗ Authentication and authorization mechanisms for message producers/consumers
✗ UI/UX design for message monitoring dashboards
✗ Long-term archival storage beyond 7 days
📌
Assumptions
  • 2 billion DAU
  • 40 messages per user per day
  • 100 bytes average message size
  • 7 days message retention
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 msg/sec
peak write qpsAssuming 2× peak factor → 1,851,852 msg/sec
read qpsAssuming 1.5× read amplification → 1,388,889 msg/sec
💾
Storage Estimate
daily storage gb(2B × 40 × 100 bytes) / (1024^3) ≈ 7.45 TB/day
retention storage gb7.45 TB × 7 days ≈ 52.15 TB
with replication gb52.15 TB × 3 replicas ≈ 156.45 TB
🖥️
Server Estimate
kafka brokersAssuming 50 MB/s per broker → 7.45 TB/day ≈ 86 MB/s → 2,000 MB/s peak → 40 brokers
consumer workersAssuming 10,000 msg/sec per worker → 1,388,889 read QPS → 139 workers
🏗️
Architecture Diagram
Architecture Diagram
          +-------------+
          |  Producers  |
          +------+------+
                 |
                 v
          +------+-------+
          | Kafka Cluster|
          +------+-------+
                 |
                 v
          +------+-------+
          | Consumer Svc |
          +------+-------+
                 |
                 v
          +------+-------+
          | Downstream   |
          | Microservices|
          +--------------+
🧩
Components
Kafka Cluster
Distributed event streaming platform for asynchronous message ingestion and delivery
Why thisApache Kafka provides partitioned logs with high throughput, durability, and built-in replication supporting 1.8M msg/sec peak writes with 99.99% availability
Why notRabbitMQ lacks Kafka's horizontal scalability and partitioning, failing at sustained writes above 900K msg/sec
Producers
Microservices or clients publishing messages asynchronously to Kafka topics
Why thisProducers decouple message generation from consumption, enabling non-blocking writes and retry logic using Kafka Producer API
Why notSynchronous HTTP calls block producers waiting for consumer responses, causing cascading failures under load
Consumer Service
Microservices consuming messages from Kafka partitions at their own pace
Why thisConsumers can scale independently, handle backpressure, and replay messages from offsets for fault tolerance using Kafka Consumer API
Why notSynchronous RPC requires consumers to be always available and fast, which is brittle at scale
ZooKeeper Ensemble
Coordination service managing Kafka broker metadata and leader election
Why thisZooKeeper ensures Kafka cluster consistency and fault tolerance with quorum-based consensus
Why notEtcd or Consul add complexity and are not natively integrated with Kafka's ecosystem
Downstream Microservices
Business logic services processing consumed messages asynchronously
Why thisAllows independent scaling and failure isolation from producers and Kafka cluster
Why notDirect synchronous calls create tight coupling and reduce system resilience
🔄
Data Flow
1
Producers → Kafka Cluster
Publish messages asynchronously to Kafka topic partitions
Producers do not wait for consumers, enabling high throughput
2
Kafka Cluster → ZooKeeper Ensemble
Maintain partition leader metadata and replication state
Ensures fault tolerance and consistent message ordering
3
Consumer Service → Kafka Cluster
Poll messages from assigned partitions at own pace
Consumers control processing speed and can replay messages
4
Consumer Service → Downstream Microservices
Deliver processed messages asynchronously for business logic
Decouples processing from message ingestion
5
Kafka Cluster → Storage (Retention)
Persist messages for configured retention period
Supports replay and audit capabilities

Practice

(1/5)
1. In the newsfeed system, at Step 4 the Delivery Worker reads a fan-out event from Kafka and attempts to deliver it to followers. What happens if a follower's device is offline at this step?
easy
A. Delivery Worker marks the message as sent but not delivered, queues for retry later
B. Delivery Worker drops the message to avoid blocking other deliveries
C. Delivery Worker retries immediately until the follower is online
D. Delivery Worker pushes the message to the follower's device cache directly

Solution

  1. Step 1: Trace Delivery Worker behavior at Step 4

    The Delivery Worker reads fan-out events from Kafka and attempts delivery. If the follower is offline, immediate retry (A) would block and reduce throughput. Dropping messages (C) causes data loss. Pushing directly to device cache (D) is impossible if offline.

  2. Step 2: Understand retry mechanism

    The system marks the message as sent but not delivered, queues it for retry later when the follower reconnects, ensuring eventual delivery without blocking.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Offline followers cause queued retries, not immediate blocking [check]
Hint: Offline followers cause queued retries, not immediate blocking [check]
Common Mistakes:
2. A typeahead system must handle millions of concurrent prefix queries with low latency and high throughput. Which component is best suited to serve these prefix lookups efficiently at scale?
easy
A. A distributed in-memory Trie cache sharded by prefix hash
B. A message queue to buffer prefix queries before processing
C. A full-text search engine like Elasticsearch without caching
D. A relational database with indexed LIKE queries on the prefix column

Solution

  1. Step 1: Identify prefix lookup requirements

    Prefix queries require fast, low-latency lookups optimized for string prefixes, which relational DB LIKE queries are inefficient at scale.

  2. Step 2: Evaluate component suitability

    A distributed in-memory Trie cache supports efficient prefix matching and can be sharded by prefix hash for scalability, unlike message queues or full-text search without caching.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Distributed Trie cache -> efficient prefix lookups at scale [check]
Hint: Trie caches excel at prefix queries, unlike DB LIKE or queues [check]
Common Mistakes:
3. A distributed system needs to generate unique 64-bit IDs that are sortable by creation time and avoid collisions across multiple data centers without central coordination. Which component design best fits this requirement?
easy
A. Use a UUID v4 generator that produces random 128-bit IDs without ordering guarantees.
B. Use a hash of the current timestamp and server IP address as the unique ID.
C. Use a Snowflake-like ID generator combining timestamp, machine ID, and sequence number.
D. Use a centralized database auto-increment ID generator with a global lock.

Solution

  1. Step 1: Identify uniqueness and ordering requirements

    The system requires unique 64-bit IDs sortable by creation time and no central coordination to avoid bottlenecks.

  2. Step 2: Evaluate options against constraints

    UUID v4 (A) is random and not sortable; centralized DB (C) creates a bottleneck; hashing timestamp and IP (D) risks collisions and no strict ordering. Snowflake (B) combines timestamp, machine ID, and sequence to ensure uniqueness and time ordering without central coordination.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Snowflake IDs are time-sortable and unique across machines [check]
Hint: Timestamp + machine ID + sequence ensures unique sortable IDs [OK]
Common Mistakes:
4. If the Kafka cluster responsible for vote event streaming goes down, what exactly breaks in the Reddit voting system?
medium
A. Vote events cannot be buffered, causing vote submission requests to block or fail immediately
B. The UI shows stale post scores but votes are queued in Redis for later processing
C. Votes continue to be processed but ranking updates become inconsistent
D. Users can still vote, but votes are lost silently without any retry

Solution

  1. Step 1: Understand Kafka's role in buffering vote events

    Kafka buffers vote events reliably; if it fails, vote events cannot be streamed to processors.

  2. Step 2: Analyze failure impact

    Vote submissions do not block immediately (Vote events cannot be buffered, causing vote submission requests to block or fail immediately wrong) because frontends can still accept votes and cache temporarily. Votes are not lost silently (Users can still vote, but votes are lost silently without any retry wrong) because Redis or other caches buffer them. UI shows stale scores due to lack of processed updates (The UI shows stale post scores but votes are queued in Redis for later processing correct). Votes continue to be processed but ranking updates become inconsistent is incorrect as votes stop flowing, not just ranking inconsistency.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Kafka failure causes stale UI scores; votes buffered in Redis [check]
Hint: Kafka down -> stale UI, votes buffered temporarily [check]
Common Mistakes:
5. For a high-throughput e-commerce platform requiring sortable, unique order IDs generated across multiple regions with minimal coordination, which ID generation approach is best?
medium
A. Use Twitter Snowflake-style custom IDs with timestamp and region bits
B. Use UUIDv4 generated randomly at each region
C. Use auto-increment IDs generated by a centralized database
D. Use auto-increment IDs generated independently per region without coordination

Solution

  1. Step 1: Identify requirements

    High throughput, sortable, unique IDs across regions with minimal coordination.

  2. Step 2: Evaluate options

    UUIDv4 (A) is unique but not sortable; centralized auto-increment (C) causes bottlenecks; independent auto-increment (D) risks collisions; Snowflake-style IDs (B) embed timestamp and region, enabling sortable, unique IDs with decentralized generation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Snowflake IDs -> sortable, unique, decentralized [check]
Hint: Sortable unique IDs at scale -> Snowflake pattern [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard