Bird
Raised Fist0

Design a Generic Newsfeed System - Pull vs Push vs Hybrid Fan-Out at Any Scale

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
API Gateway

Receives new feed item POST request

Validates and forwards request to Write Service

Protocol:HTTP POST
BeforeUser has composed a new feed item on client
AfterFeed item request accepted and forwarded to Write Service
Write Service

Publishes feed item to Kafka topic

Serializes feed item and produces to feeditem topic

Protocol:Kafka Produce
BeforeFeed item request received but not yet persisted
AfterFeed item durably stored in Kafka topic for fan-out
Fan-out Workers

Consume feed item from Kafka

Fan-out feed item to followers by writing to Redis and Cassandra

Protocol:Kafka Consume, Redis SET, Cassandra INSERT
BeforeFeed item present in Kafka but not yet delivered
AfterFeed item cached in Redis for push delivery and stored durably in Cassandra
Read Service

Reads feed for user request

Fetches feed items from Redis cache or falls back to Cassandra

Protocol:Redis GET or Cassandra Query
BeforeUser requests feed, no cached feed items loaded
AfterUser feed populated from Redis cache if available, else from Cassandra storage
Redis (Cache)

Serves cached feed items

Returns recent feed items for fast read access

Protocol:Redis GET
BeforeFeed items cached by Fan-out Workers
AfterFeed items delivered quickly to Read Service for user
Cassandra (Storage)

Stores feed items durably

Persists feed items and fan-out state for pull reads

Protocol:Cassandra INSERT
BeforeFeed item not yet stored in durable storage
AfterFeed item durably stored in Cassandra for pull-based retrieval
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────┐
│  API Gateway  │
└───────┬───────┘
        │ HTTP POST /feeditem
        ▼
┌───────────────┐
│ Write Service │
└───────┬───────┘
        │ Kafka Produce
        ▼
┌───────────────────┐
│ Kafka (Pub/Sub)   │
│  (feeditem topic) │
└───────┬───────────┘
        │ Kafka Consume
        ▼
┌───────────────┐
│Fan-out Workers│
└───────┬───────┘
        │ Redis SET (push cache)
        │ Cassandra INSERT (storage)
        ▼
┌───────────────┐
│ Redis (Cache) │
└───────┬───────┘
        │ Redis GET (pull reads)
        ▼
┌───────────────┐
│ Read Service  │
└───────────────┘
Design Rationale

This architecture shows a generic newsfeed system using a hybrid fan-out approach. The API Gateway receives new feed items and forwards them to the Write Service, which publishes them to a Kafka topic. Fan-out Workers consume from Kafka and write feed items to Redis cache for fast push delivery and to Cassandra for durable storage supporting pull reads. The Read Service serves user feed requests by first querying Redis for cached items, falling back to Cassandra if needed. This design balances durability, scalability, and latency by combining push caching with pull storage, enabling efficient feed delivery at any scale.

Primary Focus

✓

Practice

(1/5)
1. A distributed messaging system processes millions of events per second. Some messages fail processing due to transient errors and must be retried without blocking the main queue. Which component best handles isolating these failed messages for later inspection and retry?
easy
A. A Dead Letter Queue (DLQ) configured to capture failed messages separately
B. A cache layer like Redis to temporarily store failed messages for retry
C. A high-throughput relational database with retry logic in application code
D. A load balancer that redirects failed messages to a backup server

Solution

  1. Step 1: Identify the role of Dead Letter Queues

    DLQs isolate failed messages from the main queue, enabling retries without blocking normal processing.

  2. Step 2: Evaluate alternatives

    Relational DBs (A) add latency and complexity; Redis (C) lacks durability guarantees; load balancers (D) do not handle message state or retries.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DLQ isolates failures for retry without blocking main queue [check]
Hint: Dead Letter Queues isolate failures, not caches or DBs [OK]
Common Mistakes:
2. If the cache warming service crashes after partially warming keys but before marking completion, what is the immediate failure observed during launch?
medium
A. The cache contains stale data from previous runs causing incorrect responses
B. The cache warming service automatically retries from the start, causing duplicate warming and wasted resources
C. The system falls back to a read-through cache mode with no performance impact
D. Clients experience cache misses for keys not warmed, causing increased latency and possible DB overload

Solution

  1. Step 1: Trace failure impact of partial warming crash

    Without completion mark, warming restarts causing duplicate warming and resource waste.

  2. Step 2: Eliminate other options

    Stale data unlikely if warming overwrites; cache misses occur but duplicate warming is immediate failure; fallback read-through causes latency, not immediate failure.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Crash without completion causes duplicate warming retries [check]
Hint: Crash without completion mark causes duplicate warming retries [check]
Common Mistakes:
3. You must design a globally distributed database that prioritizes low latency reads and writes even during network partitions, accepting eventual consistency. Which architecture is best suited for this requirement?
medium
A. Use a CP system with synchronous replication across regions
B. Use an AP system with asynchronous replication and conflict resolution
C. Use a CA system with strong consistency and no partition tolerance
D. Use a single-region CP system with leader election

Solution

  1. Step 1: Analyze latency and partition tolerance needs

    Low latency during partitions requires availability and partition tolerance, sacrificing immediate consistency.

  2. Step 2: Match architecture to constraints

    AP systems with asynchronous replication provide availability and eventual consistency during partitions.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    AP architecture fits low latency and partition tolerance with eventual consistency [check]
Hint: AP systems enable low latency with eventual consistency during partitions [check]
Common Mistakes:
4. At 10x scale, which component in a GraphQL API gateway is most likely to become the bottleneck and why?
hard
A. The underlying database, because it cannot handle 10x query volume
B. The GraphQL query parsing and validation layer, due to CPU-intensive query complexity checks on every request
C. The network load balancer, because it cannot distribute traffic evenly at high scale
D. The client SDK, because it caches too much data locally

Solution

  1. Step 1: Identify components stressed by 10x scale

    Database scaling is often handled by sharding or replicas; load balancers scale horizontally; client SDKs are local.

  2. Step 2: Recognize GraphQL parsing and validation cost

    Parsing and complexity validation are CPU-bound and happen per request, becoming CPU bottlenecks at high request rates.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    GraphQL validation CPU cost scales linearly with requests [check]
Hint: GraphQL validation CPU cost limits scale first [check]
Common Mistakes:
5. At 10x scale, which component fails first in a real-time chat system and why?
hard
A. The database fails first due to increased write volume causing slow queries
B. The message queue fails first due to partition leader overload
C. The load balancer fails first because it cannot handle the increased TCP connections
D. The chat servers fail first because WebSocket connections are stateful and cannot be resharded easily

Solution

  1. Step 1: Identify stateful bottleneck at scale

    Chat servers maintain stateful WebSocket connections bound to server memory, making resharding difficult at 10x scale.

  2. Step 2: Compare other components

    Databases (A) can be scaled with sharding and caching; message queues (C) can add partitions; load balancers (D) scale horizontally more easily.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stateful WebSocket servers -> first bottleneck at 10x scale [check]
Hint: Stateful WebSocket servers block scaling; resharding is hard [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard