Bird
Raised Fist0

Design Quora - Q&A Feed, Search, Ranking, Spam Detection, Follow Topics

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design Quora - Q&A Feed, Search, Ranking, Spam Detection, Follow Topics
Design a scalable Q&A platform with personalized feeds, search, ranking, and spam detection at 2B DAU scale
hardSearch and DiscoveryFreq: high
Prerequisites:Inverted IndexElasticsearch
⚙️
Functional Requirements
  • Users can post questions and answers with rich text and media attachments.
  • Personalized Q&A feed showing questions and answers from followed topics and users.
  • Full-text search over questions, answers, and topics with relevance ranking.
  • Spam detection and moderation pipeline to filter low-quality or abusive content.
  • Users can follow topics and other users to customize their feed.
  • Ranking algorithm to surface relevant, recent, and high-quality content.
📊
Non-Functional Requirements
  • Support 2 billion daily active users (DAU).
  • Write throughput: up to 925K writes/sec (questions + answers + follows).
  • Read latency: 100ms P99 for feed and search queries.
  • System availability: 99.99% uptime SLA.
  • Search index freshness: under 5 seconds delay from write to searchable.
❓
Clarifying Questions to Ask
  • What is the expected ratio of questions to answers per user per day?
  • Are answers editable after posting, and how does that affect indexing?
  • What is the expected query QPS for search vs feed retrieval?
  • How personalized should the feed be? Is real-time personalization required?
  • What is the tolerance for stale data in feeds and search results?
  • What are the spam detection accuracy and latency requirements?
Out of Scope
✗ User authentication and authorization mechanisms.
✗ Detailed UI/UX design and client-side caching.
✗ Payment or monetization features.
✗ Offline or mobile sync capabilities.
📌
Assumptions
  • 2 billion DAU with 40 posts (questions + answers) per user per day on average.
  • Average post size is 100 bytes (text + metadata).
  • Feed read QPS is 10x write QPS due to high read-to-write ratio.
  • Retention period for posts is 365 days.
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak factor → 1,851,852 writes/sec
read qpswrite_qps × 10 = 9,259,260 reads/sec
💾
Storage Estimate
daily storage925,926 writes/sec × 100 bytes × 86400 sec ≈ 8 TB/day
retention storage8 TB/day × 365 days = 2.92 PB
with replication2.92 PB × 3 replicas = 8.76 PB total storage
🖥️
Server Estimate
write frontendAssuming 10K writes/sec per server → 185 servers for peak writes
search clusterElasticsearch nodes: 500 nodes to handle 9M QPS with 100ms P99 latency
🏗️
Architecture Diagram
Architecture Diagram
          +-------------+
          |  User App   |
          +------+------+
                 |
          +------+-------+
          | API Gateway  |
          +------+-------+
                 |
  +--------------+--------------+
  |                             |
  |                             |
+v----+                   +----v+
|Write|                   |Read |
|Srv  |                   |Srv  |
+--+--+                   +--+--+
   |                          |
   |                          |
+--v--------------------------v--+
| Kafka (Write Event Bus)         |
+--+--------------------------+--+
   |                          |
+--v--+                   +---v---+
|Cassandra|               |Elasticsearch|
|(Posts DB)|               |(Search Index)|
+--+------+               +---+---------+
   |                          |
   |                          |
+--v--------------------------v--+
|Spam Detection Service           |
+--+--------------------------+--+
   |                          |
   |                          |
+--v--------------------------v--+
|Feed Generation Service          |
+--+--------------------------+--+
   |                          |
   |                          |
+--v--------------------------v--+
|Cache Layer (Redis)              |
+--------------------------------+
🧩
Components
API Gateway
Entry point for all client requests, handles authentication, rate limiting, and routing.
Why thisEnvoy proxy supports high throughput, low latency, and dynamic routing for 2B DAU scale.
Why notNginx lacks native service mesh integration and dynamic routing at this scale.
Kafka (Write Event Bus)
Durable, ordered, partitioned log for all write events (questions, answers, follows).
Why thisKafka supports 10M+ writes/sec with partitioning and retention, enabling async processing and replay.
Why notRabbitMQ does not scale linearly beyond 1M messages/sec and lacks partitioned logs.
Cassandra (Posts DB)
Primary storage for questions, answers, user follows with high write throughput and linear scalability.
Why thisCassandra's wide-column model supports high write QPS with tunable consistency and multi-DC replication.
Why notPostgreSQL requires complex manual sharding at 925K writes/sec; B-tree writes don't scale linearly.
Elasticsearch (Search Index)
Full-text search engine indexing questions, answers, and topics with inverted index and ranking.
Why thisElasticsearch provides near real-time indexing with distributed inverted index and relevance scoring.
Why notSolr has higher indexing latency and less flexible scaling for sub-5 second freshness.
Spam Detection Service
Processes write events asynchronously to detect spam using ML models and heuristics.
Why thisTensorFlow Serving enables scalable, low-latency inference pipelines integrated with Kafka streams.
Why notRule-based filters alone cannot adapt to evolving spam patterns at scale.
Feed Generation Service
Generates personalized Q&A feeds by aggregating followed topics/users and ranking content.
Why thisPrecomputes feeds asynchronously using Apache Flink for stream processing and stateful aggregation.
Why notOn-demand feed generation causes high read latency and cannot meet 100ms P99 SLA.
Redis (Cache Layer)
Caches personalized feeds and hot search results to reduce read latency and backend load.
Why thisRedis TTL natively expires stale feed entries without cron jobs - sub-millisecond lookup.
Why notMemcached lacks persistence and advanced data structures needed for feed caching.
Write Frontend Servers
Handle client write requests such as posting questions, answers, and follows, validating and forwarding to Kafka.
Why thisStateless servers allow horizontal scaling and quick failover; implemented with gRPC for low latency.
Why notStateful servers increase complexity and reduce scalability at 2B DAU scale.
Read Frontend Servers
Serve feed and search read requests by querying cache and backend services with low latency.
Why thisStateless design supports autoscaling and load balancing; uses HTTP/2 for efficient multiplexing.
Why notMonolithic servers would cause bottlenecks and reduce availability.
🗄️
Posts
PK: post_id
post_iduser_idcontentmedia_urlspost_typetopic_idscreation_timelast_edit_timeis_spam
🗄️
UserFollows
PK: user_id
user_idfollowed_entity_identity_typefollow_time
🔌
PostQuestion
Request
{
  "user_id": "string",
  "content": "string",
  "media_urls": "string[]",
  "topic_ids": "string[]"
}
Response
{
  "post_id": "string",
  "status": "string"
}
Allows users to post a new question with optional media and topics.
🔌
GetFeed
Request
{
  "user_id": "string",
  "page_token": "string",
  "page_size": "int"
}
Response
{
  "posts": "Post[]",
  "next_page_token": "string"
}
Fetches personalized feed posts for the user with pagination.
🔌
Search
Request
{
  "query": "string",
  "filters": "map<string,string>",
  "page_token": "string",
  "page_size": "int"
}
Response
{
  "results": "Post[]",
  "next_page_token": "string"
}
Performs full-text search over questions, answers, and topics.

Practice

(1/5)
1. In a Cache Aside pattern, at step 3 the application checks the cache and finds a miss. What happens next at step 4?
easy
A. The application queries the database, then updates the cache with the retrieved data
B. The cache automatically loads data from the database without application intervention
C. The application returns a null response to the user and retries later
D. The cache triggers a background refresh asynchronously and returns stale data

Solution

  1. Step 1: Understand Cache Aside flow

    On cache miss, application queries DB and updates cache explicitly.

  2. Step 2: Eliminate other options

    Cache Aside does not auto-load cache (A); it does not return null or stale data (C, D).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Application loads DB and updates cache on miss [check]
Hint: Cache Aside: app loads DB and updates cache on miss [check]
Common Mistakes:
2. A ticket booking system must prevent double booking by temporarily reserving seats during payment processing. Which component is best suited to handle this seat locking with a timeout?
easy
A. Use a message queue to serialize seat booking requests and process them sequentially
B. Use a relational database transaction with row-level locks to hold seats until payment completes
C. Use a distributed in-memory cache with TTL (e.g., Redis) to lock seats temporarily
D. Use client-side locking where the user's browser holds the seat reservation state

Solution

  1. Step 1: Identify seat locking requirements

    Seat locking must be temporary with automatic expiration to avoid indefinite holds if payment fails.

  2. Step 2: Evaluate options for temporary locks

    Relational DB locks block resources and can cause deadlocks; message queues serialize but add latency and don't expire locks; client-side locking is unreliable and insecure. Distributed cache with TTL provides automatic expiration and fast access.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Redis TTL auto-expires locks, preventing stale seat holds [check]
Hint: Temporary locks need auto-expiry; Redis TTL fits best [OK]
Common Mistakes:
3. In the Snowflake ID generation flow, at Step 3 the sequence number is incremented after the timestamp and machine ID are set. What happens if the sequence number exceeds its maximum value within the same millisecond?
easy
A. The generator blocks and waits until the next millisecond to reset the sequence number.
B. The system resets the sequence to zero and continues generating IDs immediately.
C. The sequence number wraps around causing duplicate IDs within the same millisecond.
D. The machine ID is incremented to avoid sequence overflow.

Solution

  1. Step 1: Understand sequence number role in Snowflake

    The sequence number differentiates IDs generated within the same millisecond on the same machine.

  2. Step 2: Analyze overflow handling

    If the sequence exceeds its max (e.g., 4095), the generator must wait for the next millisecond to avoid duplicates. Resetting immediately (A) risks duplicates; wrapping (C) causes collisions; incrementing machine ID (D) is not part of the design.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sequence overflow -> wait next millisecond to avoid duplicates [check]
Hint: Sequence overflow blocks until next millisecond [OK]
Common Mistakes:
4. If Redis becomes unavailable in a centralized rate limiting system, what is the immediate failure mode for incoming requests?
medium
A. API servers allow all requests without rate limiting, risking overload
B. Requests are queued locally until Redis recovers, causing increased latency
C. API servers reject all requests immediately to avoid exceeding limits
D. API servers switch to a backup Redis instance automatically without delay

Solution

  1. Step 1: Understand Redis unavailability impact

    Without Redis, the centralized counter is unreachable, so rate limiting cannot be enforced.

  2. Step 2: Check fallback behavior

    Most implementations allow requests to proceed without rate limiting to avoid blocking, risking overload.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Redis down -> no rate limiting -> requests allowed unchecked [check]
Hint: No Redis means no rate limiting, requests pass unchecked [check]
Common Mistakes:
5. At 10x scale, which component in a primary-replica replication system is most likely to fail first and why?
hard
A. Replica storage I/O, due to increased write amplification from applying logs
B. Primary server CPU, due to increased write serialization overhead
C. Network bandwidth between primary and replicas, due to replication log volume growth
D. Replica query layer, due to increased read traffic from clients

Solution

  1. Step 1: Identify bottleneck at 10x scale

    Replication log volume grows linearly with writes, stressing network bandwidth between primary and replicas.

  2. Step 2: Compare other components

    Primary CPU scales with writes but often has headroom; replica storage I/O and query layer scale with reads, less impacted initially.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Replication network bandwidth -> first bottleneck at scale [check]
Hint: Replication network saturates first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard