Bird
Raised Fist0

Design Quora - Q&A Feed, Search, Ranking, Spam Detection, Follow Topics

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design Quora - Q&A Feed, Search, Ranking, Spam Detection, Follow Topics
45 min · senior
senior

You are tasked with designing a Quora-like platform serving 50 million daily active users (DAU). The system must support a personalized Q&A feed, full-text search over questions and answers, a ranking algorithm to surface relevant content, real-time spam detection, and a topic following feature. The platform must handle high write and read throughput, ensure data privacy with encryption at rest and in transit, and maintain a P99 latency under 200ms for feed and search queries.

Constraints
  • 50 million DAU generating ~10 million new questions and 50 million answers daily
  • P99 latency under 200ms for feed generation and search queries
  • Encryption at rest and in transit for all user data
  • Spam detection must operate in near real-time to prevent abusive content
  • Offline feed delivery for users with intermittent connectivity
  • Personalized ranking based on user interests and topic follows
📋
Guided Steps
1/5
Step 1
Data Storage and Write Scalability
How would you store questions and answers to support high write throughput and efficient retrieval?
💡 Hint
Consider a wide-column store optimized for high write volume and range queries by user or topic.
Key Decision
Use Apache Cassandra with partition key as question_id for answers and question_id for questions to enable linear write scalability and efficient range scans for feed generation.
Interviewer Expectation
Mid-level: Names Cassandra for write scalability. Senior: Explains partition key choice to collocate related data and avoid hotspots, rejects relational DB due to write amplification and scaling limits.
⚠️Common Wrong Answer
Using a relational database like MySQL for all writes - leads to write bottlenecks and poor horizontal scalability at 50M DAU.
Step 2
Feed Generation and Personalization
How would you design the personalized Q&A feed to meet P99 < 200ms latency and support offline delivery?
💡 Hint
Think about precomputing feeds with a distributed message queue and caching layer to reduce query time.
Key Decision
Use Apache Kafka with user_id as partition key to stream new content events, combined with Redis as a per-user feed cache with TTL for offline delivery and fast reads.
Interviewer Expectation
Mid-level: Mentions Kafka for event streaming. Senior: Explains user_id partitioning to ensure ordering and efficient feed updates, and Redis TTL caching to handle offline users and reduce backend load.
⚠️Common Wrong Answer
Polling the database directly for feed updates - causes high latency and database overload at scale.
Step 3
Search Indexing and Query
How would you implement full-text search over questions and answers with low latency and high freshness?
💡 Hint
Consider a distributed search engine that supports near real-time indexing and horizontal scaling.
Key Decision
Use Elasticsearch with daily index rollover and near real-time indexing via Kafka Connect to ingest new content, enabling sub-200ms query latency and horizontal scaling.
Interviewer Expectation
Mid-level: Names Elasticsearch for search. Senior: Details Kafka Connect ingestion pipeline for freshness, index rollover strategy for manageable shard sizes, and query routing for low latency.
⚠️Common Wrong Answer
Using a traditional RDBMS full-text search feature - does not scale well and has high query latency at 50M DAU.
Step 4
Spam Detection Pipeline
How would you design a near real-time spam detection system integrated with content ingestion?
💡 Hint
Think about streaming processing frameworks and model serving for classification before content becomes visible.
Key Decision
Use Apache Flink streaming jobs consuming from Kafka topics to run ML-based spam detection models, flagging or blocking content before writing to Cassandra and Elasticsearch.
Interviewer Expectation
Mid-level: Mentions streaming processing for spam detection. Senior: Explains Flink’s exactly-once semantics, model update strategies, and integration with content ingestion pipeline to prevent spam visibility.
⚠️Common Wrong Answer
Batch processing spam detection after content is live - allows spam to be visible and damages user experience.
Step 5
Topic Following and Ranking
How would you implement topic following and use it to rank feed content personalized per user?
💡 Hint
Consider a graph database or key-value store for user-topic relationships and a ranking service that combines signals.
Key Decision
Store user-topic follows in Redis Sorted Sets for fast membership and scoring; implement a ranking microservice using Apache Spark to compute personalized feed scores combining topic follows, recency, and engagement metrics.
Interviewer Expectation
Mid-level: Names Redis for fast lookups. Senior: Explains Sorted Sets for ranking, Spark batch jobs for offline model training, and real-time ranking service integration for feed personalization.
⚠️Common Wrong Answer
Hardcoding ranking logic in the feed service without offline model training - leads to stale and non-personalized feeds.
🔌
API Design Challenge
Sketch 3 core API endpoints for the system: posting a question or answer, fetching a personalized Q&A feed, and following/unfollowing topics.
Hints
  • How does the client submit new content and ensure consistency?
  • How does the client sync feed updates efficiently and handle pagination?
  • How does the client express interest in topics and how is that reflected in feed personalization?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of Cassandra with appropriate partition keys for write scalability
  • Kafka event streaming with partitioning strategy for feed updates
  • Elasticsearch for full-text search with near real-time indexing
  • Streaming spam detection pipeline with exactly-once processing semantics
  • Feed caching layer (Redis) to meet P99 latency and offline delivery
Good to Have - 30%
  • Personalized ranking service combining offline and real-time signals
  • Use of Redis Sorted Sets for efficient topic follow management
  • Encryption at rest and in transit for data security
  • API design that supports efficient client sync and pagination
Bonus - 10%
  • Use of Spark for offline ranking model training
  • Detailed failure mode analysis for feed generation and spam detection
  • Thoughtful API design with clear request/response contracts
Anti-patterns to Avoid
pattern: Using HTTP polling at scale
consequence: At 50M DAU, HTTP polling causes 50 million reconnects per minute just to check for messages, which collapses under load and increases latency.
pattern: Using relational databases for high write throughput
consequence: Relational DBs cause write amplification and scaling bottlenecks, leading to poor performance and inability to handle 50M DAU write volume.
pattern: Batch processing spam detection after content is live
consequence: Allows spam content to be visible to users, damaging user experience and platform trust.
pattern: Hardcoding ranking logic without offline model training
consequence: Leads to stale, non-personalized feeds that do not adapt to user interests or engagement signals.
pattern: Polling database directly for feed updates
consequence: Causes high latency and database overload at scale, failing to meet P99 latency requirements.
✅
Model Answer
▼ Reveal

This design uses Apache Cassandra for scalable writes with question_id partitioning, Apache Kafka with user_id partitioning for ordered event streaming, Redis as a per-user feed cache with TTL for offline support, Elasticsearch with Kafka Connect for near real-time full-text search, Apache Flink streaming jobs for exactly-once spam detection before persistence, Redis Sorted Sets for user-topic follows, and Apache Spark for offline ranking model training. A ranking microservice applies personalized scores in real-time. All data is encrypted in transit with TLS and at rest with AES-256. An API Gateway secures endpoints with rate limiting and authentication.

Key Decisions
  • Partition Cassandra tables by question_id to collocate related data and enable linear scalability
  • Partition Kafka topics by user_id to maintain event ordering per user for feed updates
  • Use Redis with TTL caching to support offline feed delivery and reduce backend load
  • Ingest new content into Elasticsearch via Kafka Connect for near real-time indexing and sub-200ms query latency
  • Run ML-based spam detection in Apache Flink streaming jobs with exactly-once semantics before content persistence
  • Store user-topic follows in Redis Sorted Sets for fast membership checks and scoring
  • Train ranking models offline using Apache Spark combining topic follows, recency, and engagement metrics
  • Apply personalized ranking scores in a dedicated microservice to order feed content in real-time
  • Encrypt all data in transit with TLS and at rest with AES-256 to ensure data privacy and compliance
  • Use an API Gateway with authentication and rate limiting to secure endpoints and enforce SLAs
  • Apache Cassandra with question_id partition key for scalable writes and efficient range queries
  • Apache Kafka with user_id partition key for ordered event streaming and feed update propagation
  • Redis as a per-user feed cache with TTL to enable offline feed delivery and reduce backend load
  • Elasticsearch with Kafka Connect ingestion for near real-time full-text search and horizontal scaling
  • Apache Flink streaming jobs for exactly-once spam detection integrated before content persistence
  • Redis Sorted Sets for storing user-topic follows enabling fast membership checks and ranking
  • Apache Spark for offline ranking model training combining multiple engagement signals
  • Ranking microservice applying personalized scores in real-time to order feed content
  • TLS encryption for all data in transit and AES-256 encryption at rest for user data compliance
  • API Gateway with rate limiting and authentication to secure endpoints and enforce SLAs

Practice

(1/5)
1. A system requires handling 1 million writes per second with guaranteed durability and ordered processing per user session. Which database technology best fits this requirement?
easy
A. Use a distributed log-based system like Kafka partitioned by user session
B. Use a relational database with row-level locking to ensure order and durability
C. Use an in-memory key-value store like Redis without persistence enabled
D. Use a NoSQL document store with eventual consistency and no ordering guarantees

Solution

  1. Step 1: Identify ordering and durability needs

    Relational DBs with row-level locking can't handle 1M writes/sec efficiently; Redis without persistence loses durability; NoSQL eventual consistency lacks ordering guarantees.

  2. Step 2: Recognize Kafka's partitioning for ordering

    Kafka partitions by user session ensure ordered processing and durability at high throughput.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka partitions -> ordered, durable, high-throughput writes [check]
Hint: Kafka partitions preserve order and durability at scale [check]
Common Mistakes:
2. For a system requiring cost optimization with infrequent data access but occasional retrieval within hours, which storage architecture is preferable?
medium
A. Implement warm storage with HDDs and cold storage with object storage, migrating data based on access patterns
B. Use only hot storage with SSDs for all data to ensure fastest access
C. Archive all data immediately to tape storage to minimize cost
D. Store all data in cold storage with no tiering to simplify design

Solution

  1. Step 1: Analyze access and cost requirements

    Data is infrequently accessed but needs retrieval within hours, so cold storage alone is too slow.

  2. Step 2: Evaluate tiering benefits

    Warm storage offers moderate cost and latency; cold storage reduces cost further. Tiering balances cost and retrieval time.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Warm + cold tiering balances cost and retrieval latency [check]
Hint: Warm + cold tiering balances cost and latency [check]
Common Mistakes:
3. At 10x scale, which component in a Blue-Green deployment system is most likely to fail first and why?
hard
A. Deployment pipeline because build servers cannot handle 10x concurrent builds, delaying releases
B. Green environment servers because maintaining two full production environments doubles resource usage and costs
C. Load balancer because it must instantly switch traffic for millions of users, causing connection storms
D. Feature flag service because it cannot handle 10x toggle evaluations per request

Solution

  1. Step 1: Identify scaling bottlenecks

    Blue-Green requires two full environments; load balancer scales horizontally; feature flags scale independently.

  2. Step 2: Analyze deployment pipeline

    Build servers face 10x concurrent builds, causing queueing and delays, blocking releases.

  3. Step 3: Compare other components

    Load balancer and feature flags can be sharded; doubling environment cost is expensive but not a failure.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Build servers bottleneck at 10x concurrent builds, delaying deployments [check]
Hint: Build servers choke first at 10x concurrent builds, blocking deploys [OK]
Common Mistakes:
4. At 10x scale, which component in a synchronous chat system is most likely to fail first and why?
hard
A. The database, because it cannot handle the increased write volume synchronously
B. The load balancer, because it cannot distribute synchronous requests evenly at high scale
C. The client devices, because they cannot maintain multiple synchronous connections simultaneously
D. The chat servers, because WebSocket connections are stateful and cannot be easily resharded or migrated

Solution

  1. Step 1: Identify stateful bottleneck in synchronous chat

    Chat servers maintain WebSocket connections statefully, binding clients to specific servers.

  2. Step 2: Analyze scaling challenges

    At 10x scale, resharding or migrating these stateful connections is complex, causing chat servers to fail first.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stateful WebSocket servers limit horizontal scaling, causing early failure [check]
Hint: Stateful WebSocket servers bottleneck synchronous chat scaling [check]
Common Mistakes:
5. At 10x scale, which component in a distributed unique ID generation system is most likely to fail first and why?
hard
A. The centralized timestamp authority, because it becomes a bottleneck for clock synchronization
B. The network links between data centers, due to increased latency causing clock skew and ID collisions
C. The ID generator nodes, because sequence number overflow occurs under high request rates within the same millisecond
D. The persistent storage for sequence numbers, due to write amplification and I/O saturation

Solution

  1. Step 1: Analyze bottlenecks at 10x request rate

    High request rates increase IDs generated per millisecond, stressing sequence number capacity.

  2. Step 2: Sequence number overflow causes ID collisions or generator stalls

    Sequence number bits are limited; overflow leads to duplicates or blocking until next timestamp.

  3. Step 3: Other components less likely to fail first

    Timestamp authority is decentralized or loosely synchronized; network latency affects ordering but not immediate failure; persistent storage is often avoided for sequence state.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Sequence number overflow -> first failure under 10x scale [check]
Hint: Sequence number bits limit -> overflow first at high scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard