Bird
Raised Fist0

Design a Search Ranking System - Feature Store, Model Serving, A/B Testing

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a Search Ranking System - Feature Store, Model Serving, A/B Testing
Design a scalable search ranking system integrating feature store, model serving, and A/B testing for 2B DAU
hardAI and ML System DesignFreq: high
Prerequisites:Inverted IndexElasticsearch
⚙️
Functional Requirements
  • Real-time feature retrieval for ranking models with low latency (<50ms)
  • Model serving infrastructure to score search results using ML models at 100ms P99
  • Support for multiple ranking models with A/B testing and traffic splitting
  • Ingestion pipeline to update feature store with user and item features continuously
  • Search query processing integrating Elasticsearch inverted index with ranking scores
  • Dashboard and metrics for monitoring A/B test performance and model metrics
📊
Non-Functional Requirements
  • Support 2 billion daily active users generating 40 search queries per user per day
  • System must handle 925,000 write QPS for feature updates and 37 million read QPS for ranking
  • End-to-end ranking latency must be under 100ms P99 including feature fetch and model scoring
  • Availability SLA of 99.99% for ranking service and feature store
  • Data retention for feature store: 30 days with replication for durability
❓
Clarifying Questions to Ask
  • What is the expected query per second (QPS) for search requests and feature updates?
  • Are features user-specific, item-specific, or context-specific, and how frequently do they update?
  • What types of ML models are used for ranking (e.g., tree ensembles, deep neural nets)?
  • Is the ranking model inference done synchronously during query or asynchronously?
  • What is the acceptable latency budget for feature retrieval and model scoring separately?
  • How is A/B testing traffic split managed and what metrics are critical for evaluation?
Out of Scope
✗ Design of the inverted index and basic search query parsing (assumed Elasticsearch)
✗ Training pipeline for ML ranking models (only serving and feature retrieval covered)
✗ User interface and frontend search experience
✗ Long-term offline analytics and batch processing
📌
Assumptions
  • 2 billion DAU
  • 40 search queries per user per day
  • 100 bytes average feature update message size
  • Each search query requires fetching 10 features per candidate, 5 candidates scored
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec (feature updates per query)
peak write qpsAssuming 2× peak factor → 1,851,852 writes/sec
read qps2B × 40 × 5 candidates / 86400 ≈ 4,629,630 reads/sec for feature fetch
peak read qpsAssuming 2× peak factor → 9,259,260 reads/sec
💾
Storage Estimate
daily storage925,926 writes/sec × 100 bytes × 86400 sec ≈ 8 TB/day
retention 30 days8 TB × 30 = 240 TB
with replication 3x240 TB × 3 = 720 TB total storage
🖥️
Server Estimate
feature storeAssuming Cassandra nodes with 10K writes/sec/node → 185 nodes for peak writes
model servingAssuming 1 ms per inference per core, 100ms latency budget, 9.26M QPS → ~10,000 cores (distributed GPU/CPU cluster)
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          | Search Client  |
          +-------+--------+
                  | Query
                  v
          +-------+--------+
          | Elasticsearch  |
          | (Inverted Index)|
          +-------+--------+
                  | Top-K candidates
                  v
          +-------+--------+
          | Feature Store  |
          | (Cassandra +   |
          |  Redis Cache)  |
          +-------+--------+
                  | Features
                  v
          +-------+--------+
          | Model Serving  |
          | (TensorFlow   |
          |  Serving +    |
          |  Triton)      |
          +-------+--------+
                  | Scores
                  v
          +-------+--------+
          | Ranking Logic |
          +-------+--------+
                  | Ranked results
                  v
          +-------+--------+
          | A/B Testing   |
          | Controller    |
          +---------------+
🧩
Components
Elasticsearch
Provides inverted index search to retrieve top-K candidate documents for a query
Why thisElasticsearch supports distributed inverted index with horizontal scaling and low-latency full-text search
Why notSolr requires more manual shard management and has higher latency at 10M QPS scale
Feature Store (Cassandra + Redis Cache)
Stores real-time user/item/context features for ranking; Redis caches hot features for sub-ms reads
Why thisCassandra offers linear write scalability for 1.8M writes/sec with tunable consistency; Redis TTL cache reduces read latency
Why notPostgreSQL cannot handle 1.8M writes/sec due to B-tree write amplification and complex sharding
Model Serving (TensorFlow Serving + NVIDIA Triton)
Hosts ML ranking models and serves inference requests with GPU acceleration and batching
Why thisTensorFlow Serving integrates with Triton for GPU batching, enabling 100ms P99 latency at 9M QPS
Why notCustom REST microservices lack GPU batching optimizations and cannot meet latency at scale
Ranking Logic Service
Combines model scores with business rules and finalizes ranked search results
Why thisDedicated microservice allows flexible ranking logic updates without redeploying model serving
Why notEmbedding ranking logic inside model serving reduces modularity and increases deployment risk
A/B Testing Controller
Manages traffic splits, experiment variants, and collects metrics for ranking model evaluation
Why thisFeature flag system integrated with metrics pipeline enables controlled rollout and rollback of ranking models
Why notManual traffic routing risks inconsistent user experience and lacks automated metric collection
🔄
Data Flow
1
Search Client → Elasticsearch
Send user search query
Retrieve top-K candidate documents using inverted index
2
Elasticsearch → Feature Store
Request features for candidates
Fetch user/item/context features for ranking
3
Feature Store → Model Serving
Send feature vectors
Model serving scores candidates using ML models
4
Model Serving → Ranking Logic Service
Return scores
Combine scores with business rules for final ranked results
5
Ranking Logic Service → Search Client
Return ranked search results
Deliver final ranked list to user
6
A/B Testing Controller → Model Serving
Manage traffic splits
Control which ranking model variant serves requests
7
A/B Testing Controller → Metrics Dashboard
Send experiment metrics
Monitor A/B test performance and model metrics

Practice

(1/5)
1. A global video streaming service needs to reduce latency for users worldwide by caching content close to them. Which CDN component primarily handles this latency reduction?
easy
A. Origin server that stores the master copy of all content
B. DNS servers resolving domain names to IP addresses
C. Load balancers distributing requests among origin servers
D. Edge nodes located in Points of Presence (PoPs) near users

Solution

  1. Step 1: Identify latency reduction mechanism

    Latency is reduced by caching content closer to users, which is done by edge nodes in PoPs.

  2. Step 2: Eliminate other components

    Origin servers hold master content but are far from users; load balancers and DNS do not cache content.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Edge nodes cache content near users -> latency reduced [check]
Hint: Edge nodes cache near users for low latency [check]
Common Mistakes:
2. A distributed key-value store must evenly distribute keys across nodes and handle node failures without massive data reshuffling. Which component primarily enables this?
easy
A. Gossip protocol to propagate membership and failure information
B. Vector clocks to resolve conflicting versions of data
C. Consistent hashing to assign keys to nodes with minimal remapping
D. Quorum-based replication to ensure data durability

Solution

  1. Step 1: Identify the role of consistent hashing

    Consistent hashing distributes keys to nodes such that when nodes join or leave, only a small subset of keys need reassignment, minimizing reshuffling.

  2. Step 2: Eliminate other components

    Vector clocks handle version conflicts but do not distribute keys. Gossip protocol manages cluster membership but not key distribution. Quorum replication ensures durability but does not affect key assignment.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Consistent hashing minimizes key remapping on node changes [check]
Hint: Consistent hashing minimizes key remapping on node changes [OK]
Common Mistakes:
3. For a social network with users having highly variable follower counts (from tens to millions), which fan-out architecture best balances write latency and read latency?
medium
A. Pure pull fan-out: followers fetch posts on demand from a global store
B. Pure push fan-out: push every post to all followers' feeds synchronously
C. Hybrid fan-out: push to high-follower-count users' feeds, pull for low-follower-count users
D. Push fan-out only for active users, pull for inactive users

Solution

  1. Step 1: Analyze pure push fan-out (A)

    Push to all followers synchronously causes huge write amplification for celebrities, increasing write latency and storage.

  2. Step 2: Analyze pure pull fan-out (C)

    Pull reduces write cost but increases read latency and complexity for users with many followers.

  3. Step 3: Evaluate hybrid fan-out (B)

    Hybrid pushes to users with many followers (to reduce read latency) and pulls for users with few followers (to reduce write amplification), balancing both latencies.

  4. Step 4: Evaluate push only for active users (D)

    Activity-based push doesn't address follower count skew effectively.

  5. Final Answer:

    Option C -> Option C

  6. Quick Check:

    Hybrid fan-out balances write and read latency by follower count [check]
Hint: Hybrid fan-out balances write/read latency by follower count [check]
Common Mistakes:
4. For a mobile chat app requiring exactly-once message delivery with minimal latency, which architecture is best suited?
medium
A. Use Kafka with transactional producers and consumers for exactly-once semantics
B. Use at-most-once delivery with client retries on failure
C. Use polling-based delivery with sequence numbers for ordering
D. Use a message queue with idempotent consumers and client-side deduplication

Solution

  1. Step 1: Evaluate Kafka transactions for mobile clients

    Kafka's transactional exactly-once requires stable connections and complex coordination, unsuitable for mobile clients with intermittent connectivity.

  2. Step 2: Assess idempotent consumers with client deduplication

    Idempotent consumers combined with client-side deduplication handle retries and duplicates gracefully, fitting mobile constraints and ensuring exactly-once delivery.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Idempotency + client deduplication -> practical exactly-once for mobile [check]
Hint: Mobile needs idempotency + deduplication, not Kafka transactions [OK]
Common Mistakes:
5. At 10x scale, which component in a bulkhead-isolated microservices system is most likely to fail first, and why?
hard
A. The shared database, because it becomes a global bottleneck under increased load
B. The bulkhead thread pools, because fixed-size partitions limit concurrency and cause request rejections
C. The load balancer, because it cannot handle the increased number of service instances
D. The network infrastructure, due to increased cross-service communication overhead

Solution

  1. Step 1: Analyze scaling impact on components

    Shared database (A) is a bottleneck but bulkheads isolate failures before database overload. Load balancer (C) scales horizontally. Network (D) overhead increases but is less immediate.

  2. Step 2: Identify bulkhead thread pool limits

    Bulkhead thread pools have fixed sizes; at 10x load, they saturate first, causing request rejections and failure to scale.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Bulkhead thread pools saturate first due to fixed resource limits [check]
Hint: Fixed bulkhead pools saturate first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard