Bird
Raised Fist0

Two-Tower Model for Retrieval - User and Item Towers

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Two-Tower Model for Retrieval - User and Item Towers
Efficient retrieval using dual embeddings for users and items.
system-designhardFreq: very_highRecommendations
Prerequisites:EmbeddingsTwo-Stage Retrieval
🎯
Problem Framing
Business Objective

Maximize user engagement by increasing the relevance of recommended items.

ML Objective

Maximize P(relevant|user, item) by learning effective user and item embeddings.

Constraints
  • Latency SLA: P99 < 100ms for retrieval
  • Cold start for new users and items
  • Scalability to millions of users and items
  • Data privacy and compliance with regulations
Clarifying Questions
  • What is the expected scale of users and items?
  • How frequently do new items get added to the catalog?
  • What are the latency requirements for the retrieval system?
  • How is user feedback collected and used for model updates?
  • Are there any specific privacy regulations to consider?
Out of Scope
✗ Detailed ranking model design
✗ UI/UX considerations for displaying recommendations
✗ Backend infrastructure for non-ML components
📊
Offline Metrics
Recall@K: Measures the proportion of relevant items retrieved in the top K results, important for ensuring relevant items are not missed.
Embedding Cosine Similarity: Evaluates the quality of learned embeddings by measuring similarity between user and item vectors.
📈
Online Metrics
CTR: Click-through rate of recommended items, directly tied to user engagement.
Conversion Rate: Percentage of recommended items that lead to a purchase or desired action.
⚖️
Trade-offs

High offline recall may not always translate to high CTR if the top retrieved items are not ranked effectively.

💼
Business Metric

Overall increase in user engagement measured by session duration and interaction frequency.

🗄️
Data Sources
  • User interaction logs: Clicks, views, and purchases, updated in real-time.
  • Item catalog: Metadata about items, updated daily.
  • User profiles: Demographic and preference data, updated weekly.
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions (clicks, views) is used to infer relevance.

Data Volume

Approximately 10 million interactions per day, 1 TB of data per month.

Freshness

Features should be updated in near real-time to reflect the latest user interactions.

⚠️
Data Challenges
  • Sparse labels due to implicit feedback
  • Position bias in interaction logs
  • Cold start for new users and items
  • Data privacy concerns with user data
🔧
Key Features
embedding
User Embedding
Captures user preferences and behavior patterns.
embedding
Item Embedding
Represents item characteristics and popularity.
numerical
User-Item Interaction History
Provides context on past interactions.
categorical
Time of Day
Accounts for temporal patterns in user behavior.
categorical
Device Type
Helps tailor recommendations based on device usage.
⚡
Feature Engineering Notes
Real-time vs Batch

User and item embeddings are pre-computed in batch, while interaction history is updated in real-time.

Feature Store

Required

🐍
Feature Snippet
Python
import numpy as np
user_embedding = np.random.rand(128)
item_embedding = np.random.rand(128)
interaction_score = np.dot(user_embedding, item_embedding)
🤖
Model Selection
Baseline

Popularity-based model recommending top N popular items.

Proposed Model

Two-Tower Neural Network with separate user and item towers.

Allows pre-computation of item embeddings, reducing online latency to a single dot product lookup.
✗
Alternatives Rejected
Matrix Factorization: Rejected due to inability to handle new users/items effectively.
XGBoost: Rejected due to high latency and complexity in capturing user-item interactions.
🏋️
Training Strategy

Offline training with periodic updates using batch processing to incorporate new data.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end for retrieval.
ThroughputTarget of 10,000 QPS.
ArchitectureTwo-stage (retrieval + ranking) to efficiently narrow down candidates before detailed ranking.
CachingCache popular item embeddings with a TTL of 1 hour, invalidate on catalog updates.
FallbackUse a rule-based system recommending popular items if the model is unavailable.
📡
Data Drift Signals
Shift in user interaction patterns
Change in item popularity distribution
Increase in missing or null values in features
📉
Model Drift Signals
Decrease in Recall@K
Drop in CTR
Shift in embedding similarity distributions
🔔
Alerting & Retraining
Retraining Trigger

Retrain if Recall@K drops by more than 10% over a week.

Alerting Strategy

Page on-call engineer if CTR drops by 20% in a day, log all prediction errors, auto-recover by switching to fallback.

🎯
Interview Angles
  • How does the two-tower model handle cold start problems?
  • What are the trade-offs between offline and online metrics?
  • How do you ensure data privacy in the feature engineering process?
⚠️
Common Mistakes
Ignoring the latency constraints when selecting models.
Overlooking the importance of real-time feature updates.
Failing to address cold start issues for new users/items.
Not considering data privacy and compliance in data handling.
📌
Key Decisions
  • Choosing a two-tower model for efficient retrieval.
  • Balancing between pre-computed embeddings and real-time features.
  • Implementing a robust fallback strategy for model unavailability.

Practice

(1/5)
1. In a Retrieval-Augmented Generation (RAG) system, which component is responsible for fetching relevant documents from a large corpus?
easy
A. The retriever model
B. The generator model
C. The encoder model
D. The decoder model

Solution

  1. Step 1: Identify the role of each component.

    The retriever model is designed to fetch relevant documents from a large corpus based on a query.

  2. Step 2: Eliminate wrong options.

    The generator model creates text based on retrieved documents, the encoder processes input data, and the decoder generates output from encoded data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Retriever fetches, generator creates. [OK]
Hint: Retriever fetches, generator creates. [OK]
Common Mistakes:
2. In a scenario where model accuracy is critical, which approach should be avoided to prevent potential accuracy degradation?
medium
A. KV Cache
B. Speculative Decoding
C. Batching
D. Model Pruning

Solution

  1. Step 1: Understand the impact on accuracy.

    Speculative Decoding can introduce errors by predicting future tokens.

  2. Step 2: Eliminate wrong options.

    Batching and KV Cache do not inherently affect accuracy. Model Pruning reduces model size but can be controlled to maintain accuracy.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Speculative Decoding risks accuracy for speed. [OK]
Hint: Speculative Decoding risks accuracy for speed. [OK]
Common Mistakes:
3. In an LLM serving system, what is a likely failure mode if the KV Cache is not properly managed?
medium
A. Inaccurate predictions due to speculative decoding
B. Decreased throughput due to batching inefficiencies
C. Higher memory usage leading to system crashes
D. Increased latency due to redundant computations

Solution

  1. Step 1: Identify the role of KV Cache.

    KV Cache prevents redundant computations, reducing latency.

  2. Step 2: Eliminate wrong options.

    Batching inefficiencies affect throughput, not KV Cache. Memory issues are possible but less direct. Speculative decoding affects accuracy, not KV Cache.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    KV Cache mismanagement leads to redundant computations. [OK]
Hint: KV Cache mismanagement leads to redundant computations. [OK]
Common Mistakes:
4. At 10x user scale, which component of a feed ranking system is most likely to become a bottleneck first?
hard
A. Network bandwidth
B. Data storage
C. Model training
D. Feature extraction

Solution

  1. Step 1: Analyze scaling impact

    Feature extraction often involves complex computations that scale with data volume.

  2. Step 2: Eliminate wrong options

    Data storage can be scaled with distributed systems, model training is periodic, and network bandwidth is less likely to bottleneck first.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Feature extraction scales with data. [OK]
Hint: Feature extraction scales with data. [OK]
Common Mistakes:
5. At 10x data scale in a fine-tuning pipeline using SFT, RLHF, and DPO, which component is most likely to become a bottleneck first?
hard
A. Data Preprocessing Optimization (DPO)
B. Supervised Fine-Tuning (SFT)
C. Reinforcement Learning from Human Feedback (RLHF)
D. Model Deployment Infrastructure

Solution

  1. Step 1: Analyze the impact of scaling on each component

    DPO involves handling and optimizing large volumes of data, which can become a bottleneck as data scales.

  2. Step 2: Eliminate wrong options

    B: SFT scales with model size, not data volume. C: RLHF is more about feedback processing, not data volume. D: Deployment infrastructure is affected by model size, not data volume.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DPO bottlenecks with data scale. [OK]
Hint: DPO bottlenecks with data scale. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard