Bird
Raised Fist0

Ad Click-Through Rate Prediction - Logistic Regression to DLRM

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Ad Click-Through Rate Prediction - Logistic Regression to DLRM
Designing a scalable CTR prediction system transitioning from Logistic Regression to DLRM
system-designhardFreq: very_highAds & Monetization
Prerequisites:Logistic RegressionCalibrationFeature Engineering
🎯
Problem Framing
Business Objective

Maximize revenue by increasing the click-through rate (CTR) of displayed ads.

ML Objective

Predict P(click|user, ad, context) to rank ads by expected CTR.

Constraints
  • P99 latency < 50ms for ad serving
  • High throughput: 10,000 QPS
  • Cold start for new ads and users
  • Real-time feature updates
  • Compliance with data privacy regulations
Clarifying Questions
  • What is the current CTR baseline?
  • How often do new ads and users enter the system?
  • What is the acceptable trade-off between latency and accuracy?
  • Are there any specific privacy regulations we need to comply with?
  • What is the expected scale of data ingestion per day?
Out of Scope
✗ Ad creative generation
✗ User segmentation beyond CTR prediction
✗ Non-click engagement metrics
📊
Offline Metrics
PR-AUC: Preferred due to class imbalance with typically low click rates.
Log Loss: Measures the probability estimates' calibration.
📈
Online Metrics
CTR: Directly measures the effectiveness of the ad ranking.
Revenue per user session: Indicates the monetization success.
⚖️
Trade-offs

Offline metrics like PR-AUC may not always correlate with online CTR due to real-world factors like ad fatigue or seasonal trends.

💼
Business Metric

Overall revenue generated from ad clicks.

🗄️
Data Sources
  • User interaction logs: Clicks and impressions, updated in real-time.
  • Ad metadata: Static data about ads, updated daily.
  • User profiles: Demographics and preferences, updated weekly.
  • Contextual data: Time of day, device type, updated in real-time.
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user clicks and non-clicks.

Data Volume

Approximately 1 billion rows per day, 10 TB per month.

Freshness

Features must be updated in real-time to reflect the latest user interactions.

⚠️
Data Challenges
  • Sparse labels due to low click-through rates.
  • Position bias affecting click likelihood.
  • Cold start for new ads and users.
  • Survivorship bias in user interaction data.
🔧
Key Features
numerical
User past click history
Indicates user interest patterns.
categorical
Ad category
Helps in understanding user-ad relevance.
numerical
Time of day
Captures temporal patterns in ad engagement.
categorical
Device type
Different devices may have different engagement rates.
embedding
User demographics
Encodes user characteristics for better personalization.
⚡
Feature Engineering Notes
Real-time vs Batch

User interaction features need real-time computation, while user demographics can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
import pandas as pd

def compute_user_click_rate(user_logs):
    clicks = user_logs['clicks'].sum()
    impressions = user_logs['impressions'].sum()
    return clicks / impressions if impressions > 0 else 0
🤖
Model Selection
Baseline

Logistic Regression with calibrated probabilities.

Proposed Model

Deep Learning Recommendation Model (DLRM)

DLRM captures complex user-item interactions through embeddings and is scalable for high-dimensional data.
✗
Alternatives Rejected
XGBoost
Fails to efficiently handle high-dimensional sparse features and user-item interactions.
Matrix Factorization
Limited to linear interactions and lacks the ability to incorporate rich contextual features.
🏋️
Training Strategy

Hybrid approach with offline training on historical data and online updates for real-time adaptation.

🚀
Serving Infrastructure
LatencyP99 < 50ms end-to-end
Throughput10,000 QPS
ArchitectureTwo-stage architecture with retrieval followed by ranking to efficiently handle large candidate sets.
CachingCache popular ad predictions with a TTL of 5 minutes to reduce computation load.
FallbackUse a rule-based system based on historical CTR if the model is unavailable.
📡
Data Drift Signals
Shift in user feature distributions
Change in ad category distribution
Temporal drift in click patterns
📉
Model Drift Signals
Decrease in PR-AUC
Shift in prediction score distribution
Increase in log loss
🔔
Alerting & Retraining
Retraining Trigger

Retrain if PR-AUC drops by more than 5% over a week.

Alerting Strategy

On-call engineer paged for significant metric drops, logs for prediction errors, auto-recovery for minor drifts.

🎯
Interview Angles
  • How to handle cold start problems in CTR prediction?
  • Trade-offs between model complexity and serving latency.
  • Strategies for real-time feature updates and their impact on model performance.
⚠️
Common Mistakes
Ignoring the impact of position bias on click data.
Overlooking the need for real-time feature updates.
Choosing models without considering serving latency constraints.
Failing to address data privacy compliance in feature engineering.
📌
Key Decisions
  • Choosing DLRM for its ability to handle high-dimensional sparse data.
  • Implementing a two-stage serving architecture to balance latency and accuracy.
  • Designing a robust monitoring system to detect and respond to data and model drift.

Practice

(1/5)
1. In a two-stage retrieval system, what is the primary role of the candidate generation stage?
easy
A. To filter out irrelevant items from the entire dataset.
B. To rank the candidates based on relevance scores.
C. To apply complex models for precise ranking.
D. To ensure all items are considered for ranking.

Solution

  1. Step 1: Understand the candidate generation role

    The candidate generation stage is responsible for quickly filtering out irrelevant items to reduce the dataset size for the ranking stage.

  2. Step 2: Eliminate wrong options

    To rank the candidates based on relevance scores. is incorrect because ranking is done in the second stage. To apply complex models for precise ranking. is incorrect because complex models are typically used in the ranking stage. To ensure all items are considered for ranking. is incorrect because not all items are considered for ranking, only the filtered candidates.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Candidate generation filters, not ranks. [OK]
Hint: Candidate generation filters, not ranks. [OK]
Common Mistakes:
2. When choosing between Isolation Forest and Autoencoder for anomaly detection, which scenario favors using an Autoencoder?
medium
A. When the dataset is small and labeled.
B. When interpretability of the model is crucial.
C. When the anomalies are expected to be in high-dimensional data.
D. When the dataset is large and unlabeled.

Solution

  1. Step 1: Consider the strengths of Autoencoders.

    Autoencoders are particularly effective for high-dimensional data due to their ability to learn complex representations.

  2. Step 2: Eliminate wrong options.

    When the dataset is small and labeled. is incorrect because Autoencoders require more data to train effectively. When interpretability of the model is crucial. is incorrect because Autoencoders are less interpretable than Isolation Forests. When the dataset is large and unlabeled. is incorrect because Autoencoders are computationally intensive and may not be ideal for very large datasets.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Autoencoder = high-dimensional data strength [OK]
Hint: Autoencoder = high-dimensional data strength [OK]
Common Mistakes:
3. When designing a vision-language model for real-time applications, which trade-off is most critical to consider?
medium
A. Batch size vs. convergence speed
B. Model accuracy vs. training time
C. Data augmentation vs. overfitting
D. Model size vs. inference latency

Solution

  1. Step 1: Identify real-time application constraints

    Real-time applications require fast inference, making latency a critical factor.

  2. Step 2: Evaluate trade-offs

    Model size directly impacts inference latency, especially in resource-constrained environments.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Real-time needs low latency; size affects speed. [OK]
Hint: Real-time needs low latency; size affects speed. [OK]
Common Mistakes:
4. As the number of users and ads increases tenfold in an ad auction system, which component is most likely to fail first?
hard
A. User Data Storage
B. Ad Ranking Module
C. Bid Management System
D. Network Bandwidth

Solution

  1. Step 1: Consider scale impact

    With a tenfold increase in users and ads, the User Data Storage is likely to face capacity issues first due to the sheer volume of data.

  2. Step 2: Eliminate wrong options

    • Ad Ranking Module: Computationally intensive but not storage-bound.
    • Bid Management System: Handles bids, not directly affected by user volume.
    • Network Bandwidth: Affects data transfer, but storage is a more immediate bottleneck.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Storage fails first with data volume increase. [OK]
Hint: Storage fails first with data volume increase. [OK]
Common Mistakes:
5. As the number of users and items grows significantly, which component of a Two-Tower Model for Retrieval is likely to become a bottleneck first?
hard
A. Similarity score computation
B. Item embedding computation
C. User embedding computation
D. Model training time

Solution

  1. Step 1: Consider the scaling impact on components

    As users and items grow, computing similarity scores between all pairs becomes computationally expensive.

  2. Step 2: Eliminate wrong options

    User embedding computation and B scale linearly with users and items. Model training time is affected but not as immediately as similarity computation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Similarity computation scales poorly with size. [OK]
Hint: Similarity computation scales poorly with size. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard