Bird
Raised Fist0

Ad Auction System - Ranking Ads by Expected Value

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Ad Auction System - Ranking Ads by Expected Value
Optimize ad ranking by predicting expected value to maximize revenue.
system-designhardFreq: highAds & Monetization
Prerequisites:Ad CTR PredictionCalibration
🎯
Problem Framing
Business Objective

Maximize revenue from ad auctions by optimizing the ranking of ads based on expected value.

ML Objective

Predict the expected value of an ad impression, defined as P(click|user, ad, context) * bid amount.

Constraints
  • P99 latency < 50ms for ad ranking
  • High throughput: 10,000 QPS
  • Cold start for new ads and users
  • Regulatory compliance with user data privacy
  • Limited labeled data for new ad campaigns
Clarifying Questions
  • What is the latency requirement for ad ranking?
  • How often do new ads and users enter the system?
  • What is the scale of data we are dealing with?
  • Are there any specific privacy regulations we need to comply with?
  • What is the acceptable trade-off between latency and accuracy?
Out of Scope
✗ Ad creative generation
✗ User segmentation
✗ Bid optimization strategies
✗ Fraud detection
📊
Offline Metrics
PR-AUC: Handles class imbalance better than ROC-AUC, crucial for low click-through rates.
Log Loss: Measures the uncertainty of predictions, penalizing overconfident wrong predictions.
📈
Online Metrics
CTR: Directly measures the click-through rate of displayed ads.
Revenue per session: Tracks the revenue generated per user session.
⚖️
Trade-offs

Offline metrics like PR-AUC may show improvement, but if CTR does not increase, the model may not be capturing real-world user behavior effectively.

💼
Business Metric

Total revenue generated from ad impressions.

🗄️
Data Sources
  • User interaction logs: Clicks and impressions, updated in real-time.
  • Ad inventory database: Details of available ads, updated daily.
  • Bid data: Real-time bids from advertisers.
  • User profile data: Demographics and preferences, updated weekly.
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user clicks and non-clicks, as explicit labels are not feasible at scale.

Data Volume

Approximately 100 million rows per day, equating to 10 TB of data per month.

Freshness

Features must be updated in real-time to reflect the latest user interactions and ad bids.

⚠️
Data Challenges
  • Sparse labels due to low click-through rates.
  • Position bias affecting click probability.
  • Survivorship bias from only successful ads being analyzed.
  • Cold start for new ads and users with no historical data.
🔧
Key Features
categorical
User demographics
Personalizes ad relevance.
categorical
Ad category
Matches user interests with ad content.
numerical
Historical CTR
Indicates ad performance over time.
categorical
Time of day
Captures temporal patterns in user behavior.
categorical
Device type
Optimizes ad format for different devices.
numerical
Bid amount
Directly impacts expected revenue.
embedding
User-ad interaction embeddings
Captures complex user-ad relationships.
⚡
Feature Engineering Notes
Real-time vs Batch

Real-time computation for user interactions and bid amounts; batch for user demographics and historical CTR.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_user_ad_embedding(user_id, ad_id):
    user_vector = user_embedding_lookup(user_id)
    ad_vector = ad_embedding_lookup(ad_id)
    return np.dot(user_vector, ad_vector)
🤖
Model Selection
Baseline

Logistic regression using basic features like historical CTR and bid amount.

Proposed Model

Two-tower neural network model.

The two-tower model efficiently captures user-ad interactions through embeddings, allowing precomputation of ad embeddings to meet latency constraints.
✗
Alternatives Rejected
XGBoost: Rejected due to inability to handle high-dimensional sparse features efficiently.
DeepFM: Rejected due to increased complexity and latency overhead not justified by performance gains.
Collaborative filtering: Rejected as it fails to incorporate real-time context and bid information.
🏋️
Training Strategy

Hybrid approach with offline training on historical data and online updates for real-time adaptation.

🚀
Serving Infrastructure
LatencyP99 < 50ms end-to-end for ad ranking.
Throughput10,000 QPS
ArchitectureTwo-stage architecture with retrieval followed by ranking to efficiently handle large ad inventories.
CachingCache top N ads per user segment with a TTL of 5 minutes to reduce computation load.
FallbackUse rule-based ranking based on historical CTR and bid amount if the model is unavailable.
📡
Data Drift Signals
Shift in user demographics distribution.
Change in ad category distribution.
Variation in bid amount distribution.
📉
Model Drift Signals
Decrease in PR-AUC over time.
Shift in prediction score distribution.
Increase in log loss indicating prediction uncertainty.
🔔
Alerting & Retraining
Retraining Trigger

Retrain if PR-AUC drops by more than 5% over a month.

Alerting Strategy

On-call engineer paged for significant metric drops; logs for prediction errors; auto-recovery by rolling back to previous model version.

🎯
Interview Angles
  • How do you handle cold start problems for new ads?
  • What trade-offs do you consider between latency and model complexity?
  • How do you ensure compliance with data privacy regulations?
⚠️
Common Mistakes
Ignoring the impact of position bias on click predictions.
Overlooking the need for real-time feature updates.
Choosing overly complex models that exceed latency constraints.
Failing to account for data privacy in feature engineering.
📌
Key Decisions
  • Selecting a two-tower model to balance complexity and latency.
  • Implementing a hybrid training strategy for real-time adaptability.
  • Designing a two-stage serving architecture to handle large-scale ad inventories.

Practice

(1/5)
1. In the context of avoiding filter bubbles in recommendation systems, which component is responsible for ensuring content diversity?
easy
A. Collaborative filtering
B. Diversity re-ranking layer
C. User profiling
D. Content-based filtering

Solution

  1. Step 1: Identify the role of each component

    Collaborative and content-based filtering focus on user preferences, while diversity re-ranking ensures varied content.

  2. Step 2: Eliminate wrong options

    Collaborative filtering (A) and content-based filtering (B) do not inherently ensure diversity. User profiling (D) focuses on understanding user preferences, not diversity.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Diversity re-ranking ensures varied content [OK]
Hint: Diversity re-ranking ensures varied content [OK]
Common Mistakes:
2. When deploying a Two-Tower Model for Retrieval, which approach is better for handling cold-start users and why?
medium
A. Use random embeddings until enough data is collected.
B. Use collaborative filtering to generate initial embeddings.
C. Use content-based features to generate initial embeddings.
D. Use embeddings from a pre-trained language model.

Solution

  1. Step 1: Consider the cold-start problem

    Cold-start users lack interaction data, making content-based features useful for initial embeddings.

  2. Step 2: Eliminate wrong options

    Use collaborative filtering to generate initial embeddings. relies on interaction data, which cold-start users lack. Use random embeddings until enough data is collected. provides no meaningful information. Use embeddings from a pre-trained language model. might not capture user-specific features effectively.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Content-based features help with cold-start. [OK]
Hint: Content-based features help with cold-start. [OK]
Common Mistakes:
3. In a visual search system, when should you prefer using an Approximate Nearest Neighbor (ANN) index over an exact search method?
medium
A. When low latency is crucial
B. When the dataset is small
C. When exact matches are critical
D. When the search space is one-dimensional

Solution

  1. Step 1: Consider the trade-offs

    ANN is preferred when low latency is crucial because it speeds up search at the cost of some accuracy.

  2. Step 2: Eliminate wrong options

    Exact matches require exact search, small datasets don't benefit much from ANN, and one-dimensional search doesn't need ANN complexity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    ANN for speed, not exactness. [OK]
Hint: ANN for speed, not exactness. [OK]
Common Mistakes:
4. As a content-based filtering system scales to 10x more items, which component is most likely to become a bottleneck?
hard
A. Item embedding computation
B. User profile storage
C. Recommendation generation
D. User-item interaction logging

Solution

  1. Step 1: Analyze scaling impact

    Scaling to more items increases the demand for computing and updating item embeddings, making it a potential bottleneck.

  2. Step 2: Eliminate wrong options

    A: User profile storage scales with users, not items. C: Recommendation generation is more affected by user scale. D: Interaction logging is more about user activity than item count.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    More items strain embedding computation. [OK]
Hint: More items strain embedding computation. [OK]
Common Mistakes:
5. As the number of queries increases tenfold, which component of a search ranking system is most likely to become a bottleneck first?
hard
A. Ranking computation
B. Document retrieval
C. Query parsing
D. Index updating

Solution

  1. Step 1: Consider the impact of scaling queries

    Ranking computation is resource-intensive and scales with query volume.

  2. Step 2: Evaluate other components

    Query parsing and document retrieval are less computationally intensive, and index updating is not query-dependent.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Ranking computation scales with query volume. [OK]
Hint: Ranking computation scales with query volume. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard